WO2022088430A1

WO2022088430A1 - Inspection and cleaning method and apparatus of robot, robot, and storage medium

Info

Publication number: WO2022088430A1
Application number: PCT/CN2020/136691
Authority: WO
Inventors: 沈孝通; 侯林杰; 秦宝星; 程昊天
Original assignee: 上海高仙自动化科技发展有限公司
Priority date: 2020-10-29
Filing date: 2020-12-16
Publication date: 2022-05-05

Abstract

Disclosed are an inspection and cleaning method and apparatus of a robot, a robot, and a storage medium. The inspection and cleaning method of a robot comprises: collecting visual data in the field of view of the robot; performing detection on the visual data by means of a preset network and obtaining an object position and an object type of a target object, wherein the preset network is obtained by means of training of original sample data and sample annotation data corresponding to the original sample data; and controlling the robot to perform an inspection and cleaning task according to the object position and the object type of the target object.

Description

Robot inspection and cleaning method, device, robot and storage medium

This application claims the priority of the Chinese patent application with the application number 202011182064.X filed with the China Patent Office on October 29, 2020, and the Chinese patent application with the application number 202011182069.2 filed with the China Patent Office on October 29, 2020 , claim the priority of the Chinese patent application with the application number 202011186175.8 submitted to the China Patent Office on October 29, 2020, and the entire contents of the three applications are incorporated into this application by reference.

technical field

The present application relates to the field of robotics, for example, to a robot inspection and cleaning method, device, robot and storage medium.

Background technique

With the rapid development of automation technology and artificial intelligence, robots are widely used in various scenarios. Taking the cleaning scene as an example, the cleaning robot can complete simple and repetitive cleaning tasks through unmanned driving technology, greatly reducing labor costs and realizing the automation of cleaning work.

When the robot is patrolling and cleaning, it generally drives according to the pre-planned navigation map, and fully covers and cleans the ground during the driving process. However, the above patrol cleaning method results in low cleaning efficiency of the robot.

SUMMARY OF THE INVENTION

The present application provides a robot inspection and cleaning method, device, robot and storage medium.

Provide a robot inspection and cleaning method, including:

Collect visual data within the robot's field of vision;

The visual data is detected by a preset network to obtain the object position and object type of the target object, wherein the preset network is obtained by training the original sample data and the sample annotation data corresponding to the original sample data;

The robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.

Provide a robot inspection and cleaning device, including:

The acquisition module is set to collect visual data within the field of vision of the robot;

The detection module is configured to detect the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network uses the original sample data and the sample annotation data corresponding to the original sample data. trained;

The control module is configured to control the robot to perform the inspection and cleaning task according to the object position and the object type of the target object.

A robot is provided, comprising: at least one processor, a memory, and at least one program, wherein the at least one program is stored in the memory and executed by the at least one processor, the at least one program includes using Instructions for executing the above-mentioned robot inspection and cleaning method.

Provide a non-volatile computer-readable storage medium containing computer-executable instructions, when the computer-executable instructions are executed by at least one processor, the at least one processor is made to execute the above-mentioned robot inspection and cleaning method .

Description of drawings

1 is a schematic structural diagram of a robot according to an embodiment of the present application;

2 is a schematic flowchart of a method for patrolling and cleaning a robot according to an embodiment of the present application;

3 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application;

4 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application;

5 is a schematic diagram of the principle of a robot recognizing visual data according to an embodiment of the present application;

6 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a contamination detection network provided by an embodiment of the present application;

FIG. 8 is a schematic flowchart of a training method for a contamination detection network provided by an embodiment of the present application;

9 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application;

10 is a schematic structural diagram of a robot inspection and cleaning device provided in an embodiment of the application;

11 is a schematic structural diagram of another robot inspection and cleaning device according to an embodiment of the application;

12 is a schematic structural diagram of another robot inspection and cleaning device according to an embodiment of the application;

13 is a schematic structural diagram of another robot inspection and cleaning device provided by an embodiment of the application;

FIG. 14 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present application.

Detailed ways

The robot inspection and cleaning method provided in the embodiment of the present application can be applied to the robot as shown in FIG. 1 . As shown in FIG. 1 , the robot may include: a sensor 10 , a controller 11 and an execution component 12 . Among them, the sensor 10 includes a perception sensor and a positioning sensor installed on the robot body, and the sensor 10 is used to collect visual data within the field of view, which can be different types of cameras, lidar, infrared ranging, ultrasonic IMU (Inertial Measurement Unit, inertial measurement unit), odometer and other single or multiple sensors. The controller 11 may include a chip and a control circuit, mainly by receiving the visual data collected by the sensor 10, to actively identify the target objects (such as garbage and obstacles, etc.) existing in the field of view of the robot, and to execute based on the position and type of the target object. Patrol cleaning tasks. The execution component 12 includes a walking component and a cleaning component, and is configured to receive control instructions from the controller 11, navigate to the location of the target object according to the planned travel path, and perform cleaning operations.

The technical solutions in the embodiments of the present application will be described through the following embodiments in conjunction with the accompanying drawings.

The execution subject of the following method embodiments may be a robot inspection and cleaning device, and the device may be implemented as part or all of the above robot through software, hardware, or a combination of software and hardware. The following method embodiments are described by taking the execution subject being a robot as an example.

FIG. 2 is a schematic flowchart of a method for patrolling and cleaning a robot according to an embodiment of the present application. This embodiment relates to the process of how the robot performs inspection and cleaning of the workspace. As shown in Figure 2, the method may include:

S201 , collecting visual data within the field of view of the robot.

In order to automate the cleaning work, the area to be cleaned can be inspected and cleaned by the robot. The to-be-cleaned area refers to an area where the robot needs to perform inspection and cleaning, which may correspond to the environment where the robot is located. The robot can generate a field of vision path for covering the field of vision of the area to be cleaned through its own field of view and the electronic map of the area to be cleaned. Wherein, electronic maps include but are not limited to grid maps, topological maps and vector maps. The robot drives according to the vision path, and actively collects visual data within the field of view during the driving process, and actively identifies and cleans the target objects in the visual data, so as to realize active inspection of the area to be cleaned.

A vision sensor is arranged on the robot, so that the robot can collect data from an area within its field of view through the vision sensor, thereby obtaining visual data within the field of view of the vision sensor. When the sensor types corresponding to the visual sensors used for collecting visual data are different, the types of visual data collected by the visual sensors are also different. The above-mentioned visual data may be image data, video data, or point cloud data. For example, the above-mentioned visual sensor may be a camera. The robot can continuously shoot the area within its field of view through the camera to obtain surveillance video, and use the surveillance video as visual data to be identified. The robot can also directly photograph the area within the field of view through the camera to obtain the photographed image, and use the photographed image as the visual data to be recognized.

S202. Identify the visual data through a preset network to obtain an object position and an object type of the target object.

The preset network is obtained by training the original sample data and the sample labeling data corresponding to the original sample data.

After obtaining the visual data within the field of view, the robot inputs the visual data to the preset network, identifies the target object in the visual data through the preset network, and outputs the object position and object type of the target object. The target object may be garbage and/or obstacles. When the target object is garbage, the object type may include various types of garbage, such as plastic bags, napkins, paper scraps, fruit peels, and vegetable leaves. The object type may also include the results of classifying various types of garbage based on garbage classification criteria, such as recyclable garbage, kitchen waste, hazardous garbage, and other garbage. When the target object is an obstacle, the object types may include large-sized obstacles, small-sized obstacles, dynamic obstacles, static obstacles, and semi-static obstacles.

The training data of the preset network may be an original sample data set collected according to actual training requirements, or may be an original sample data set in a training database.

S203. Control the robot to perform an inspection and cleaning task according to the object position and object type of the target object.

After determining that there is a target object in the field of view, the robot can perform targeted inspection and cleaning tasks based on the object position and object type of the target object.

The inspection and cleaning method for a robot provided by the embodiment of the present application collects visual data within the field of view of the robot, identifies the visual data through a preset network, obtains the object position and object type of the target object, and determines the target object according to the target object. The object position and object type control the robot to perform patrol cleaning tasks. During the inspection and cleaning process, the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the preset network, so that the robot only needs to focus on the target in the workspace. Objects, and perform inspection and cleaning tasks based on the location and type of the target object, eliminating the need for full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot.

In one embodiment, the preset network is a pre-trained neural network, the original sample data is visual sample data, and the sample annotation data corresponding to the original sample data is the visual sample object position and type of the sample object that have been marked. sample. FIG. 3 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application. As shown in Figure 3, the method may include:

S301 , collect visual data within the field of vision of the robot.

S302. Identify the visual data through a pre-trained neural network to obtain the object position and object type of the target object.

The pre-trained neural network is obtained by training the visual sample data and the visual sample data marked with the position of the sample object and the type of the sample object.

A pre-trained neural network can be a network model that is pre-established and configured in the robot after training to recognize the target object in the visual data and output the object position and object type of the target object. The above pre-trained neural networks can be based on You Only Look Once (YOLO), RetinaNet, Single Shot MultiBox Detector (SSD) or faster regional convolutional neural networks (Faster- Region Convolutional Neural Networks, Faster-RCNN) and other networks.

After obtaining the visual data within the field of view, the robot inputs the visual data to the pre-trained neural network, identifies the target object in the visual data through the pre-trained neural network, and outputs the object position and object type of the target object.

The training data of the above-mentioned pre-trained neural network may be a visual sample data set collected according to actual training requirements, or may be a visual sample data set in a training database. Wherein, the visual sample data set includes visual sample data to be identified, and visual sample data marked with the position of the sample object and the type of the sample object. In this embodiment, the sample objects may include garbage and obstacles on the ground, and the garbage may include plastic bags, napkins, paper scraps, fruit peels, and the like. After obtaining the visual sample data set for training, the visual sample data is used as the input of the pre-trained neural network, and the sample object position and sample object type existing in the visual sample data are used as the expected output of the pre-trained neural network, and the corresponding The loss function trains the pre-trained neural network until the convergence condition of the loss function is reached, thereby obtaining the above-mentioned pre-trained neural network.

Taking YOLOv3 as the basic network used in the pre-training neural network as an example, first, the selected visual sample data set can be clustered to obtain reference frames (anchors) with different aspect ratios and different sizes. For example, taking a common garbage dataset as an example, k-means clustering operation is performed on the garbage dataset to learn reference frames with different aspect ratios and different sizes from the garbage dataset.

Taking the visual data collected by the robot as the image to be detected as an example, the robot can input the image to be detected into the pre-trained neural network. The pre-trained neural network can extract the feature map corresponding to the image to be detected through the darknet sub-network, and for each grid on the feature map, predict the description information of the above reference frames with different aspect ratios and different sizes, wherein the description information includes The confidence level of the reference frame, the location information of the reference frame, and the category information of the reference frame. Next, based on the confidence of the reference frame and the category information of the reference frame, filter out the reference frame with lower probability, and perform non-maximum suppression processing on the remaining reference frame to obtain the final detection result, which is the visual data. The object position and object type of the target object.

S303. Control the robot to perform a patrol cleaning task according to the object position and the object type.

As an optional implementation manner, when the target object is garbage and/or dirt, the above S303 may include:

S3031. Select a target storage component and a target cleaning component according to the object type.

After the object type of the target object is determined, the target storage component and the target cleaning component can be respectively selected for the object type. The robot may be provided with recyclable garbage storage components, kitchen waste storage components, hazardous waste storage components and other garbage storage components. In this way, after obtaining the object type of the target object, the robot can select the target storage component from all the set storage components based on the object type. For example, when the object type of the target object obtained by the robot is vegetable leaves, the robot can select the kitchen waste storage component as the target storage component.

For target objects of different object types, the degree of pollution to the ground is inconsistent. Therefore, when the object type of the target cleaning object is obtained, the target cleaning component can be selected from all the set cleaning components based on the object type. Among them, the robot may be provided with a vacuuming assembly, a dry mopping assembly, a wet mopping assembly, a drying assembly, a water absorbing assembly, and the like. When the object type of the target object is obtained, the target cleaning component can be selected from all the set cleaning components based on the object type. For example, target objects such as vegetable leaves and fruit peels may leave stains on the ground. Therefore, after the robot cleans such target objects to the kitchen waste storage component, it needs to be wiped with the wet mop component, and then used to dry Dry components are dried, according to which the robot can select wet mopping components and drying components as target cleaning components.

S3032. Control the robot to navigate to the object position, and control the robot to clean the target object into the target storage assembly, and clean the cleaned area through the target cleaning assembly.

After selecting the target storage assembly and the target cleaning assembly, the robot can plan a cleaning route based on the object position of the target object, and control the robot to drive along the cleaning route to the target position, and then clean the target object into the target storage assembly at the target position. , and use the selected target cleaning component to clean the cleaned area based on the corresponding cleaning strategy.

As another optional implementation manner, when the target object is an obstacle, the above S303 may include: according to the object type, determining whether the robot can pass the target object; if the robot cannot pass the target object, generate an escape path according to the object position and the target navigation point, and control the robot to drive to the target navigation point according to the escape path.

The object types of the target object may include large-sized obstacles and small-sized obstacles. For small-sized obstacles, the unique chassis structure of the robot enables the robot to cross small-sized obstacles; for large-sized obstacles, due to the large size of the obstacles, it is difficult for the robot to go over the large-sized obstacles. OK, causing the robot to be trapped. Therefore, when the robot performs the inspection and cleaning task, the robot needs to determine whether the robot can pass the target object according to the object type of the identified target object. That is, when the identified target object is a large-sized obstacle, the robot cannot move forward over the large-sized obstacle. At this time, the robot enters the escape mode to avoid the large-sized obstacle. In order to continue to perform the inspection and cleaning task, the robot selects a path point from the initial cleaning path as the target navigation point, and generates an escape path based on the position information of the large-sized obstacle and the target navigation point, and controls the robot to drive according to the escape path to target navigation point.

The robot inspection and cleaning method provided by the embodiment of the present application collects visual data within the field of vision of the robot, identifies the visual data through a pre-trained neural network, obtains the object position and object type of the target object, and determines the object position and object type according to the object. The location and object type control the robot to perform patrol cleaning tasks. During the inspection and cleaning process, the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the pre-trained neural network, so that the robot only needs to focus on the objects in the workspace. Target objects, and perform inspection and cleaning tasks based on the location and type of the target objects, eliminating the need for full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot.

In one embodiment, there is also provided a process of identifying visual data within a field of view through a pretrained neural network. On the basis of the foregoing embodiment, optionally, the foregoing pre-trained neural network may include a feature extraction layer, a feature fusion layer, and an object recognition layer. As shown in Figure 4, the above S302 may include:

S401. Extract multi-scale feature data in the visual data through the feature extraction layer.

Robots can choose a deep learning network as this feature extraction layer. The feature extraction layer may be a darknet network or other network structures. The robot inputs the collected visual data into the pre-trained neural network, and extracts the features in the visual data through the feature extraction layer in the pre-trained neural network to obtain multi-scale feature data. Wherein, each scale feature data includes description information of a reference frame corresponding to each grid in the scale feature data. The description information includes the confidence level of the reference frame, the location information of the reference frame, and the category information of the reference frame. The above reference frame can be obtained by performing a clustering operation on the training data of the pre-trained neural network.

To improve the recognition speed of the pretrained neural network, the number of feature extraction blocks in the feature extraction layer can be reduced. Optionally, the feature extraction layer includes two feature extraction blocks, which are a first feature extraction block and a second feature extraction block, respectively. Optionally, the process of the above S401 may be: extracting the first scale feature data in the visual data through the first feature extraction block, and extracting the second scale feature data in the visual data through the second feature extraction block. Among them, the first scale feature data and the second scale feature data can be arbitrarily selected from 13*13 scale feature data, 26*26 scale feature data and 52*52 scale feature data for combination.

In practical applications, in order to improve the recognition speed of the pre-trained neural network, optionally, the first scale feature data may be 13*13 scale feature data, and the second scale feature data may be 26*26 scale feature data.

S402. Perform feature fusion on the multi-scale feature data through the feature fusion layer to obtain fused feature data.

In order to improve the recognition ability of the pre-trained neural network for small targets, the feature extraction layer inputs the extracted multi-scale feature data to the feature fusion layer, and the multi-scale feature data is feature-fused through the feature fusion layer. Optionally, when the feature data extracted from the visual data through the feature extraction layer is 13*13 scale feature data and 26*26 scale feature data, the robot uses the feature fusion layer to combine the 13*13 scale feature data and 26*26 scale feature data. Feature fusion is performed on the scale feature data to obtain the fused feature data.

S403. Determine the object position and object type of the target object through the object recognition layer according to the multi-scale feature data and the fused feature data.

The feature extraction layer inputs the extracted multi-scale feature data to the object recognition layer, and the feature fusion layer also inputs the fused feature data to the object recognition layer, and the multi-scale feature data and the fused feature data are processed by the object recognition layer. Process to obtain the object position and object type of the target object. The object recognition layer can perform coordinate transformation and coordinate scaling on the reference frame in the multi-scale feature data and the fused feature data, and restore the reference frame in the multi-scale feature data and the fused feature data to the original data. The restored frame of reference. Next, non-maximum suppression processing is performed on the restored reference frame, redundant reference frames are filtered out, and description information of the filtered reference frame is output, so as to obtain the object position and object type of the target object.

Taking the pre-trained neural network shown in FIG. 5 as an example, the above-mentioned recognition process of visual data through the pre-trained neural network is introduced. The robot inputs the collected visual data within the visual field to the feature extraction layer 501 in the pre-trained neural network, and extracts the features in the visual data through the first feature extraction block 5011 in the feature extraction layer 501 to obtain 13*13 scale feature data , and extract the features in the visual data through the second feature extraction block 5012 in the feature extraction layer to obtain 26*26 scale feature data. Next, the robot inputs the 13*13 scale feature data and the 26*26 scale feature data into the feature fusion layer 502 in the pre-trained neural network, and the feature fusion layer 502 performs the 13*13 scale feature data and the 26*26 scale feature data. Feature fusion to obtain the fused feature data. The robot inputs the 13*13 scale feature data, 26*26 scale feature data and the fused feature data into the object recognition layer 503 in the pre-trained neural network, and the 13*13 scale feature data, 26*26 The scale feature data and the fused feature data are processed by coordinate transformation, coordinate scaling and non-maximum suppression, so as to identify the target object in the visual data, and output the object position and object type of the target object.

In this embodiment, the pre-trained neural network can perform feature fusion on the multi-scale feature data in the visual data, and recognize the target object based on the fused feature data and the multi-scale feature data, thereby improving the robot recognition effect. At the same time, the feature extraction layer in the pre-trained neural network only includes two feature extraction blocks. Compared with the feature extraction layer including three feature extraction blocks, the feature extraction layer is reduced on the premise that the recognition effect of the robot can be satisfied. The number of feature extraction blocks in the middle, thereby improving the recognition speed of the robot.

In practical applications, the robot usually collects visual data within the field of view through a camera. At this time, the object position of the target object recognized by the robot through the pre-trained neural network is calculated in the image coordinate system. In view of this situation, that is, when the visual data is collected based on the image coordinate system of the robot, on the basis of the foregoing embodiment, optionally, before the foregoing S303, the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, Transform the object position.

Optionally, the obtaining of the first correspondence between the image coordinate system of the robot and the radar coordinate system may include: respectively obtaining the first correspondence between the robot's pixel coordinate system and the radar coordinate system for the same object to be collected. data and second data; match the first data and the second data to obtain multiple sets of matched feature points; determine the image coordinate system and radar coordinates of the robot according to the multiple sets of matched feature points The first correspondence between the systems.

The object to be collected can be set on a corner in advance. The robot is provided with a camera and a laser radar, and the robot controls the camera and the laser radar to collect data from different angles of the object to be collected set on the corner of the wall, thereby obtaining the first data and the second data. Next, the feature points in the first data and the second data are respectively detected, and the feature points in the first data and the second data are matched to obtain multiple sets of matched feature points. Usually, four sets of matching feature points or more need to be determined. Then, through multiple sets of matching specific points, the corresponding equations are established, and the correspondence between the robot's image coordinate system and the radar coordinate system can be obtained by solving the equations.

In this embodiment, the robot converts the obtained object position through the first correspondence between the robot's image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system, so that the final obtained object position is obtained. The actual position of the target object is more accurate, and then the robot is controlled to perform inspection and cleaning tasks based on the accurate object position, which improves the cleaning accuracy and cleaning efficiency of the robot.

In one embodiment, the target object is dirt, and the preset network is a preset dirt detection network; the dirt detection network is obtained by training a visual semantic segmentation dataset and a dirt dataset, wherein, The dirty data set includes the original sample data and sample annotation data corresponding to the original sample data. FIG. 6 is a schematic flowchart of another method for patrolling and cleaning a robot according to an embodiment of the present application. As shown in Figure 6, the method may include:

S601. Collect visual data within the field of view of the robot.

S602. Detect the visual data through a preset contamination detection network to obtain a target contamination location and a target contamination type.

The dirty detection network is obtained by training a visual semantic segmentation data set and a dirty data set, and the dirty data set includes original sample data and sample labeling data corresponding to the original sample data.

The above-mentioned contamination detection network is a deep learning model, which can be a network model that is pre-established and configured in the robot after training on the visual semantic segmentation dataset and the contamination dataset, so as to detect the target contamination existing in the visual data. Wherein, the above-mentioned visual semantic segmentation data set may be the Cityscapes data set; the above-mentioned sample labeling data refers to the original sample data for which the dirty position of the sample and the dirty type of the sample have been marked. The above-mentioned contamination detection network can be established based on a weighted convolutional harmonic dense network (Fully Convolutional Harmonic Dense Net, FCHarDNet), U-Net, V-Net, and Pyramid Scene Parsing Net (PSPNet) and other networks.

After obtaining the visual data within the field of view, the robot inputs the visual data into the trained dirt detection network, detects the target dirt in the visual data through the dirt detection network, and outputs the target dirt position and target dirt. type of contamination. The target soiling types may include liquid soiling and solid soiling. Optionally, the robot can extract dirt features in the visual data through a dirt detection network, and determine the target dirt type according to the dirt features. Among them, the soil characteristics may include soil particle size and soil transparency (soil transparency refers to the light transmittance of soil). When the contamination feature satisfies the preset condition, the target contamination type is determined to be liquid contamination; otherwise, the target contamination type is determined to be solid contamination. The above preset conditions include: the dirt particles are larger than the preset particle size, and the dirt transparency is larger than the preset transparency.

As an optional implementation manner, the above-mentioned contamination detection network may include a downsampling layer and a deconvolution layer. In view of this situation, on the basis of the foregoing embodiment, optionally, the foregoing S602 may include: performing a hierarchical downsampling operation on the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map; The deconvolution layer performs a hierarchical deconvolution operation on the multi-resolution intermediate feature map to obtain the target dirt position and target dirt type in the visual data.

The network structure of the above-mentioned contamination detection network can be shown in Figure 7. As can be seen from Figure 7, the contamination detection network includes N downsampling layers and N deconvolution layers. The input layer in Figure 7 is used to input visual data, and the output layer is used to output the target dirty position and target dirty type in the visual data. First, the robot inputs the collected visual data within the field of view into the input layer, and performs hierarchical down-sampling operations on the visual data through N down-sampling layers to extract the dirty features in the visual data and obtain different resolutions. feature map. Next, perform hierarchical deconvolution operations on the feature maps of different resolutions through N deconvolution layers until the last deconvolution layer is processed, so as to output the target dirty position and the target dirty position in the visual data through the output layer. Target soiling type.

In practical applications, in order to better focus the attention of the dirty detection network to the dirty area. Optionally, the contamination detection network may further include an attention threshold block. The above process of performing a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer may be: enhancing and suppressing the multi-resolution intermediate feature map layer by layer through an attention threshold block , and perform a deconvolution operation.

FIG. 7 only shows that the number N of downsampling layers and deconvolution layers included in the contamination detection network is 4 as an example. This embodiment does not limit the downsampling layers and deconvolution layers included in the contamination detection network. The number N of downsampling layers and deconvolution layers included in the dirt detection network can be set correspondingly according to the actual application requirements.

The upsampling operation of the multi-resolution intermediate feature map is realized through the deconvolution layer, and only the intermediate feature map and the convolution kernel in the deconvolution layer need to be deconvolved. Compared with using bilinear For the interpolated upsampling layer, the time of contamination detection is greatly shortened, and the efficiency of contamination detection is improved.

S603. Control the robot to perform a patrol cleaning task according to the target dirt position and target dirt type.

After determining that the target is dirty within the field of view, the robot navigates to the target dirty position, and cleans the target dirty position in a targeted manner based on the target dirt type.

Optionally, the process of the above S603 may be: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the target dirt location, and using the target cleaning strategy to clean the target dirt location .

The robot can generate a target cleaning strategy for cleaning according to the obtained target dirt type. When the target soiling type is liquid soiling, since it is liquid, the liquid can be sucked dry first, and then wiped with a dry mop. According to this, the target cleaning strategy generated by the robot can be used to first use the water absorbing component to absorb the liquid. , and then use the dry mop component to wipe the ground. When the target dirt type is solid dirt, since it is solid, the solid can be cleaned and then wiped with a wet mop. According to this, the target cleaning strategy generated by the robot can be to use the vacuum component to clean the solid first, and then use the vacuum cleaner to clean the solid. Use the wet mopping component to wipe the floor, and then use the drying component to dry the floor. In the process of generating the target cleaning strategy, the material of the ground can also be combined. For example, when the material of the ground is floor and floor tiles, you can use the vacuuming assembly to vacuum, and then use the mopping assembly to mop the floor after vacuuming; when the material of the ground is carpet, you can use the vacuuming assembly only Do vacuuming.

After obtaining the target cleaning strategy, the robot navigates to the target dirty position, and uses the generated target cleaning strategy to clean the target dirty position. After cleaning the target dirty position, the robot can continue to collect visual data within its own field of view at the target dirty position to identify the target dirt that needs to be cleaned in the next step, that is, repeat the above process of S601-S603, so as to complete the process of cleaning. Inspection and cleaning of the entire work space. The robot can also return to the target navigation point in the field of view path, take the target navigation point as the starting point for cleaning, and continue to collect visual data within the field of view.

The robot inspection and cleaning method provided by the embodiment of the present application collects the visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, obtains the target contamination position and the target contamination type, and Control the robot to perform patrol cleaning tasks according to the target dirty position and target dirty type. During the inspection and cleaning process, the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot.

In an embodiment, an acquisition process of a contamination detection network is also provided, that is, how to train a contamination detection network. On the basis of the foregoing embodiment, optionally, as shown in FIG. 8 , the training process of the contamination detection network may include:

S801. Pre-train a detection network by using the visual semantic segmentation data set to obtain an initial contamination detection network.

Although the contamination identification parallel data (that is, the contamination data set) can improve the detection performance of the contamination detection network, due to the lack of the contamination identification parallel data, the training of the contamination detection network is very time-consuming and labor-intensive. Still not meeting expectations. However, there are many visual-semantic segmentation datasets. Therefore, the dirt detection network can be pre-trained through the visual-semantic segmentation dataset with a large number of samples, and the initial dirt detection network trained on the visual-semantic segmentation dataset can be obtained. Optionally, the visual semantic segmentation dataset can be the Cityscapes dataset. Through this pre-training method, the long and slow learning phase in the early stage of network training can be avoided, thereby greatly reducing the network training time. At the same time, a lot of tedious hyperparameter tuning can be avoided.

S802. Use the original sample data as the input of the initial contamination detection network, use the sample labeling data as the expected output of the initial contamination detection network, and use a preset loss function to continue to detect the initial contamination The detection network is trained to obtain the dirty detection network.

After pre-training the contamination detection network with the visual semantic segmentation data set to obtain the initial contamination detection network, you can continue to use the collected contamination data set to fine-tune the initial contamination detection network for training. The original sample data in the dirty data set is used as the input of the initial dirty detection network, the sample annotation data in the dirty data set is taken as the expected output of the initial dirty detection network, and the preset loss function is used for the initial dirty detection network. The parameters are trained and adjusted until the convergence condition of the loss function is reached, so as to obtain a trained dirty detection network. Optionally, the loss function can be a cross-entropy loss function.

In order to make more data in the dirty dataset used in the network training process, data augmentation can be performed on the dirty dataset. To this end, on the basis of the above embodiment, optionally, before continuing to train the initial contamination detection network by using a preset loss function, the method further includes: performing data enhancement processing on the contamination data set. The method of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.

The dirty data set can be expanded by horizontal flip mirroring; the dirty data set can also be cropped to realize the data expansion of the image segmentation data set, that is, a position is randomly selected as the cropping center, and each dirty data is processed. Cropping; each dirty data can also be color jittered for data augmentation of dirty datasets.

In this embodiment, in the process of training the contamination detection network, a visual semantic segmentation data set with a large number of samples can be used to pre-train the contamination detection network, and then the contamination data set is used for pre-training to obtain The initial dirty detection network is fine-tuned for training. Through this pre-training method, the long and slow learning phase in the early stage of network training can be avoided, thereby greatly reducing the network training time. At the same time, a lot of tedious hyperparameter tuning can be avoided. That is to say, the technical solutions adopted in the embodiments of the present application shorten the training time of the contamination detection network and improve the accuracy of the contamination detection network.

In practical applications, the robot usually collects visual data within the field of view through a camera. At this time, the target dirty position detected by the robot through the trained dirt detection network is calculated in the image coordinate system. In view of this situation, that is, when the visual data is collected based on the image coordinate system of the robot, on the basis of the above-mentioned embodiment, optionally, before the above-mentioned S603, the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, The target dirty location is converted.

After acquiring the first correspondence between the robot's image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system, the robot performs projection transformation on the target dirty position according to the first correspondence , and then convert the dirty position after projection transformation based on the second correspondence, so as to obtain the actual position of the target dirt in the world coordinate system. The operation steps of obtaining the first corresponding relationship and the second corresponding relationship, and converting the target dirty position according to the first corresponding relationship and the second corresponding relationship and the resulting effects have been recorded in the above-mentioned embodiments, and are not described here. Repeat.

In one embodiment, the number of the target objects is at least one; the method further includes: performing path planning on at least one object position to obtain a target cleaning path; The robot performing the inspection and cleaning task includes: controlling the robot to navigate to the at least one object position in sequence according to the target cleaning path, and performing the inspection and cleaning task based on the object type corresponding to the current object position. FIG. 9 is a schematic flowchart of another method for patrolling and cleaning a robot according to an embodiment of the present application. As shown in Figure 9, the method may include:

S901. Collect visual data within the field of view of the robot.

S902. Detect the visual data through a preset contamination detection network to obtain at least one target contamination location and a target contamination type corresponding to the at least one target contamination location.

The preset contamination detection network and the target contamination type in this embodiment have been recorded in the above-mentioned embodiments, and will not be repeated here.

In this embodiment, when the number of target objects is multiple, each target object corresponds to an object position and an object type, and the multiple object types corresponding to the multiple target objects may be identical or completely different, or Parts are the same, which is not limited.

Optionally, the above-mentioned contamination detection network may include a downsampling layer and a deconvolution layer. In view of this situation, on the basis of the foregoing embodiment, optionally, the foregoing S902 may include: performing a hierarchical downsampling operation on the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map; The deconvolution layer performs a hierarchical deconvolution operation on the multi-resolution intermediate feature map to obtain at least one target dirty position in the visual data and at least one target dirty position corresponding to the target dirty position. type.

The network structure, working principle, and effects of the above-mentioned contamination detection network have been recorded in the above-mentioned embodiments, and will not be repeated here.

S903. Perform shortest path planning on the at least one target dirty location to obtain a target cleaning path.

The target cleaning path refers to the cleaning path with the shortest distance among all the cleaning paths for the robot to reach at least one target dirty position. When at least one target dirty position is detected, the robot can generate at least one target through the shortest path planning algorithm according to the at least one target dirty position, the historical obstacle map of the area to be cleaned, and the current obstacle map of the area to be cleaned. Target cleaning path for dirty locations. The shortest path planning algorithm may be Dijkstra algorithm, Floyd algorithm, and ant colony algorithm.

S904 , controlling the robot to navigate to the corresponding target dirty position in sequence according to the target cleaning path, and perform the inspection and cleaning task based on the corresponding target dirt type.

After the target cleaning path is planned, the robot can navigate to the corresponding target dirty position in sequence according to the target cleaning path, and clean the target dirty position in a targeted manner based on the target dirt type corresponding to the target dirty position.

Optionally, the process of the above S904 may be: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the corresponding target dirt position in sequence according to the target cleaning path, and adopting the target cleaning method. The strategy cleans the target dirty location.

The robot can generate a target cleaning strategy for cleaning according to the obtained target dirt type. When the target soiling type is liquid soiling, since it is liquid, the liquid can be sucked dry first, and then wiped with a dry mop. According to this, the target cleaning strategy generated by the robot can be first used to absorb the liquid with a water absorbing component. , and then use the dry mop component to wipe the ground. When the target dirt type is solid dirt, since it is solid, the solid can be cleaned and then wiped with a wet mop. According to this, the target cleaning strategy generated by the robot can be: Use the wet mopping component to wipe the floor, and then use the drying component to dry the floor. In the process of generating the target cleaning strategy, the material of the ground can also be combined. For example, when the material of the floor is floor and floor tiles, you can use the vacuum cleaner for vacuuming, and then use the mopping unit to mop the floor after vacuuming; when the material of the floor is carpet, you can use the vacuum cleaner only Do vacuuming.

After obtaining the target cleaning strategy, the robot navigates to the target dirty location in sequence according to the target cleaning path, and uses the generated target cleaning strategy to clean at least one target dirty location. After cleaning the last target dirty position, the robot can control itself to rotate, and collect visual data within the visual range during the rotation to identify the target dirt that needs to be cleaned in the next step, that is, repeatedly execute the above S901-S904 process, so as to complete the inspection and cleaning of the entire workspace.

The inspection and cleaning method for a robot provided by the embodiment of the present application collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains at least one target dirty position and at least one target dirty location. According to the target dirt type corresponding to the dirty position, the shortest path planning is performed on at least one target dirty position to obtain the target cleaning path, and the robot is controlled to navigate to the corresponding target dirty position in turn according to the target cleaning path, and based on the corresponding target dirty position Type to perform patrol cleaning tasks. During the inspection and cleaning process, the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot. At the same time, when at least one target dirty position is detected, the robot can also perform shortest path planning for the at least one target dirty position, so that the robot can navigate to the at least one target dirty position with the shortest path, which improves the cleaning efficiency of the robot.

The acquisition process of the contamination detection network in this embodiment has been recorded in the above-mentioned embodiments, and will not be repeated here.

In practical applications, in order to expand the data collection range of the robot and improve the efficiency of the robot's active inspection, on the basis of the above embodiment, optionally, the process of the above S901 may be: controlling the robot to rotate, and during the rotation process Collect the visual data within the field of view of the robot. Optionally, the way of controlling the robot to rotate may be: controlling the robot to rotate based on the field of view of at least one sensor.

After generating a field of vision path for covering the field of view of the area to be cleaned based on the robot's field of view and the electronic map of the area to be cleaned, the robot inspects and cleans the area to be cleaned according to the planned field of view path. In order to expand the data collection range of the robot, before the robot starts to travel, the robot can be controlled to rotate on the spot based on the field of view of at least one sensor, and the visual data within the field of view of the robot can be collected during the rotation process. In this way, with the rotation of the robot , the direction of the robot's field of vision is continuously adjusted, so that the robot can collect visual data in a wider range at the current position. It is also possible to control the robot to rotate based on the field of view of at least one sensor during the movement of the robot, and continuously collect visual data within the field of view of the robot during the rotation. In actual use, the rotation timing of the robot may be set, which is not limited in this embodiment.

In order to expand the data collection range of the robot, optionally, the above process of controlling the robot to rotate may be: controlling the robot to rotate once. That is, before the robot starts to travel, control the robot to rotate once in place, or control the robot to rotate once during the process of the robot, and collect the visual data within the robot's field of vision during the rotation process, so that the robot can perceive its own vision within a 360-degree range. Data collection greatly expands the data collection range of the robot, enabling the robot to actively identify visual data in a wider range, and perform overall cleaning of the identified target dirt, thereby improving the cleaning efficiency of the robot.

In practical applications, visual data in the robot workspace can be collected through vision sensors. Optionally, a first vision sensor and a second vision sensor are installed on the robot. The first visual sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line; the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located below the horizontal line, which is parallel to the horizontal line. intersect. In view of this situation, on the basis of the above-mentioned embodiment, optionally, the above-mentioned process of S901 may be: controlling the first visual sensor and the second visual sensor to rotate, and collecting visual data within their respective visual fields during the rotating process.

Since the line of sight of the first visual sensor is head-up, it can obtain a relatively large sensing range for sensing environmental information farther away in the area to be cleaned. Since the line of sight of the second visual sensor is downward and can be directly aimed at the ground, the second visual sensor can more clearly perceive the environmental information on the ground nearby, and can effectively make up for the blind spot of the first visual sensor. In view of this situation, during the inspection and cleaning process of the robot, the first vision sensor and the second vision sensor can be controlled to collect visual data within their respective fields of view, so that the robot can not only collect data within the far field of view, but also can The second vision sensor collects the data in the blind area of the first vision sensor, which greatly expands the data collection range of the robot.

It is also possible to control the first vision sensor and the second sensor to rotate, and collect visual data within their respective fields of view during the rotation process, so that with the rotation of the first vision sensor and the second vision sensor, the direction of the robot's field of vision is continuously adjusted. , so that the robot can collect visual data in a wider range and expand the data collection range of the robot. In practical applications, the rotation angles of the first vision sensor and the second vision sensor can be controlled according to actual requirements. Optionally, the rotation angle may be 360 degrees.

In this embodiment, the robot is controlled to rotate, and the visual data within the field of view of the robot is collected during the rotation, or the first vision sensor and the second vision sensor of the robot are controlled to rotate, and the respective visual data are collected during the rotation. Visual data within the field of view. Through this technical solution, the data collection range of the robot is greatly expanded, so that the robot can actively identify visual data in a wider range, and perform overall cleaning on the identified target dirt, thereby improving the cleaning efficiency of the robot.

In practical applications, the robot usually collects visual data within the field of view through a camera. At this time, the target dirty position detected by the robot through the trained dirt detection network is calculated in the image coordinate system. In view of this situation, that is, in the case where the visual data is collected based on the image coordinate system of the robot, on the basis of the above embodiment, optionally, before the above S903, the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, Translating the at least one target dirty location. The operation steps of obtaining the first corresponding relationship and the second corresponding relationship, and converting the target dirty position according to the first corresponding relationship and the second corresponding relationship and the resulting effects have been recorded in the above-mentioned embodiments, and are not described here. Repeat.

FIG. 10 is a schematic structural diagram of a robot inspection and cleaning device according to an embodiment of the present application. As shown in FIG. 10 , the apparatus may include: a collection module 100 , an identification module 101 and a control module 102 .

The acquisition module 100 is set to collect visual data within the field of vision of the robot; the recognition module 101 is set to detect the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network passes the original The sample data and the sample labeling data corresponding to the original sample data are obtained by training; the control module 102 is configured to control the robot to perform an inspection and cleaning task according to the object position and object type of the target object.

On the basis of the above-mentioned embodiment, optionally, the preset network is a pre-trained neural network, the original sample data is visual sample data, and the sample labeling data corresponding to the original sample data is the marked sample object position and The visual sample data of the sample object type.

The robot inspection and cleaning device provided by an optional embodiment of the present application collects visual data within the field of vision of the robot, identifies the visual data through a pre-trained neural network, and obtains the object position and object type of the target object, and according to the The object position and object type control the robot to perform patrol cleaning tasks. During the inspection and cleaning process, the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the pre-trained neural network, so that the robot only needs to focus on the objects in the workspace. Target objects, and perform inspection and cleaning tasks based on the location and type of the target objects, eliminating the need for full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot.

As shown in FIG. 11 , on the basis of the above embodiment, optionally, the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer; the identification module 101 includes: a feature extraction unit 1011 , a feature fusion unit 1012 and the identification unit 1013; the feature extraction unit 1011 is set to extract the multi-scale feature data in the visual data through the feature extraction layer; the feature fusion unit 1012 is set to perform the multi-scale feature data through the feature fusion layer. Feature fusion to obtain fused feature data; the identifying unit 1013 is configured to determine the object position and object type of the target object through the object recognition layer according to the multi-scale feature data and the fused feature data.

On the basis of the above embodiment, optionally, the feature extraction layer includes a first feature extraction block and a second feature extraction block; the feature extraction unit 1011 is configured to extract the visual data through the first feature extraction block feature data of the first scale, and extract the second scale feature data in the visual data through the second feature extraction block.

Optionally, the first scale feature data is 13*13 scale feature data, and the second scale feature data is 26*26 scale feature data.

On the basis of the above embodiment, optionally, when the target object is garbage and/or dirt, the control module 102 is configured to select a target storage assembly and a target cleaning assembly according to the type of the object; control the robot Navigating to the object position, and controlling the robot to clean the target object into the target storage assembly, and to clean the cleaned area through the target cleaning assembly.

On the basis of the above embodiment, optionally, when the target object is an obstacle, the control module 102 is configured to determine whether the robot can pass the target object according to the object type; When the target object cannot be crossed, an escape path is generated according to the position of the object and the target navigation point, and the robot is controlled to travel to the target navigation point according to the escape path.

As shown in FIG. 11 , on the basis of the foregoing embodiment, optionally, in the case that the visual data is collected based on the image coordinate system of the robot, the apparatus further includes: an acquisition module 103 and Conversion module 104 .

The acquisition module 103 is configured to acquire the first correspondence between the image coordinate system and the radar coordinate system of the robot before the control module 102 controls the robot to perform the inspection and cleaning task according to the object position and the object type. The second corresponding relationship between the radar coordinate system and the world coordinate system; the conversion module 104 is configured to convert the position of the object according to the corresponding relationship.

On the basis of the above embodiment, optionally, the target object is dirt, and the preset network is a preset dirt detection network; the dirt detection network segments the dataset and the dirt data by visual semantics The dirty data set includes the original sample data and sample labeling data corresponding to the original sample data.

The inspection and cleaning device for a robot provided by an optional embodiment of the present application collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains the target contamination location and target contamination type , and control the robot to perform inspection and cleaning tasks according to the target dirty position and target dirty type. During the inspection and cleaning process, the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot.

On the basis of the above embodiment, optionally, the contamination detection network includes a downsampling layer and a deconvolution layer; the detection module 101 is configured to perform hierarchical downsampling on the visual data through the downsampling layer operation to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the target dirty position and target in the visual data. Dirt type.

As shown in FIG. 12 , on the basis of the above-mentioned embodiment, optionally, the apparatus further includes: a network training module 105; Pre-training to obtain an initial contamination detection network; using the original sample data as the input of the initial contamination detection network, using the sample labeling data as the expected output of the initial contamination detection network, and using a preset loss The function continues to train the initial dirty detection network.

On the basis of the above-mentioned embodiment, optionally, the apparatus further includes: a training data processing module 106; the training data processing module 106 is configured to use a preset loss function in the network training module 105 to continue the initial contamination detection network Data augmentation is performed on the dirty dataset before training.

On the basis of the foregoing embodiment, optionally, the manner of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.

On the basis of the above embodiment, optionally, the control module 102 is configured to generate a target cleaning strategy according to the target dirt type; control the robot to navigate to the target dirt position, and use the target cleaning strategy to Target dirty locations for cleaning.

As shown in FIG. 13 , on the basis of the foregoing embodiment, optionally, the number of the target objects is at least one; the apparatus further includes a path planning module 107 , and the path planning module 107 is configured to target at least one target object. The target cleaning path is obtained by performing path planning on the object position; the control module 102 is configured to: control the robot to navigate to the at least one object position in sequence according to the target cleaning path, and perform patrolling based on the object type corresponding to the current object position Check cleaning tasks.

The inspection and cleaning device for a robot provided by an optional embodiment of the present application collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains at least one target dirty position and at least one The target dirt type corresponding to the target dirty position, perform shortest path planning for at least one target dirty position, obtain the target cleaning path, and control the robot to navigate to the corresponding target dirty position in sequence according to the target cleaning path, and based on the corresponding target Dirty types perform patrol cleaning tasks. During the inspection and cleaning process, the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focusing on the target dirt in the workspace, and performing targeted inspection and cleaning tasks based on the location and type of target dirt, there is no need to perform full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot. At the same time, when at least one target dirty position is detected, the robot can also perform shortest path planning for the at least one target dirty position, so that the robot can navigate to multiple target dirty positions with the shortest path, which improves the cleaning efficiency of the robot.

On the basis of the above embodiment, optionally, the acquisition module 100 is configured to control the robot to rotate, and collect visual data within the field of view of the robot during the rotation.

On the basis of the foregoing embodiment, optionally, the acquisition module 100 is configured to control the rotation of the robot based on the field of view of at least one sensor.

On the basis of the above embodiment, optionally, the acquisition module 100 is configured to control the first visual sensor and the second visual sensor to rotate, and collect visual data within their respective fields of view during the rotation; The vision sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line, the second visual sensor is the downward viewing sensor of the robot, and the central axis of the viewing angle is located below the horizontal line and intersects the horizontal line.

In one embodiment, a robot is provided, the schematic diagram of which can be shown in FIG. 1 . The robot may include: one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, The program includes instructions for executing the robot inspection and cleaning method described in any of the above embodiments.

The above-mentioned one or more processors implement the following steps when executing the program:

Collect the visual data within the field of vision of the robot; identify the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network passes the original sample data and the corresponding data of the original sample data. The sample labeling data is obtained by training; the robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.

In one embodiment, the preset network is a pre-trained neural network, the original sample data is visual sample data, and the sample labeling data corresponding to the original sample data is the labeled sample object position and sample object type. Visual sample data.

In one embodiment, the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer; when the one or more processors execute the program, the following steps are further implemented: extracting the multi-scale feature data in the visual data; feature fusion is performed on the multi-scale feature data through the feature fusion layer to obtain the fused feature data; according to the multi-scale feature data and the fused feature data, The object position and object type of the target object are determined by the object recognition layer.

In one embodiment, the feature extraction layer includes a first feature extraction block and a second feature extraction block; when the one or more processors execute the program, the following steps are further implemented: extracting through the first feature extraction block feature data of the first scale in the visual data, and extract the feature data of the second scale in the visual data through the second feature extraction block.

In one embodiment, when the target object is garbage and/or dirty, the one or more processors further implement the following steps when executing the program: selecting a target storage component and a target cleaning according to the type of the object assembly; controlling the robot to navigate to the object position, and controlling the robot to clean the target object into the target storage assembly, and to clean the cleaned area through the target cleaning assembly.

In one embodiment, when the target object is an obstacle, the one or more processors further implement the following step when executing the program: determining whether the robot can pass the target object according to the type of the object ; if not, generate an escape path according to the object position and the target navigation point, and control the robot to drive to the target navigation point according to the escape path.

In one embodiment, when the visual data is collected based on the image coordinate system of the robot, the one or more processors further implement the following steps when executing the program: acquiring the robot's image coordinate system. The first correspondence between the image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, the position of the object is to convert.

In one embodiment, the target object is dirt, and the preset network is a preset dirt detection network; the dirt detection network is obtained by training a visual semantic segmentation dataset and a dirt dataset, wherein , the dirty data set includes the original sample data and sample labeling data corresponding to the original sample data.

In one embodiment, the contamination detection network includes a downsampling layer and a deconvolution layer; when the one or more processors execute the program, the following steps are further implemented: the visual data is processed by the downsampling layer. Perform a hierarchical downsampling operation to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the visual data. Target soiling location and target soiling type.

In one embodiment, the one or more processors further implement the following steps when executing the program: pre-training the contamination detection network by using the visual semantic segmentation data set to obtain an initial contamination detection network; The original sample data is used as the input of the initial contamination detection network, the sample annotation data is used as the expected output of the initial contamination detection network, and the initial contamination detection network is continued to be performed using a preset loss function. train.

In one embodiment, the one or more processors further implement the following step when executing the program: performing data enhancement processing on the dirty data set.

Optionally, the manner of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.

In one embodiment, when the one or more processors execute the program, the following steps are further implemented: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the target dirt position, using the The target cleaning strategy cleans the target soiled locations.

In one embodiment, the number of the target object is at least one; when the one or more processors execute the program, the following steps are further implemented: performing path planning on at least one object position to obtain a target cleaning path; controlling the The robot navigates to the at least one object position in sequence according to the target cleaning path, and performs the inspection and cleaning task based on the object type corresponding to the current object position.

In one embodiment, when the one or more processors execute the program, the following steps are further implemented: controlling the robot to rotate, and collecting visual data within the field of view of the robot during the rotation.

In one embodiment, when the one or more processors execute the program, the following steps are further implemented: controlling the rotation of the robot based on the field of view of the at least one sensor.

In one embodiment, the one or more processors further implement the following steps when executing the program: controlling the first vision sensor and the second vision sensor to rotate, and collecting visual data within their respective fields of view during the rotation; Wherein, the first visual sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line, the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located on the horizontal line Below, intersects the horizontal line.

In one embodiment, as shown in FIG. 14 , a non-transitory computer-readable storage medium 140 containing computer-executable instructions 1401 is provided, when the computer-executable instructions are executed by one or more processors 141 , causing the processor 141 to perform the following steps:

In one embodiment, the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer; when the computer-executable instructions are executed by the processor, the following steps are further implemented: extracting the visual data through the feature extraction layer The multi-scale feature data in the feature fusion layer; the feature fusion is performed on the multi-scale feature data through the feature fusion layer to obtain the fused feature data; according to the multi-scale feature data and the fused feature data, through the The object recognition layer determines the object location and object type of the target object.

In one embodiment, the feature extraction layer includes a first feature extraction block and a second feature extraction block; when the computer-executable instructions are executed by the processor, the following steps are further implemented: extracting the visual image by using the first feature extraction block feature data of the first scale in the data, and extract the feature data of the second scale in the visual data through the second feature extraction block.

In one embodiment, when the target object is garbage and/or dirty, the computer-executable instructions, when executed by the processor, further implement the following steps: selecting a target storage component and a target cleaning component according to the object type; controlling The robot navigates to the object position, and controls the robot to clean the target object into the target storage assembly, and clean the cleaned area through the target cleaning assembly.

In one embodiment, when the target object is an obstacle, the computer-executable instructions further implement the following steps when executed by the processor: according to the type of the object, determine whether the robot can pass the target object; if not , then according to the position of the object and the target navigation point, an escape path is generated, and the robot is controlled to travel to the target navigation point according to the escape path.

In one embodiment, when the visual data is collected based on the image coordinate system of the robot, when the computer-executable instruction is executed by the processor, the following step is further implemented: acquiring the image coordinate system of the robot and the first correspondence between the radar coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, the object position is converted.

In one embodiment, the contamination detection network includes a downsampling layer and a deconvolution layer; the computer-executable instructions, when executed by the processor, further implement the following step: hierarchizing the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the target dirt in the visual data. Location and target soiling type.

In one embodiment, when the computer-executable instructions are executed by the processor, the following steps are further implemented: pre-training the contamination detection network by using the visual semantic segmentation data set to obtain an initial contamination detection network; The sample data is used as the input of the initial contamination detection network, the sample labeled data is used as the expected output of the initial contamination detection network, and the initial contamination detection network is continuously trained by using a preset loss function.

In one embodiment, the computer-executable instructions, when executed by the processor, further implement the step of: performing a data augmentation process on the dirty data set.

In one embodiment, the computer-executable instructions, when executed by the processor, further implement the following steps: generating a target cleaning strategy according to the target soiling type; controlling the robot to navigate to the target soiling location, and adopting the target cleaning strategy The target dirty location is cleaned.

In one embodiment, the number of the target objects is at least one; when the computer-executable instructions are executed by the processor, the following steps are further implemented: performing path planning on the position of at least one object to obtain a target cleaning path; controlling the robot to follow the specified path. The target cleaning path is navigated to the at least one object position in sequence, and a patrol cleaning task is performed based on the object type corresponding to the current object position.

In one embodiment, when the computer-executable instructions are executed by the processor, the following steps are further implemented: controlling the robot to rotate, and collecting visual data within the field of view of the robot during the rotation.

In one embodiment, the computer-executable instructions, when executed by the processor, further implement the step of: controlling the robot to rotate based on the field of view of the at least one sensor.

In one embodiment, when the computer-executable instructions are executed by the processor, the following steps are further implemented: controlling the first vision sensor and the second vision sensor to rotate, and collecting visual data within their respective fields of view during the rotation; wherein, the The first vision sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line, the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located below the horizontal line, which is the same as the horizontal line. Horizontal lines intersect.

The robot inspection and cleaning device, the robot, and the storage medium provided in the above embodiments can execute the robot inspection and cleaning method provided by any embodiment of the present application, and have corresponding functional modules and effects for executing the method. For technical details that are not described in detail in the foregoing embodiments, reference may be made to the inspection and cleaning method for a robot provided by any embodiment of the present application.

It can be understood that all or part of the process in the method of the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it may include the flow of the above-mentioned multiple method embodiments. Wherein, any reference to memory, storage, database or other medium used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include Read-Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (Electrically PROM, EPROM), Electrically Erasable Programmable ROM (Electrically Erasable) PROM, EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM SDRAM, DDRSDRAM), enhanced SDRAM (Enhanced SDRAM, ESDRAM), synchronous link DRAM (Synchlink DRAM, SLDRAM), memory bus direct RAM (Rambus Direct RAM, RDRAM), direct memory bus dynamic RAM (Dynamic RDRAM, DRDRAM), And memory bus dynamic RAM (Rambus Dynamic RAM, RDRAM) and so on.

The multiple technical features of the above-described embodiments can be combined arbitrarily. To simplify the description, all possible combinations of the multiple technical features in the above-described embodiments are not described.

Claims

A robot inspection and cleaning method, comprising:

Collect visual data within the robot's field of vision;

The visual data is detected by a preset network to obtain the object position and object type of the target object, wherein the preset network is obtained by training the original sample data and the sample annotation data corresponding to the original sample data;

The robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.
The method according to claim 1, wherein the preset network is a pre-trained neural network, the original sample data is visual sample data, and the sample labeling data corresponding to the original sample data is the position of the labeled sample object and the sample The visual sample data of the object type.
The method according to claim 2, wherein the pre-trained neural network comprises a feature extraction layer, a feature fusion layer and an object recognition layer;

Detecting the visual data through the preset network to obtain the object position and object type of the target object, including:

Extracting multi-scale feature data in the visual data through the feature extraction layer;

Perform feature fusion on the multi-scale feature data through the feature fusion layer to obtain fused feature data;

According to the multi-scale feature data and the fused feature data, the object position and object type of the target object are determined through the object recognition layer.
The method of claim 3, wherein the feature extraction layer comprises a first feature extraction block and a second feature extraction block;

The extracting multi-scale feature data in the visual data through the feature extraction layer includes:

The first scale feature data in the visual data is extracted by the first feature extraction block, and the second scale feature data in the visual data is extracted by the second feature extraction block.
The method according to claim 4, wherein the first scale feature data is 13*13 scale feature data, and the second scale feature data is 26*26 scale feature data.
The method according to any one of claims 2 to 5, wherein, in a case where the target object is at least one of garbage and dirt, the controlling according to the object position and object type of the target object The robot performs inspection and cleaning tasks, including:

selecting a target storage assembly and a target cleaning assembly according to the object type;

The robot is controlled to navigate to the object position, and the robot is controlled to clean the target object into the target storage assembly, and the cleaned area is cleaned by the target cleaning assembly.
The method according to any one of claims 2 to 5, wherein when the target object is an obstacle, the robot is controlled to perform patrol cleaning according to the object position and object type of the target object tasks, including:

According to the object type, determining whether the robot can pass the target object;

In response to the robot being unable to go over the target object, an escape path is generated according to the position of the object and the target navigation point, and the robot is controlled to travel to the target navigation point according to the escape path.
The method according to claim 1, wherein the target object is dirt, and the preset network is a preset dirt detection network;

The dirty detection network is obtained by training a visual semantic segmentation dataset and a dirty dataset, wherein the dirty dataset includes the original sample data and sample labeling data corresponding to the original sample data.
The method of claim 8, wherein the contamination detection network comprises a downsampling layer and a deconvolution layer;

Detecting the visual data through the preset network to obtain the object position and object type of the target object, including:

Perform a hierarchical downsampling operation on the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map;

A hierarchical deconvolution operation is performed on the multi-resolution intermediate feature map through the deconvolution layer to obtain the object position and object type of the target object in the visual data.
The method according to claim 8, wherein the acquisition process of the contamination detection network comprises:

The detection network is pre-trained by the visual semantic segmentation data set to obtain an initial dirty detection network;

The original sample data is used as the input of the initial contamination detection network, the sample labeled data is used as the expected output of the initial contamination detection network, and the initial contamination detection network is performed using a preset loss function. Training to obtain the dirty detection network.
The method according to claim 10, before using the preset loss function to train the initial contamination detection network, further comprising:

Data augmentation is performed on the dirty dataset.
The method according to claim 11, wherein the manner of performing data enhancement processing on the dirty data set comprises at least one of the following: random cropping, horizontal flipping, and color dithering.
The method according to any one of claims 8 to 12, wherein the controlling the robot to perform an inspection and cleaning task according to the object position and the object type of the target object comprises:

generating a target cleaning strategy according to the object type;

The robot is controlled to navigate to the object location, and the object location is cleaned using the target cleaning strategy.
The method of any one of claims 1, 3, or 6-10, wherein the number of the target objects is at least one;

The method also includes:

Perform path planning on at least one object position to obtain a target cleaning path;

The controlling the robot to perform the inspection and cleaning task according to the object position and object type of the target object includes:

The robot is controlled to navigate to the at least one object position in sequence according to the target cleaning path, and perform patrol cleaning tasks based on the object type corresponding to the current object position.
The method according to claim 14, wherein the collecting visual data within the field of view of the robot comprises:

The robot is controlled to rotate, and visual data within the field of view of the robot is collected during the rotation.
16. The method of claim 15, wherein the controlling the robot to rotate comprises:

The robot is controlled to rotate based on the field of view of the at least one sensor.
The method according to claim 14, wherein the collecting visual data within the field of view of the robot comprises:

Control the first visual sensor and the second visual sensor to rotate, and control the first visual sensor and the second visual sensor to collect visual data within the field of view during the rotation; wherein, the first visual sensor is all The forward-looking sensor of the robot, and the central axis of the viewing angle of the first visual sensor is parallel to the horizontal line, the second visual sensor is the downward-looking sensor of the robot, and the viewing angle of the second visual sensor is The central axis is below the horizontal line and intersects the horizontal line.
The method according to any one of claims 2, 8 to 12, or 14 to 17, wherein, in the case where the visual data is collected based on the image coordinate system of the robot, in the Before the object position and object type of the target object control the robot to perform the inspection and cleaning task, it also includes:

obtaining the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system;

The object position is converted according to the first correspondence and the second correspondence.
A robot inspection and cleaning device, comprising:

The acquisition module is set to collect visual data within the field of vision of the robot;

The detection module is configured to detect the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network uses the original sample data and the sample annotation data corresponding to the original sample data. trained;

The control module is configured to control the robot to perform the inspection and cleaning task according to the object position and the object type of the target object.
The device according to claim 19, wherein the preset network is a pre-trained neural network, the original sample data is visual sample data, and the sample labeling data corresponding to the original sample data is the position of the labeled sample object and the sample The visual sample data of the object type.
The device according to claim 19, wherein the target object is dirt, and the preset network is a preset dirt detection network;

The dirty detection network is obtained by training a visual semantic segmentation dataset and a dirty dataset, wherein the dirty dataset includes the original sample data and sample labeling data corresponding to the original sample data.
The apparatus according to claim 19 or 21, wherein the number of the target objects is at least one;

The device further includes a path planning module, which is configured to perform path planning on at least one object position to obtain a target cleaning path;

The control module is configured to: control the robot to navigate to the at least one object position in sequence according to the target cleaning path, and perform the inspection and cleaning task based on the object type corresponding to the current object position.
A robot comprising: at least one processor, a memory, and at least one program, wherein the at least one program is stored in the memory and executed by the at least one processor, the at least one program includes using Instructions for executing the inspection and cleaning method of a robot according to any one of claims 1 to 18.
A non-volatile computer-readable storage medium containing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform any one of claims 1 to 18 The robot inspection and cleaning method.