WO2022088430A1 - Procédé et appareil d'inspection et de nettoyage d'un robot, robot et support de stockage - Google Patents

Procédé et appareil d'inspection et de nettoyage d'un robot, robot et support de stockage Download PDF

Info

Publication number
WO2022088430A1
WO2022088430A1 PCT/CN2020/136691 CN2020136691W WO2022088430A1 WO 2022088430 A1 WO2022088430 A1 WO 2022088430A1 CN 2020136691 W CN2020136691 W CN 2020136691W WO 2022088430 A1 WO2022088430 A1 WO 2022088430A1
Authority
WO
WIPO (PCT)
Prior art keywords
robot
data
target
visual
cleaning
Prior art date
Application number
PCT/CN2020/136691
Other languages
English (en)
Chinese (zh)
Inventor
沈孝通
侯林杰
秦宝星
程昊天
Original Assignee
上海高仙自动化科技发展有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202011182069.2A external-priority patent/CN112287834A/zh
Priority claimed from CN202011182064.XA external-priority patent/CN112287833A/zh
Priority claimed from CN202011186175.8A external-priority patent/CN112315383B/zh
Application filed by 上海高仙自动化科技发展有限公司 filed Critical 上海高仙自动化科技发展有限公司
Publication of WO2022088430A1 publication Critical patent/WO2022088430A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47L—DOMESTIC WASHING OR CLEANING; SUCTION CLEANERS IN GENERAL
    • A47L11/00—Machines for cleaning floors, carpets, furniture, walls, or wall coverings
    • A47L11/24—Floor-sweeping machines, motor-driven
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods

Definitions

  • the present application relates to the field of robotics, for example, to a robot inspection and cleaning method, device, robot and storage medium.
  • the cleaning robot can complete simple and repetitive cleaning tasks through unmanned driving technology, greatly reducing labor costs and realizing the automation of cleaning work.
  • the robot When the robot is patrolling and cleaning, it generally drives according to the pre-planned navigation map, and fully covers and cleans the ground during the driving process.
  • the above patrol cleaning method results in low cleaning efficiency of the robot.
  • the present application provides a robot inspection and cleaning method, device, robot and storage medium.
  • a robot inspection and cleaning method including:
  • the visual data is detected by a preset network to obtain the object position and object type of the target object, wherein the preset network is obtained by training the original sample data and the sample annotation data corresponding to the original sample data;
  • the robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.
  • a robot inspection and cleaning device including:
  • the acquisition module is set to collect visual data within the field of vision of the robot
  • the detection module is configured to detect the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network uses the original sample data and the sample annotation data corresponding to the original sample data. trained;
  • the control module is configured to control the robot to perform the inspection and cleaning task according to the object position and the object type of the target object.
  • a robot comprising: at least one processor, a memory, and at least one program, wherein the at least one program is stored in the memory and executed by the at least one processor, the at least one program includes using Instructions for executing the above-mentioned robot inspection and cleaning method.
  • a non-volatile computer-readable storage medium containing computer-executable instructions, when the computer-executable instructions are executed by at least one processor, the at least one processor is made to execute the above-mentioned robot inspection and cleaning method .
  • FIG. 1 is a schematic structural diagram of a robot according to an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of a method for patrolling and cleaning a robot according to an embodiment of the present application
  • FIG. 3 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the principle of a robot recognizing visual data according to an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a contamination detection network provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of a training method for a contamination detection network provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a robot inspection and cleaning device provided in an embodiment of the application.
  • FIG. 11 is a schematic structural diagram of another robot inspection and cleaning device according to an embodiment of the application.
  • FIG. 12 is a schematic structural diagram of another robot inspection and cleaning device according to an embodiment of the application.
  • FIG. 13 is a schematic structural diagram of another robot inspection and cleaning device provided by an embodiment of the application.
  • FIG. 14 is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present application.
  • the robot may include: a sensor 10 , a controller 11 and an execution component 12 .
  • the sensor 10 includes a perception sensor and a positioning sensor installed on the robot body, and the sensor 10 is used to collect visual data within the field of view, which can be different types of cameras, lidar, infrared ranging, ultrasonic IMU (Inertial Measurement Unit, inertial measurement unit), odometer and other single or multiple sensors.
  • the controller 11 may include a chip and a control circuit, mainly by receiving the visual data collected by the sensor 10, to actively identify the target objects (such as garbage and obstacles, etc.) existing in the field of view of the robot, and to execute based on the position and type of the target object. Patrol cleaning tasks.
  • the execution component 12 includes a walking component and a cleaning component, and is configured to receive control instructions from the controller 11, navigate to the location of the target object according to the planned travel path, and perform cleaning operations.
  • the execution subject of the following method embodiments may be a robot inspection and cleaning device, and the device may be implemented as part or all of the above robot through software, hardware, or a combination of software and hardware.
  • the following method embodiments are described by taking the execution subject being a robot as an example.
  • FIG. 2 is a schematic flowchart of a method for patrolling and cleaning a robot according to an embodiment of the present application. This embodiment relates to the process of how the robot performs inspection and cleaning of the workspace. As shown in Figure 2, the method may include:
  • the area to be cleaned can be inspected and cleaned by the robot.
  • the to-be-cleaned area refers to an area where the robot needs to perform inspection and cleaning, which may correspond to the environment where the robot is located.
  • the robot can generate a field of vision path for covering the field of vision of the area to be cleaned through its own field of view and the electronic map of the area to be cleaned.
  • electronic maps include but are not limited to grid maps, topological maps and vector maps.
  • the robot drives according to the vision path, and actively collects visual data within the field of view during the driving process, and actively identifies and cleans the target objects in the visual data, so as to realize active inspection of the area to be cleaned.
  • a vision sensor is arranged on the robot, so that the robot can collect data from an area within its field of view through the vision sensor, thereby obtaining visual data within the field of view of the vision sensor.
  • the types of visual data collected by the visual sensors are also different.
  • the above-mentioned visual data may be image data, video data, or point cloud data.
  • the above-mentioned visual sensor may be a camera.
  • the robot can continuously shoot the area within its field of view through the camera to obtain surveillance video, and use the surveillance video as visual data to be identified.
  • the robot can also directly photograph the area within the field of view through the camera to obtain the photographed image, and use the photographed image as the visual data to be recognized.
  • the preset network is obtained by training the original sample data and the sample labeling data corresponding to the original sample data.
  • the robot After obtaining the visual data within the field of view, the robot inputs the visual data to the preset network, identifies the target object in the visual data through the preset network, and outputs the object position and object type of the target object.
  • the target object may be garbage and/or obstacles.
  • the object type may include various types of garbage, such as plastic bags, napkins, paper scraps, fruit peels, and vegetable leaves.
  • the object type may also include the results of classifying various types of garbage based on garbage classification criteria, such as recyclable garbage, kitchen waste, hazardous garbage, and other garbage.
  • the object types When the target object is an obstacle, the object types may include large-sized obstacles, small-sized obstacles, dynamic obstacles, static obstacles, and semi-static obstacles.
  • the training data of the preset network may be an original sample data set collected according to actual training requirements, or may be an original sample data set in a training database.
  • S203 Control the robot to perform an inspection and cleaning task according to the object position and object type of the target object.
  • the robot After determining that there is a target object in the field of view, the robot can perform targeted inspection and cleaning tasks based on the object position and object type of the target object.
  • the inspection and cleaning method for a robot collects visual data within the field of view of the robot, identifies the visual data through a preset network, obtains the object position and object type of the target object, and determines the target object according to the target object.
  • the object position and object type control the robot to perform patrol cleaning tasks.
  • the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the preset network, so that the robot only needs to focus on the target in the workspace.
  • Objects, and perform inspection and cleaning tasks based on the location and type of the target object eliminating the need for full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot.
  • the preset network is a pre-trained neural network
  • the original sample data is visual sample data
  • the sample annotation data corresponding to the original sample data is the visual sample object position and type of the sample object that have been marked. sample.
  • FIG. 3 is a schematic flowchart of another robot inspection and cleaning method according to an embodiment of the present application. As shown in Figure 3, the method may include:
  • the pre-trained neural network is obtained by training the visual sample data and the visual sample data marked with the position of the sample object and the type of the sample object.
  • a pre-trained neural network can be a network model that is pre-established and configured in the robot after training to recognize the target object in the visual data and output the object position and object type of the target object.
  • the above pre-trained neural networks can be based on You Only Look Once (YOLO), RetinaNet, Single Shot MultiBox Detector (SSD) or faster regional convolutional neural networks (Faster- Region Convolutional Neural Networks, Faster-RCNN) and other networks.
  • the robot After obtaining the visual data within the field of view, the robot inputs the visual data to the pre-trained neural network, identifies the target object in the visual data through the pre-trained neural network, and outputs the object position and object type of the target object.
  • the training data of the above-mentioned pre-trained neural network may be a visual sample data set collected according to actual training requirements, or may be a visual sample data set in a training database.
  • the visual sample data set includes visual sample data to be identified, and visual sample data marked with the position of the sample object and the type of the sample object.
  • the sample objects may include garbage and obstacles on the ground, and the garbage may include plastic bags, napkins, paper scraps, fruit peels, and the like.
  • the visual sample data is used as the input of the pre-trained neural network, and the sample object position and sample object type existing in the visual sample data are used as the expected output of the pre-trained neural network, and the corresponding The loss function trains the pre-trained neural network until the convergence condition of the loss function is reached, thereby obtaining the above-mentioned pre-trained neural network.
  • the selected visual sample data set can be clustered to obtain reference frames (anchors) with different aspect ratios and different sizes. For example, taking a common garbage dataset as an example, k-means clustering operation is performed on the garbage dataset to learn reference frames with different aspect ratios and different sizes from the garbage dataset.
  • the robot can input the image to be detected into the pre-trained neural network.
  • the pre-trained neural network can extract the feature map corresponding to the image to be detected through the darknet sub-network, and for each grid on the feature map, predict the description information of the above reference frames with different aspect ratios and different sizes, wherein the description information includes The confidence level of the reference frame, the location information of the reference frame, and the category information of the reference frame. Next, based on the confidence of the reference frame and the category information of the reference frame, filter out the reference frame with lower probability, and perform non-maximum suppression processing on the remaining reference frame to obtain the final detection result, which is the visual data.
  • the object position and object type of the target object can extract the feature map corresponding to the image to be detected through the darknet sub-network, and for each grid on the feature map, predict the description information of the above reference frames with different aspect ratios and different sizes, wherein the description information includes The confidence level of the reference frame, the location information of the reference frame, and the category information of the reference frame
  • the above S303 may include:
  • the target storage component and the target cleaning component can be respectively selected for the object type.
  • the robot may be provided with recyclable garbage storage components, kitchen waste storage components, hazardous waste storage components and other garbage storage components. In this way, after obtaining the object type of the target object, the robot can select the target storage component from all the set storage components based on the object type. For example, when the object type of the target object obtained by the robot is vegetable leaves, the robot can select the kitchen waste storage component as the target storage component.
  • the target cleaning component can be selected from all the set cleaning components based on the object type.
  • the robot may be provided with a vacuuming assembly, a dry mopping assembly, a wet mopping assembly, a drying assembly, a water absorbing assembly, and the like.
  • the target cleaning component can be selected from all the set cleaning components based on the object type. For example, target objects such as vegetable leaves and fruit peels may leave stains on the ground. Therefore, after the robot cleans such target objects to the kitchen waste storage component, it needs to be wiped with the wet mop component, and then used to dry Dry components are dried, according to which the robot can select wet mopping components and drying components as target cleaning components.
  • the robot can plan a cleaning route based on the object position of the target object, and control the robot to drive along the cleaning route to the target position, and then clean the target object into the target storage assembly at the target position. , and use the selected target cleaning component to clean the cleaned area based on the corresponding cleaning strategy.
  • the above S303 may include: according to the object type, determining whether the robot can pass the target object; if the robot cannot pass the target object, generate an escape path according to the object position and the target navigation point, and control the robot to drive to the target navigation point according to the escape path.
  • the object types of the target object may include large-sized obstacles and small-sized obstacles.
  • small-sized obstacles the unique chassis structure of the robot enables the robot to cross small-sized obstacles; for large-sized obstacles, due to the large size of the obstacles, it is difficult for the robot to go over the large-sized obstacles. OK, causing the robot to be trapped. Therefore, when the robot performs the inspection and cleaning task, the robot needs to determine whether the robot can pass the target object according to the object type of the identified target object. That is, when the identified target object is a large-sized obstacle, the robot cannot move forward over the large-sized obstacle. At this time, the robot enters the escape mode to avoid the large-sized obstacle.
  • the robot selects a path point from the initial cleaning path as the target navigation point, and generates an escape path based on the position information of the large-sized obstacle and the target navigation point, and controls the robot to drive according to the escape path to target navigation point.
  • the robot inspection and cleaning method collects visual data within the field of vision of the robot, identifies the visual data through a pre-trained neural network, obtains the object position and object type of the target object, and determines the object position and object type according to the object.
  • the location and object type control the robot to perform patrol cleaning tasks.
  • the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the pre-trained neural network, so that the robot only needs to focus on the objects in the workspace.
  • the foregoing pre-trained neural network may include a feature extraction layer, a feature fusion layer, and an object recognition layer.
  • the above S302 may include:
  • Robots can choose a deep learning network as this feature extraction layer.
  • the feature extraction layer may be a darknet network or other network structures.
  • the robot inputs the collected visual data into the pre-trained neural network, and extracts the features in the visual data through the feature extraction layer in the pre-trained neural network to obtain multi-scale feature data.
  • each scale feature data includes description information of a reference frame corresponding to each grid in the scale feature data.
  • the description information includes the confidence level of the reference frame, the location information of the reference frame, and the category information of the reference frame.
  • the above reference frame can be obtained by performing a clustering operation on the training data of the pre-trained neural network.
  • the number of feature extraction blocks in the feature extraction layer can be reduced.
  • the feature extraction layer includes two feature extraction blocks, which are a first feature extraction block and a second feature extraction block, respectively.
  • the process of the above S401 may be: extracting the first scale feature data in the visual data through the first feature extraction block, and extracting the second scale feature data in the visual data through the second feature extraction block.
  • the first scale feature data and the second scale feature data can be arbitrarily selected from 13*13 scale feature data, 26*26 scale feature data and 52*52 scale feature data for combination.
  • the first scale feature data may be 13*13 scale feature data
  • the second scale feature data may be 26*26 scale feature data
  • the feature extraction layer inputs the extracted multi-scale feature data to the feature fusion layer, and the multi-scale feature data is feature-fused through the feature fusion layer.
  • the robot uses the feature fusion layer to combine the 13*13 scale feature data and 26*26 scale feature data.
  • Feature fusion is performed on the scale feature data to obtain the fused feature data.
  • S403. Determine the object position and object type of the target object through the object recognition layer according to the multi-scale feature data and the fused feature data.
  • the feature extraction layer inputs the extracted multi-scale feature data to the object recognition layer, and the feature fusion layer also inputs the fused feature data to the object recognition layer, and the multi-scale feature data and the fused feature data are processed by the object recognition layer.
  • the object recognition layer can perform coordinate transformation and coordinate scaling on the reference frame in the multi-scale feature data and the fused feature data, and restore the reference frame in the multi-scale feature data and the fused feature data to the original data.
  • the restored frame of reference Next, non-maximum suppression processing is performed on the restored reference frame, redundant reference frames are filtered out, and description information of the filtered reference frame is output, so as to obtain the object position and object type of the target object.
  • the above-mentioned recognition process of visual data through the pre-trained neural network is introduced.
  • the robot inputs the collected visual data within the visual field to the feature extraction layer 501 in the pre-trained neural network, and extracts the features in the visual data through the first feature extraction block 5011 in the feature extraction layer 501 to obtain 13*13 scale feature data , and extract the features in the visual data through the second feature extraction block 5012 in the feature extraction layer to obtain 26*26 scale feature data.
  • the robot inputs the 13*13 scale feature data and the 26*26 scale feature data into the feature fusion layer 502 in the pre-trained neural network, and the feature fusion layer 502 performs the 13*13 scale feature data and the 26*26 scale feature data.
  • Feature fusion to obtain the fused feature data.
  • the robot inputs the 13*13 scale feature data, 26*26 scale feature data and the fused feature data into the object recognition layer 503 in the pre-trained neural network, and the 13*13 scale feature data, 26*26
  • the scale feature data and the fused feature data are processed by coordinate transformation, coordinate scaling and non-maximum suppression, so as to identify the target object in the visual data, and output the object position and object type of the target object.
  • the pre-trained neural network can perform feature fusion on the multi-scale feature data in the visual data, and recognize the target object based on the fused feature data and the multi-scale feature data, thereby improving the robot recognition effect.
  • the feature extraction layer in the pre-trained neural network only includes two feature extraction blocks. Compared with the feature extraction layer including three feature extraction blocks, the feature extraction layer is reduced on the premise that the recognition effect of the robot can be satisfied. The number of feature extraction blocks in the middle, thereby improving the recognition speed of the robot.
  • the robot usually collects visual data within the field of view through a camera.
  • the object position of the target object recognized by the robot through the pre-trained neural network is calculated in the image coordinate system.
  • the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, Transform the object position.
  • the obtaining of the first correspondence between the image coordinate system of the robot and the radar coordinate system may include: respectively obtaining the first correspondence between the robot's pixel coordinate system and the radar coordinate system for the same object to be collected. data and second data; match the first data and the second data to obtain multiple sets of matched feature points; determine the image coordinate system and radar coordinates of the robot according to the multiple sets of matched feature points The first correspondence between the systems.
  • the object to be collected can be set on a corner in advance.
  • the robot is provided with a camera and a laser radar, and the robot controls the camera and the laser radar to collect data from different angles of the object to be collected set on the corner of the wall, thereby obtaining the first data and the second data.
  • the feature points in the first data and the second data are respectively detected, and the feature points in the first data and the second data are matched to obtain multiple sets of matched feature points.
  • four sets of matching feature points or more need to be determined.
  • the corresponding equations are established, and the correspondence between the robot's image coordinate system and the radar coordinate system can be obtained by solving the equations.
  • the robot converts the obtained object position through the first correspondence between the robot's image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system, so that the final obtained object position is obtained.
  • the actual position of the target object is more accurate, and then the robot is controlled to perform inspection and cleaning tasks based on the accurate object position, which improves the cleaning accuracy and cleaning efficiency of the robot.
  • the target object is dirt
  • the preset network is a preset dirt detection network
  • the dirt detection network is obtained by training a visual semantic segmentation dataset and a dirt dataset, wherein,
  • the dirty data set includes the original sample data and sample annotation data corresponding to the original sample data.
  • FIG. 6 is a schematic flowchart of another method for patrolling and cleaning a robot according to an embodiment of the present application. As shown in Figure 6, the method may include:
  • the dirty detection network is obtained by training a visual semantic segmentation data set and a dirty data set, and the dirty data set includes original sample data and sample labeling data corresponding to the original sample data.
  • the above-mentioned contamination detection network is a deep learning model, which can be a network model that is pre-established and configured in the robot after training on the visual semantic segmentation dataset and the contamination dataset, so as to detect the target contamination existing in the visual data.
  • the above-mentioned visual semantic segmentation data set may be the Cityscapes data set;
  • the above-mentioned sample labeling data refers to the original sample data for which the dirty position of the sample and the dirty type of the sample have been marked.
  • the above-mentioned contamination detection network can be established based on a weighted convolutional harmonic dense network (Fully Convolutional Harmonic Dense Net, FCHarDNet), U-Net, V-Net, and Pyramid Scene Parsing Net (PSPNet) and other networks.
  • FCHarDNet Weighted convolutional Harmonic Dense Net
  • FCHarDNet Weighted convolutional Harmonic Dense Net
  • U-Net U-Net
  • V-Net V-Net
  • PSPNet Pyramid Scene Parsing Net
  • the robot After obtaining the visual data within the field of view, the robot inputs the visual data into the trained dirt detection network, detects the target dirt in the visual data through the dirt detection network, and outputs the target dirt position and target dirt. type of contamination.
  • the target soiling types may include liquid soiling and solid soiling.
  • the robot can extract dirt features in the visual data through a dirt detection network, and determine the target dirt type according to the dirt features.
  • the soil characteristics may include soil particle size and soil transparency (soil transparency refers to the light transmittance of soil).
  • the above-mentioned contamination detection network may include a downsampling layer and a deconvolution layer.
  • the foregoing S602 may include: performing a hierarchical downsampling operation on the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map;
  • the deconvolution layer performs a hierarchical deconvolution operation on the multi-resolution intermediate feature map to obtain the target dirt position and target dirt type in the visual data.
  • the contamination detection network includes N downsampling layers and N deconvolution layers.
  • the input layer in Figure 7 is used to input visual data
  • the output layer is used to output the target dirty position and target dirty type in the visual data.
  • the robot inputs the collected visual data within the field of view into the input layer, and performs hierarchical down-sampling operations on the visual data through N down-sampling layers to extract the dirty features in the visual data and obtain different resolutions. feature map.
  • Target soiling type is used to output the target dirty position and target dirty type in the visual data.
  • the contamination detection network may further include an attention threshold block.
  • the above process of performing a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer may be: enhancing and suppressing the multi-resolution intermediate feature map layer by layer through an attention threshold block , and perform a deconvolution operation.
  • FIG. 7 only shows that the number N of downsampling layers and deconvolution layers included in the contamination detection network is 4 as an example. This embodiment does not limit the downsampling layers and deconvolution layers included in the contamination detection network.
  • the number N of downsampling layers and deconvolution layers included in the dirt detection network can be set correspondingly according to the actual application requirements.
  • the upsampling operation of the multi-resolution intermediate feature map is realized through the deconvolution layer, and only the intermediate feature map and the convolution kernel in the deconvolution layer need to be deconvolved. Compared with using bilinear For the interpolated upsampling layer, the time of contamination detection is greatly shortened, and the efficiency of contamination detection is improved.
  • the robot After determining that the target is dirty within the field of view, the robot navigates to the target dirty position, and cleans the target dirty position in a targeted manner based on the target dirt type.
  • the process of the above S603 may be: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the target dirt location, and using the target cleaning strategy to clean the target dirt location .
  • the robot can generate a target cleaning strategy for cleaning according to the obtained target dirt type.
  • the target soiling type is liquid soiling
  • the target cleaning strategy generated by the robot can be used to first use the water absorbing component to absorb the liquid. , and then use the dry mop component to wipe the ground.
  • the target dirt type is solid dirt, since it is solid, the solid can be cleaned and then wiped with a wet mop.
  • the target cleaning strategy generated by the robot can be to use the vacuum component to clean the solid first, and then use the vacuum cleaner to clean the solid.
  • Use the wet mopping component to wipe the floor, and then use the drying component to dry the floor.
  • the material of the ground can also be combined.
  • the robot navigates to the target dirty position, and uses the generated target cleaning strategy to clean the target dirty position.
  • the robot can continue to collect visual data within its own field of view at the target dirty position to identify the target dirt that needs to be cleaned in the next step, that is, repeat the above process of S601-S603, so as to complete the process of cleaning. Inspection and cleaning of the entire work space.
  • the robot can also return to the target navigation point in the field of view path, take the target navigation point as the starting point for cleaning, and continue to collect visual data within the field of view.
  • the robot inspection and cleaning method collects the visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, obtains the target contamination position and the target contamination type, and Control the robot to perform patrol cleaning tasks according to the target dirty position and target dirty type.
  • the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot.
  • an acquisition process of a contamination detection network is also provided, that is, how to train a contamination detection network.
  • the training process of the contamination detection network may include:
  • the contamination identification parallel data (that is, the contamination data set) can improve the detection performance of the contamination detection network
  • the training of the contamination detection network is very time-consuming and labor-intensive. Still not meeting expectations.
  • the dirt detection network can be pre-trained through the visual-semantic segmentation dataset with a large number of samples, and the initial dirt detection network trained on the visual-semantic segmentation dataset can be obtained.
  • the visual semantic segmentation dataset can be the Cityscapes dataset.
  • the contamination detection network After pre-training the contamination detection network with the visual semantic segmentation data set to obtain the initial contamination detection network, you can continue to use the collected contamination data set to fine-tune the initial contamination detection network for training.
  • the original sample data in the dirty data set is used as the input of the initial dirty detection network
  • the sample annotation data in the dirty data set is taken as the expected output of the initial dirty detection network
  • the preset loss function is used for the initial dirty detection network.
  • the parameters are trained and adjusted until the convergence condition of the loss function is reached, so as to obtain a trained dirty detection network.
  • the loss function can be a cross-entropy loss function.
  • the method further includes: performing data enhancement processing on the contamination data set.
  • the method of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.
  • the dirty data set can be expanded by horizontal flip mirroring; the dirty data set can also be cropped to realize the data expansion of the image segmentation data set, that is, a position is randomly selected as the cropping center, and each dirty data is processed. Cropping; each dirty data can also be color jittered for data augmentation of dirty datasets.
  • a visual semantic segmentation data set with a large number of samples can be used to pre-train the contamination detection network, and then the contamination data set is used for pre-training to obtain The initial dirty detection network is fine-tuned for training.
  • the long and slow learning phase in the early stage of network training can be avoided, thereby greatly reducing the network training time.
  • a lot of tedious hyperparameter tuning can be avoided. That is to say, the technical solutions adopted in the embodiments of the present application shorten the training time of the contamination detection network and improve the accuracy of the contamination detection network.
  • the robot usually collects visual data within the field of view through a camera.
  • the target dirty position detected by the robot through the trained dirt detection network is calculated in the image coordinate system.
  • the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, The target dirty location is converted.
  • the robot After acquiring the first correspondence between the robot's image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system, the robot performs projection transformation on the target dirty position according to the first correspondence , and then convert the dirty position after projection transformation based on the second correspondence, so as to obtain the actual position of the target dirt in the world coordinate system.
  • the operation steps of obtaining the first corresponding relationship and the second corresponding relationship, and converting the target dirty position according to the first corresponding relationship and the second corresponding relationship and the resulting effects have been recorded in the above-mentioned embodiments, and are not described here. Repeat.
  • the number of the target objects is at least one; the method further includes: performing path planning on at least one object position to obtain a target cleaning path;
  • the robot performing the inspection and cleaning task includes: controlling the robot to navigate to the at least one object position in sequence according to the target cleaning path, and performing the inspection and cleaning task based on the object type corresponding to the current object position.
  • FIG. 9 is a schematic flowchart of another method for patrolling and cleaning a robot according to an embodiment of the present application. As shown in Figure 9, the method may include:
  • S901. Collect visual data within the field of view of the robot.
  • each target object corresponds to an object position and an object type
  • the multiple object types corresponding to the multiple target objects may be identical or completely different, or Parts are the same, which is not limited.
  • the above-mentioned contamination detection network may include a downsampling layer and a deconvolution layer.
  • the foregoing S902 may include: performing a hierarchical downsampling operation on the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map;
  • the deconvolution layer performs a hierarchical deconvolution operation on the multi-resolution intermediate feature map to obtain at least one target dirty position in the visual data and at least one target dirty position corresponding to the target dirty position. type.
  • the target cleaning path refers to the cleaning path with the shortest distance among all the cleaning paths for the robot to reach at least one target dirty position.
  • the robot can generate at least one target through the shortest path planning algorithm according to the at least one target dirty position, the historical obstacle map of the area to be cleaned, and the current obstacle map of the area to be cleaned.
  • the shortest path planning algorithm may be Dijkstra algorithm, Floyd algorithm, and ant colony algorithm.
  • the robot can navigate to the corresponding target dirty position in sequence according to the target cleaning path, and clean the target dirty position in a targeted manner based on the target dirt type corresponding to the target dirty position.
  • the process of the above S904 may be: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the corresponding target dirt position in sequence according to the target cleaning path, and adopting the target cleaning method.
  • the strategy cleans the target dirty location.
  • the robot can generate a target cleaning strategy for cleaning according to the obtained target dirt type.
  • the target soiling type is liquid soiling
  • the target cleaning strategy generated by the robot can be first used to absorb the liquid with a water absorbing component. , and then use the dry mop component to wipe the ground.
  • the target dirt type is solid dirt, since it is solid, the solid can be cleaned and then wiped with a wet mop.
  • the target cleaning strategy generated by the robot can be: Use the wet mopping component to wipe the floor, and then use the drying component to dry the floor.
  • the material of the ground can also be combined. For example, when the material of the floor is floor and floor tiles, you can use the vacuum cleaner for vacuuming, and then use the mopping unit to mop the floor after vacuuming; when the material of the floor is carpet, you can use the vacuum cleaner only Do vacuuming.
  • the robot navigates to the target dirty location in sequence according to the target cleaning path, and uses the generated target cleaning strategy to clean at least one target dirty location.
  • the robot can control itself to rotate, and collect visual data within the visual range during the rotation to identify the target dirt that needs to be cleaned in the next step, that is, repeatedly execute the above S901-S904 process, so as to complete the inspection and cleaning of the entire workspace.
  • the inspection and cleaning method for a robot collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains at least one target dirty position and at least one target dirty location. According to the target dirt type corresponding to the dirty position, the shortest path planning is performed on at least one target dirty position to obtain the target cleaning path, and the robot is controlled to navigate to the corresponding target dirty position in turn according to the target cleaning path, and based on the corresponding target dirty position Type to perform patrol cleaning tasks.
  • the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot.
  • the robot can also perform shortest path planning for the at least one target dirty position, so that the robot can navigate to the at least one target dirty position with the shortest path, which improves the cleaning efficiency of the robot.
  • the process of the above S901 may be: controlling the robot to rotate, and during the rotation process Collect the visual data within the field of view of the robot.
  • the way of controlling the robot to rotate may be: controlling the robot to rotate based on the field of view of at least one sensor.
  • the robot After generating a field of vision path for covering the field of view of the area to be cleaned based on the robot's field of view and the electronic map of the area to be cleaned, the robot inspects and cleans the area to be cleaned according to the planned field of view path.
  • the robot can be controlled to rotate on the spot based on the field of view of at least one sensor, and the visual data within the field of view of the robot can be collected during the rotation process. In this way, with the rotation of the robot , the direction of the robot's field of vision is continuously adjusted, so that the robot can collect visual data in a wider range at the current position.
  • the robot it is also possible to control the robot to rotate based on the field of view of at least one sensor during the movement of the robot, and continuously collect visual data within the field of view of the robot during the rotation.
  • the rotation timing of the robot may be set, which is not limited in this embodiment.
  • the above process of controlling the robot to rotate may be: controlling the robot to rotate once. That is, before the robot starts to travel, control the robot to rotate once in place, or control the robot to rotate once during the process of the robot, and collect the visual data within the robot's field of vision during the rotation process, so that the robot can perceive its own vision within a 360-degree range.
  • Data collection greatly expands the data collection range of the robot, enabling the robot to actively identify visual data in a wider range, and perform overall cleaning of the identified target dirt, thereby improving the cleaning efficiency of the robot.
  • visual data in the robot workspace can be collected through vision sensors.
  • a first vision sensor and a second vision sensor are installed on the robot.
  • the first visual sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line;
  • the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located below the horizontal line, which is parallel to the horizontal line. intersect.
  • the above-mentioned process of S901 may be: controlling the first visual sensor and the second visual sensor to rotate, and collecting visual data within their respective visual fields during the rotating process.
  • the line of sight of the first visual sensor Since the line of sight of the first visual sensor is head-up, it can obtain a relatively large sensing range for sensing environmental information farther away in the area to be cleaned. Since the line of sight of the second visual sensor is downward and can be directly aimed at the ground, the second visual sensor can more clearly perceive the environmental information on the ground nearby, and can effectively make up for the blind spot of the first visual sensor.
  • the first vision sensor and the second vision sensor can be controlled to collect visual data within their respective fields of view, so that the robot can not only collect data within the far field of view, but also can The second vision sensor collects the data in the blind area of the first vision sensor, which greatly expands the data collection range of the robot.
  • the first vision sensor and the second sensor can rotate, and collect visual data within their respective fields of view during the rotation process, so that with the rotation of the first vision sensor and the second vision sensor, the direction of the robot's field of vision is continuously adjusted. , so that the robot can collect visual data in a wider range and expand the data collection range of the robot.
  • the rotation angles of the first vision sensor and the second vision sensor can be controlled according to actual requirements.
  • the rotation angle may be 360 degrees.
  • the robot is controlled to rotate, and the visual data within the field of view of the robot is collected during the rotation, or the first vision sensor and the second vision sensor of the robot are controlled to rotate, and the respective visual data are collected during the rotation.
  • Visual data within the field of view Through this technical solution, the data collection range of the robot is greatly expanded, so that the robot can actively identify visual data in a wider range, and perform overall cleaning on the identified target dirt, thereby improving the cleaning efficiency of the robot.
  • the robot usually collects visual data within the field of view through a camera.
  • the target dirty position detected by the robot through the trained dirt detection network is calculated in the image coordinate system.
  • the method may further include: Obtain the first correspondence between the image coordinate system of the robot and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, Translating the at least one target dirty location.
  • FIG. 10 is a schematic structural diagram of a robot inspection and cleaning device according to an embodiment of the present application.
  • the apparatus may include: a collection module 100 , an identification module 101 and a control module 102 .
  • the acquisition module 100 is set to collect visual data within the field of vision of the robot; the recognition module 101 is set to detect the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network passes the original
  • the sample data and the sample labeling data corresponding to the original sample data are obtained by training; the control module 102 is configured to control the robot to perform an inspection and cleaning task according to the object position and object type of the target object.
  • the preset network is a pre-trained neural network
  • the original sample data is visual sample data
  • the sample labeling data corresponding to the original sample data is the marked sample object position and The visual sample data of the sample object type.
  • the robot inspection and cleaning device collects visual data within the field of vision of the robot, identifies the visual data through a pre-trained neural network, and obtains the object position and object type of the target object, and according to the The object position and object type control the robot to perform patrol cleaning tasks.
  • the robot can achieve the field of vision coverage of the entire workspace through its own field of view, and the robot can actively identify the target objects existing in the field of view through the pre-trained neural network, so that the robot only needs to focus on the objects in the workspace.
  • the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer;
  • the identification module 101 includes: a feature extraction unit 1011 , a feature fusion unit 1012 and the identification unit 1013;
  • the feature extraction unit 1011 is set to extract the multi-scale feature data in the visual data through the feature extraction layer;
  • the feature fusion unit 1012 is set to perform the multi-scale feature data through the feature fusion layer.
  • the identifying unit 1013 is configured to determine the object position and object type of the target object through the object recognition layer according to the multi-scale feature data and the fused feature data.
  • the feature extraction layer includes a first feature extraction block and a second feature extraction block; the feature extraction unit 1011 is configured to extract the visual data through the first feature extraction block feature data of the first scale, and extract the second scale feature data in the visual data through the second feature extraction block.
  • the first scale feature data is 13*13 scale feature data
  • the second scale feature data is 26*26 scale feature data.
  • control module 102 is configured to select a target storage assembly and a target cleaning assembly according to the type of the object; control the robot Navigating to the object position, and controlling the robot to clean the target object into the target storage assembly, and to clean the cleaned area through the target cleaning assembly.
  • control module 102 is configured to determine whether the robot can pass the target object according to the object type; When the target object cannot be crossed, an escape path is generated according to the position of the object and the target navigation point, and the robot is controlled to travel to the target navigation point according to the escape path.
  • the apparatus further includes: an acquisition module 103 and Conversion module 104 .
  • the acquisition module 103 is configured to acquire the first correspondence between the image coordinate system and the radar coordinate system of the robot before the control module 102 controls the robot to perform the inspection and cleaning task according to the object position and the object type.
  • the second corresponding relationship between the radar coordinate system and the world coordinate system; the conversion module 104 is configured to convert the position of the object according to the corresponding relationship.
  • the target object is dirt
  • the preset network is a preset dirt detection network; the dirt detection network segments the dataset and the dirt data by visual semantics
  • the dirty data set includes the original sample data and sample labeling data corresponding to the original sample data.
  • the inspection and cleaning device for a robot collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains the target contamination location and target contamination type , and control the robot to perform inspection and cleaning tasks according to the target dirty position and target dirty type.
  • the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focuses on the target dirt in the workspace, and performs targeted inspection and cleaning tasks based on the location and type of target dirt, eliminating the need for full-path cleaning of the entire workspace, improving the cleaning efficiency of the robot.
  • the contamination detection network includes a downsampling layer and a deconvolution layer; the detection module 101 is configured to perform hierarchical downsampling on the visual data through the downsampling layer operation to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the target dirty position and target in the visual data. Dirt type.
  • the apparatus further includes: a network training module 105; Pre-training to obtain an initial contamination detection network; using the original sample data as the input of the initial contamination detection network, using the sample labeling data as the expected output of the initial contamination detection network, and using a preset loss The function continues to train the initial dirty detection network.
  • a network training module 105 Pre-training to obtain an initial contamination detection network; using the original sample data as the input of the initial contamination detection network, using the sample labeling data as the expected output of the initial contamination detection network, and using a preset loss The function continues to train the initial dirty detection network.
  • the apparatus further includes: a training data processing module 106; the training data processing module 106 is configured to use a preset loss function in the network training module 105 to continue the initial contamination detection network Data augmentation is performed on the dirty dataset before training.
  • the manner of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.
  • control module 102 is configured to generate a target cleaning strategy according to the target dirt type; control the robot to navigate to the target dirt position, and use the target cleaning strategy to Target dirty locations for cleaning.
  • the number of the target objects is at least one; the apparatus further includes a path planning module 107 , and the path planning module 107 is configured to target at least one target object.
  • the target cleaning path is obtained by performing path planning on the object position; the control module 102 is configured to: control the robot to navigate to the at least one object position in sequence according to the target cleaning path, and perform patrolling based on the object type corresponding to the current object position Check cleaning tasks.
  • the inspection and cleaning device for a robot collects visual data within the field of view of the robot, detects the collected visual data through a preset contamination detection network, and obtains at least one target dirty position and at least one The target dirt type corresponding to the target dirty position, perform shortest path planning for at least one target dirty position, obtain the target cleaning path, and control the robot to navigate to the corresponding target dirty position in sequence according to the target cleaning path, and based on the corresponding target Dirty types perform patrol cleaning tasks.
  • the robot can cover the entire workspace through its own field of view, and the robot can actively identify the target dirt in the field of view through the trained dirt detection network, so that the robot only needs to Focusing on the target dirt in the workspace, and performing targeted inspection and cleaning tasks based on the location and type of target dirt, there is no need to perform full-path cleaning of the entire workspace, which greatly improves the cleaning efficiency of the robot.
  • the robot can also perform shortest path planning for the at least one target dirty position, so that the robot can navigate to multiple target dirty positions with the shortest path, which improves the cleaning efficiency of the robot.
  • the acquisition module 100 is configured to control the robot to rotate, and collect visual data within the field of view of the robot during the rotation.
  • the acquisition module 100 is configured to control the rotation of the robot based on the field of view of at least one sensor.
  • the acquisition module 100 is configured to control the first visual sensor and the second visual sensor to rotate, and collect visual data within their respective fields of view during the rotation;
  • the vision sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line, the second visual sensor is the downward viewing sensor of the robot, and the central axis of the viewing angle is located below the horizontal line and intersects the horizontal line.
  • a robot is provided, the schematic diagram of which can be shown in FIG. 1 .
  • the robot may include: one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors,
  • the program includes instructions for executing the robot inspection and cleaning method described in any of the above embodiments.
  • Collect the visual data within the field of vision of the robot identify the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network passes the original sample data and the corresponding data of the original sample data.
  • the sample labeling data is obtained by training; the robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.
  • the preset network is a pre-trained neural network
  • the original sample data is visual sample data
  • the sample labeling data corresponding to the original sample data is the labeled sample object position and sample object type. Visual sample data.
  • the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer; when the one or more processors execute the program, the following steps are further implemented: extracting the multi-scale feature data in the visual data; feature fusion is performed on the multi-scale feature data through the feature fusion layer to obtain the fused feature data; according to the multi-scale feature data and the fused feature data, The object position and object type of the target object are determined by the object recognition layer.
  • the feature extraction layer includes a first feature extraction block and a second feature extraction block; when the one or more processors execute the program, the following steps are further implemented: extracting through the first feature extraction block feature data of the first scale in the visual data, and extract the feature data of the second scale in the visual data through the second feature extraction block.
  • the first scale feature data is 13*13 scale feature data
  • the second scale feature data is 26*26 scale feature data.
  • the one or more processors when the target object is garbage and/or dirty, further implement the following steps when executing the program: selecting a target storage component and a target cleaning according to the type of the object assembly; controlling the robot to navigate to the object position, and controlling the robot to clean the target object into the target storage assembly, and to clean the cleaned area through the target cleaning assembly.
  • the one or more processors when the target object is an obstacle, further implement the following step when executing the program: determining whether the robot can pass the target object according to the type of the object ; if not, generate an escape path according to the object position and the target navigation point, and control the robot to drive to the target navigation point according to the escape path.
  • the one or more processors when the visual data is collected based on the image coordinate system of the robot, the one or more processors further implement the following steps when executing the program: acquiring the robot's image coordinate system. The first correspondence between the image coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, the position of the object is to convert.
  • the target object is dirt
  • the preset network is a preset dirt detection network
  • the dirt detection network is obtained by training a visual semantic segmentation dataset and a dirt dataset, wherein , the dirty data set includes the original sample data and sample labeling data corresponding to the original sample data.
  • the contamination detection network includes a downsampling layer and a deconvolution layer; when the one or more processors execute the program, the following steps are further implemented: the visual data is processed by the downsampling layer. Perform a hierarchical downsampling operation to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the visual data.
  • the visual data is processed by the downsampling layer.
  • perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the visual data.
  • Target soiling location and target soiling type when the one or more processors execute the program, the following steps are further implemented: the visual data is processed by the downsampling layer. Perform a hierarchical downsampling operation to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi
  • the one or more processors further implement the following steps when executing the program: pre-training the contamination detection network by using the visual semantic segmentation data set to obtain an initial contamination detection network;
  • the original sample data is used as the input of the initial contamination detection network, the sample annotation data is used as the expected output of the initial contamination detection network, and the initial contamination detection network is continued to be performed using a preset loss function. train.
  • the one or more processors further implement the following step when executing the program: performing data enhancement processing on the dirty data set.
  • the manner of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.
  • the following steps are further implemented: generating a target cleaning strategy according to the target dirt type; controlling the robot to navigate to the target dirt position, using the The target cleaning strategy cleans the target soiled locations.
  • the number of the target object is at least one; when the one or more processors execute the program, the following steps are further implemented: performing path planning on at least one object position to obtain a target cleaning path; controlling the The robot navigates to the at least one object position in sequence according to the target cleaning path, and performs the inspection and cleaning task based on the object type corresponding to the current object position.
  • the following steps are further implemented: controlling the robot to rotate, and collecting visual data within the field of view of the robot during the rotation.
  • the following steps are further implemented: controlling the rotation of the robot based on the field of view of the at least one sensor.
  • the one or more processors further implement the following steps when executing the program: controlling the first vision sensor and the second vision sensor to rotate, and collecting visual data within their respective fields of view during the rotation;
  • the first visual sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line
  • the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located on the horizontal line Below, intersects the horizontal line.
  • a non-transitory computer-readable storage medium 140 containing computer-executable instructions 1401 is provided, when the computer-executable instructions are executed by one or more processors 141 , causing the processor 141 to perform the following steps:
  • Collect the visual data within the field of vision of the robot identify the visual data through a preset network to obtain the object position and object type of the target object, wherein the preset network passes the original sample data and the corresponding data of the original sample data.
  • the sample labeling data is obtained by training; the robot is controlled to perform inspection and cleaning tasks according to the object position and object type of the target object.
  • the preset network is a pre-trained neural network
  • the original sample data is visual sample data
  • the sample labeling data corresponding to the original sample data is the labeled sample object position and sample object type. Visual sample data.
  • the pre-trained neural network includes a feature extraction layer, a feature fusion layer, and an object recognition layer; when the computer-executable instructions are executed by the processor, the following steps are further implemented: extracting the visual data through the feature extraction layer The multi-scale feature data in the feature fusion layer; the feature fusion is performed on the multi-scale feature data through the feature fusion layer to obtain the fused feature data; according to the multi-scale feature data and the fused feature data, through the The object recognition layer determines the object location and object type of the target object.
  • the feature extraction layer includes a first feature extraction block and a second feature extraction block; when the computer-executable instructions are executed by the processor, the following steps are further implemented: extracting the visual image by using the first feature extraction block feature data of the first scale in the data, and extract the feature data of the second scale in the visual data through the second feature extraction block.
  • the first scale feature data is 13*13 scale feature data
  • the second scale feature data is 26*26 scale feature data.
  • the computer-executable instructions when executed by the processor, further implement the following steps: selecting a target storage component and a target cleaning component according to the object type; controlling The robot navigates to the object position, and controls the robot to clean the target object into the target storage assembly, and clean the cleaned area through the target cleaning assembly.
  • the computer-executable instructions when executed by the processor: according to the type of the object, determine whether the robot can pass the target object; if not , then according to the position of the object and the target navigation point, an escape path is generated, and the robot is controlled to travel to the target navigation point according to the escape path.
  • the following step is further implemented: acquiring the image coordinate system of the robot and the first correspondence between the radar coordinate system and the radar coordinate system and the second correspondence between the radar coordinate system and the world coordinate system; according to the first correspondence and the second correspondence, the object position is converted.
  • the target object is dirt
  • the preset network is a preset dirt detection network
  • the dirt detection network is obtained by training a visual semantic segmentation dataset and a dirt dataset, wherein , the dirty data set includes the original sample data and sample labeling data corresponding to the original sample data.
  • the contamination detection network includes a downsampling layer and a deconvolution layer; the computer-executable instructions, when executed by the processor, further implement the following step: hierarchizing the visual data through the downsampling layer to obtain a multi-resolution intermediate feature map; perform a hierarchical deconvolution operation on the multi-resolution intermediate feature map through the deconvolution layer to obtain the target dirt in the visual data. Location and target soiling type.
  • the following steps are further implemented: pre-training the contamination detection network by using the visual semantic segmentation data set to obtain an initial contamination detection network;
  • the sample data is used as the input of the initial contamination detection network, the sample labeled data is used as the expected output of the initial contamination detection network, and the initial contamination detection network is continuously trained by using a preset loss function.
  • the computer-executable instructions when executed by the processor, further implement the step of: performing a data augmentation process on the dirty data set.
  • the manner of performing data enhancement processing on the dirty data set includes at least one of the following: random cropping, horizontal flipping, and color dithering.
  • the computer-executable instructions when executed by the processor, further implement the following steps: generating a target cleaning strategy according to the target soiling type; controlling the robot to navigate to the target soiling location, and adopting the target cleaning strategy The target dirty location is cleaned.
  • the number of the target objects is at least one; when the computer-executable instructions are executed by the processor, the following steps are further implemented: performing path planning on the position of at least one object to obtain a target cleaning path; controlling the robot to follow the specified path.
  • the target cleaning path is navigated to the at least one object position in sequence, and a patrol cleaning task is performed based on the object type corresponding to the current object position.
  • the following steps are further implemented: controlling the robot to rotate, and collecting visual data within the field of view of the robot during the rotation.
  • the computer-executable instructions when executed by the processor, further implement the step of: controlling the robot to rotate based on the field of view of the at least one sensor.
  • the following steps are further implemented: controlling the first vision sensor and the second vision sensor to rotate, and collecting visual data within their respective fields of view during the rotation; wherein, the The first vision sensor is the forward-looking sensor of the robot, and the central axis of the viewing angle is parallel to the horizontal line, the second visual sensor is the downward-looking sensor of the robot, and the central axis of the viewing angle is located below the horizontal line, which is the same as the horizontal line. Horizontal lines intersect.
  • the robot inspection and cleaning device, the robot, and the storage medium provided in the above embodiments can execute the robot inspection and cleaning method provided by any embodiment of the present application, and have corresponding functional modules and effects for executing the method.
  • the inspection and cleaning method for a robot provided by any embodiment of the present application.
  • Non-volatile memory may include Read-Only Memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (Electrically PROM, EPROM), Electrically Erasable Programmable ROM (Electrically Erasable) PROM, EEPROM) or flash memory.
  • ROM Read-Only Memory
  • PROM Programmable ROM
  • EPROM Electrically Programmable ROM
  • EPROM Electrically Erasable Programmable ROM
  • EEPROM Electrically Erasable Programmable ROM
  • Volatile memory may include random access memory (RAM) or external cache memory.
  • RAM random access memory
  • RAM is available in various forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM SDRAM, DDRSDRAM), enhanced SDRAM (Enhanced SDRAM, ESDRAM), synchronous link DRAM (Synchlink DRAM, SLDRAM), memory bus direct RAM (Rambus Direct RAM, RDRAM), direct memory bus dynamic RAM (Dynamic RDRAM, DRDRAM), And memory bus dynamic RAM (Rambus Dynamic RAM, RDRAM) and so on.
  • SRAM Static RAM
  • DRAM Dynamic RAM
  • SDRAM Synchronous DRAM
  • SDRAM Double Data Rate SDRAM SDRAM
  • DDRSDRAM Double Data Rate SDRAM SDRAM
  • SDRAM Double Data Rate SDRAM SDRAM
  • SDRAM Double Data Rate SDRAM SDRAM
  • DDRSDRAM Double Data Rate SDRAM SDRAM
  • ESDRAM enhanced SDRAM
  • synchronous link DRAM Synchlink D

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

L'invention porte sur un procédé et sur un appareil d'inspection et de nettoyage d'un robot, sur un robot et sur un support de stockage. Le procédé d'inspection et de nettoyage d'un robot consiste : à collecter des données visuelles dans le champ de vision du robot ; à réaliser une détection sur les données visuelles au moyen d'un réseau prédéfini et à obtenir une position d'objet et un type d'objet d'un objet cible, le réseau prédéfini étant obtenu au moyen de l'apprentissage de données d'échantillon d'origine et de données d'annotation d'échantillon correspondant aux données d'échantillon d'origine ; et à commander le robot pour effectuer une tâche d'inspection et de nettoyage en fonction de la position d'objet et du type d'objet de l'objet cible.
PCT/CN2020/136691 2020-10-29 2020-12-16 Procédé et appareil d'inspection et de nettoyage d'un robot, robot et support de stockage WO2022088430A1 (fr)

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN202011182069.2 2020-10-29
CN202011182069.2A CN112287834A (zh) 2020-10-29 2020-10-29 机器人的巡检清洁方法、装置、机器人和存储介质
CN202011186175.8 2020-10-29
CN202011182064.XA CN112287833A (zh) 2020-10-29 2020-10-29 机器人的巡检清洁方法、装置、机器人和存储介质
CN202011182064.X 2020-10-29
CN202011186175.8A CN112315383B (zh) 2020-10-29 2020-10-29 机器人的巡检清洁方法、装置、机器人和存储介质

Publications (1)

Publication Number Publication Date
WO2022088430A1 true WO2022088430A1 (fr) 2022-05-05

Family

ID=81383429

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136691 WO2022088430A1 (fr) 2020-10-29 2020-12-16 Procédé et appareil d'inspection et de nettoyage d'un robot, robot et support de stockage

Country Status (1)

Country Link
WO (1) WO2022088430A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100106298A1 (en) * 2008-10-27 2010-04-29 Eusebio Guillermo Hernandez Outdoor home cleaning robot
CN107414866A (zh) * 2017-09-07 2017-12-01 苏州三体智能科技有限公司 一种巡检清扫机器人系统及其巡检清扫方法
CN110924340A (zh) * 2019-11-25 2020-03-27 武汉思睿博特自动化系统有限公司 一种用于智能捡垃圾的移动机器人系统与实现方法
CN111543902A (zh) * 2020-06-08 2020-08-18 深圳市杉川机器人有限公司 地面清洁方法、装置、智能清洁设备和存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100106298A1 (en) * 2008-10-27 2010-04-29 Eusebio Guillermo Hernandez Outdoor home cleaning robot
CN107414866A (zh) * 2017-09-07 2017-12-01 苏州三体智能科技有限公司 一种巡检清扫机器人系统及其巡检清扫方法
CN110924340A (zh) * 2019-11-25 2020-03-27 武汉思睿博特自动化系统有限公司 一种用于智能捡垃圾的移动机器人系统与实现方法
CN111543902A (zh) * 2020-06-08 2020-08-18 深圳市杉川机器人有限公司 地面清洁方法、装置、智能清洁设备和存储介质

Similar Documents

Publication Publication Date Title
CN112315383B (zh) 机器人的巡检清洁方法、装置、机器人和存储介质
Lu et al. L3-net: Towards learning based lidar localization for autonomous driving
CN112287834A (zh) 机器人的巡检清洁方法、装置、机器人和存储介质
Sales et al. Adaptive finite state machine based visual autonomous navigation system
CN112287833A (zh) 机器人的巡检清洁方法、装置、机器人和存储介质
Kim et al. UAV-UGV cooperative 3D environmental mapping
Makhubela et al. A review on vision simultaneous localization and mapping (vslam)
CN111768489B (zh) 一种室内导航地图构建方法和系统
KR20200027087A (ko) 로봇 및 그의 제어 방법
CN111679664A (zh) 基于深度相机的三维地图构建方法及扫地机器人
Zarzar et al. Efficient bird eye view proposals for 3d siamese tracking
Liu et al. An enhanced lidar inertial localization and mapping system for unmanned ground vehicles
Hu et al. Robot-assisted mobile scanning for automated 3D reconstruction and point cloud semantic segmentation of building interiors
CN111609853A (zh) 三维地图构建方法、扫地机器人及电子设备
CN117152249A (zh) 基于语义一致性的多无人机协同建图与感知方法及系统
Liu et al. D-lc-nets: Robust denoising and loop closing networks for lidar slam in complicated circumstances with noisy point clouds
Liu An integrated lidar-slam system for complex environment with noisy point clouds
CN112087573A (zh) 环境的绘制
Liu A lidar-inertial-visual slam system with loop detection
CN111609854A (zh) 基于多个深度相机的三维地图构建方法及扫地机器人
WO2022088430A1 (fr) Procédé et appareil d'inspection et de nettoyage d'un robot, robot et support de stockage
Ji et al. Vision-aided localization and navigation for autonomous vehicles
Livatino et al. Autonomous robot navigation with automatic learning of visual landmarks
Li et al. Road edge and obstacle detection on the SmartGuard navigation system
Reddy et al. Image based obstacle detection and path planning for solar powered autonomous vehicle

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20959588

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25.10.2023)