RU2785769C1 - Method and system for indoor cleaning using automated apparatus - Google Patents

Abstract

FIELD: controlling.

SUBSTANCE: technical solution relates to the field of computer technology, in particular, to control of automated apparatus for indoor cleaning. The claimed technical result is achieved by implementing a method for indoor cleaning using automated cleaning apparatus, including the following stages: using the processor of a computing apparatus to obtain at least images of the floor of the room from at least one video surveillance camera; building the fouling map of the room based on the images obtained, wherein the map is generated using a machine learning model trained on room fouling images of; analysing the client flow at the time of generation of the fouling map; sending the generated fouling map and the client flow data to at least one automated cleaning apparatus; using at least one automated cleaning to obtain data from the computing apparatus; forming a cleaning route of the room based on the obtained fouling map and customer flow; cleaning the room.

EFFECT: higher efficiency of indoor cleaning using automated devices due to the dynamically built fouling map of the room and the customer traffic data used when forming the route of movement of automated cleaning apparatus.

10 cl, 5 dwg

Description

FIELD OF TECHNOLOGY

[0001] This technical solution relates to the field of computer technology, in particular the control of automated cleaning devices.

BACKGROUND OF THE INVENTION

[0002] Existing solutions in the field of cleaning the premises using automated devices, as a rule, are directed to various ways of constructing a route for the movement of such devices, for example, robotic devices.

[0003] A device and a method for tracking completed cleaning work are known (patent RU 2727215 C2), which is a program management system for cleaning work to simplify cleaning surfaces and record cleaning work performed. This system contains a sensor means that allows you to register various cleaning parameters (contact pressure, etc.) in order to track the cleaning work performed.

[0004] Another solution discloses an automatic cleaning system, a cleaning robot and a method for controlling a cleaning robot (patent RU 2620236 C1), which is a robot cleaning control that receives movement instructions (including concentrated cleaning areas, restricted area input instructions, and etc.) from the user during cleaning operations.

[0005] A common problem of the known solutions is the insufficient speed of updating information about the cleaning control area, as well as the lack of analysis of the density of client traffic, which is critical for timely planning of the cleaning route in the premises while people are inside.

SUMMARY OF THE INVENTION

[0006] The claimed invention is aimed at solving the technical problem of insufficient control efficiency of automated cleaning devices due to untimely updating of information about the interior environment of the room.

[0007] The technical result consists in increasing the efficiency of cleaning the premises with the help of automated devices, due to the dynamic construction of a pollution map of the premises and data on the client flow used in the formation of the route of movement of automated cleaning devices.

[0008] The claimed technical result is achieved through the implementation of a method for cleaning a room using automated cleaning devices, comprising the steps of:

using the processor of the computing device receive at least images of the floor of the room from at least one video surveillance camera;

generating a map of indoor pollution based on the obtained images, wherein the map is generated using a machine learning model trained on the images of indoor pollution;

carry out the analysis of the client flow at the time of formation of the pollution map;

transmitting the generated pollution map and client flow data to at least one automated cleaning device; using at least one automated cleaning device, receiving data from a computing device;

form the route of cleaning the premises on the basis of the received pollution map and client flow;

cleaning the premises.

[0009] In one of the particular examples of the implementation of the method, data on the contamination of objects in the room is additionally obtained from the video surveillance camera.

[0010] In another particular example of the implementation of the method, when building a pollution map, the resulting image is segmented and processed by a machine learning model to determine the presence and degree of pollution for each segment.

[0011] In another particular example of the implementation of the method for the segments, at least one of: the priority of the cleaning segment, the cleaning intensity, the use of the detergent composition, the method of using the detergent or cleaning agent is determined.

[0012] In another particular example of the implementation of the method, the route of movement of the automated device for cleaning is formed based on the priority of cleaning segments.

[0013] In another particular example of the implementation of the method, the start of cleaning the premises begins at a given threshold value of the client flow and / or the level of contamination of one or more segments.

[0014] In another particular example of the implementation of the method for an automated cleaning device, a map of the room is additionally preliminarily formed.

[0015] In another particular example of the implementation of the method, a room map is generated based on data from at least one optical sensor installed on an automated device.

[0016] In another particular example of the implementation of the method, the pollution map is dynamically updated within a given time range.

[0017] In another particular example of the implementation of the method for cleaning a room, several automated cleaning devices are simultaneously used.

[0018] In another particular example of the implementation of the method, each automated tool calculates the cleaning route, taking into account the movement of other automated devices.

[0019] The claimed technical result is also achieved using a room cleaning system using automated cleaning devices, which contains:

a computing device comprising at least one processor and associated with at least one video surveillance camera;

at least one cleaning device controlled by a data link with a computing device;

and

using the processor of the computing device receive at least images of the floor of the room from at least one video surveillance camera;

generating a map of indoor pollution based on the obtained images, wherein the map is generated using a machine learning model trained on the images of indoor pollution;

carry out the analysis of the client flow at the time of formation of the pollution map;

transmitting the generated pollution map and client flow data to control at least one automated cleaning device;

using at least one automated cleaning device, receiving data from a computing device;

form the route of cleaning the premises on the basis of the received pollution map and client flow;

cleaning the premises.

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 illustrates a general view of the claimed system.

[0021] FIG. 2A illustrates an example of pollution mapping.

[0022] FIG. 2B illustrates an example machine learning model architecture.

[0023] FIG. 3 illustrates a block diagram of the claimed method.

[0024] FIG. 4 illustrates a circuit diagram of a computing device.

IMPLEMENTATION OF THE INVENTION

[0025] In FIG. 1 shows a general view of the claimed solution. The solution is based on the technology of analyzing images obtained from one or more video surveillance cameras (110) installed in the cleaning area (10). The cleaning area (10) is understood to be an open or closed area, such as a bank branch, a shop, a restaurant, etc. Information received from video surveillance cameras (software) is transmitted via a data transmission channel (120), for example, TCP / IP, to a computing device (130), which performs subsequent processing of the received data in terms of analyzing images of surfaces, in particular, the floor (102) for the subsequent formation of data on the degree of its pollution.

[0026] The video surveillance cameras (110) can be any suitable type of cameras capable of capturing video, such as IP cameras, PTZ cameras, and others. As part of the data link (120), any suitable type of communication can be used, in particular, the data transfer protocol supported by the cameras (110), which ensures the transfer of information from them to the computing device (130). Examples of this kind of communication type can be: TCP/IP, LAN, WLAN, WAN, PAN, Wi-Fi, Ethernet, etc.

[0027] As the computing device (130), for example, a personal computer, laptop, server, tablet, or the like can be used.

[0028] Based on the information from the cameras (110) processed by the device (130), a contamination map of the cleaning area (10) is formed, which is built on the basis of the obtained images characterizing the degree of contamination of the floor (102). It also takes into account information about the number and density of client traffic, calculated by recognizing the presence of people (101) in the cleaning area (10).

[0029] The resulting pollution map and client flow data are transmitted to an automated cleaning device (140), for example, a robot vacuum cleaner, robot cleaner, automated scrubber, etc., which forms a movement route to ensure the cleaning process of the area (10) in particular floor surfaces (102). The automated cleaning device (140) can be either a completely autonomous device (robot vacuum cleaner) or an operator-controlled device, such as a scrubber drier. In this case, the operator may be provided with information about the preferred travel route for effective floor cleaning (102), for example, using the image of the travel route on the display of the device (140).

[0030] In FIG. 2A shows the principle of generating a pollution map (200). A pollution map (200) is generated by floor image processing (102) with a machine learning model (210). Model (210) consists of several modules that include specialized detectors that are part of the video analytics system and are used to detect objects (and/or change objects - object movements, etc.) in the image and find the boundaries of these objects through the use of algorithms image processing. In particular, the detector module within the framework of the model (210) is used to identify people and is trained on an open dataset with people (COCO Dataset), enriched with images of people from CCTV cameras in various premises and in transport. The module responsible for classifying a person in an image, determining the presence of a floor under it and characterizing the purity of this floor, was trained on images of people cut out from images from surveillance cameras with the expansion of the boundaries of a person by 10% of the long side (usually height). Each of these images was assigned a class for the classification task, a sign of the presence of sex, and the degree of floor contamination from 0 to 1 with a step of 0.2.

[0031] Model (210) in one of the particular examples of implementation can be one or more neural networks, for example, a convolutional neural network (CNN), which receives an image from a video surveillance camera (110) as input. On FIG. 2B shows one example implementation of the machine learning model architecture (210). The input image (211) is processed using the detector module (new image), and the subsequent processing of the detection results in the convolutional neural network module (210), consisting of convolutional layers (212, 213), and fully connected modules responsible for generating model responses (214 ) - (216). Module (214) classifies the current action of the person, module (215) performs a binary classification based on the presence of gender in the frame, and module (216) performs a regression task to determine the floor contamination level from 0 to 1.

[0032] The overall pollution map (200) is dynamically generated by constantly scanning the degree of pollution of the floor (102) by processing images from the cameras (110) by the model (210). The pollution map (200) divides the floor image area (102) into segments (201), for which the degree of pollution is determined from 0 to 1 using the model (210). The result of the execution of the model (210) is an estimate of the level of contamination of the floor (102) under the person (11) and around him in each segment (201).

[0033] The data on the cleanliness of the floor (102) obtained at the output of the model (210) are added to the overall pollution map (200). To do this, the original image with the floor (102) is divided into m*n identical segments (201). Each segment (201) corresponds to a list of the last k floor contamination values for that area (k>=1). The current pollution level of a particular segment (201) is the average of all k values. The total current level of floor contamination (102) is considered to be the average value q of the most polluted segments (1 <=q <=m*n).

[0034] The image of a person and the area of the floor under him is formed based on the results of processing the frame by the detector and processed by the next module of the model. The following sequence of actions is applied to the resulting image:

1. Its image is fed to the module for determining the presence and degree of contamination of the floor of the model (210);

2. If there is no gender in the image, then the map is not updated. Further steps are indicated for the case when the gender in the image is found;

3. The area of the map is determined, in which the center of the image received from the detector falls;

4. A new value is added to the list of pollution values for the found area of the map;

5. If the number of elements in the list is greater than k, then the oldest pollution value is removed from the list;

6. The current pollution value of the area is the average value of the list items.

[0035] The proposed approach proposes to scan with a given frequency the floor contamination level (102) of all people (101) in the frame, and use the obtained data to compile/update the current contamination map (200). Also, the approach makes it possible to effectively estimate the level of floor contamination from video data with different scales of the cleaning area (10) on the frame, since it works with the results of the detector, which already have the same scale. Updating the pollution map (200) is carried out by changing the pollution values of the areas and can be represented as colors of the map pixels (200) on the corresponding segments (201) depending on the degree of their pollution.

[0036] The use of a pollution map allows you to configure the rules for updating the segments (201) of the floor (102) to quickly update information about the cleanliness in the cleaning area (10), as well as to obtain several values characterizing the cleanliness (for example, the value of the most dirty segment, the average value of five the most polluted segments, the average value of all segments, etc., which allows you to prioritize the places of cleaning during the subsequent construction of the route of movement of the automated device (140).

[0037] In FIG. 3 is a flowchart of the method (300) for cleaning a room. At the first stage (301), data from the camera (110), in particular, the image of the area of its control in the form of video, is transmitted to the computing device (130), which subsequently processes the data using the model (210) and forms the current pollution map (200 ) floor (102) in the cleaning area (10).

[0038] Next, at step (302), based on the received images from the camera (110), the density of the client flow is calculated as the number of recognized people (101) on the frames of the cleaning area at the current time. Traffic density is used to plan cleaning (including cleaning start time, cleaning route and modes, etc.) using rules based on threshold values (client traffic, pollution levels, etc.) or other methods. The received cleaning plans are transmitted via the transmission channel (120) together with the calculated pollution map (200) to the automated device (140) for cleaning.

[0039] At step (303), the program logic of the device (140), based on the generated pollution map (200) and data on the density of the client flow, forms the route of movement of the device (140). The route can be formed on the basis of segments (201) of the pollution map (200), for which the priority of their cleaning is determined, taking into account the possibility of free movement around the cleaning zone (10) in the presence of people (11). The route of movement of the device (140) can also be formed directly on the computing device (130) with subsequent transmission via the data link (120) of the finished route to the automated device (140) for cleaning.

[0040] The automated device (140) can also be equipped with sensors, such as LIDAR, optical sensor, sound sensor, and others, which additionally provide information about the environment, based on which the movement of the device (140) is corrected. Using the sensors of the device (140), a map of the cleaning area (10), i.e. of the movement room of the device (140) can be preliminarily formed in terms of obtaining information about the surrounding space in the room and its geometric characteristics. The floor segments (102) corresponding to the segments (201) of the pollution map (200) can be characterized by spatial coordinates that provide the determination of the route of movement of the device (140).

[0041] Additionally, for the segments (201), the intensity of cleaning, the use of the detergent composition for the specified segment, or the method of using the detergent or cleaning agent can be determined. The calculation of the concentration and composition of the products used is based on the density of the client flow and the obtained level of floor contamination (102) in the corresponding segment (201). For example, for light soil removal and maintenance cleaning of segments (201) with low customer traffic, robotic cleaning with mild alkaline detergents can be used. And for heavily soiled segments (201) - manual cleaning with strongly alkaline detergents.

[0042] At step (304), the device (140) extends along the generated movement route to ensure cleaning of the area (10). As a rule, robotic means directly clean the floor (102), in particular wet and/or dry cleaning. Additionally, based on data from the camera (110), surfaces of elements of the environment, for example, ATMs (103) in a bank branch, which are also subject to contamination, can be determined. Such data can also be used in the formation of a pollution map (200) and transmitted both to devices (140) performing offline cleaning and augmented reality (AR-glasses) worn by service personnel. In this case, such surfaces will be highlighted with the help of their pixel processing and will allow you to respond more quickly to the need for cleaning.

[0043] When extending automated devices (140) into the cleaning area (10), the current density of the client flow is checked and, if the number of people (11) is higher than the specified threshold value, then the device (140) will expect a decrease in this indicator, which will fix during iterative processing by the computing device (130) of data from the camera (110).

[0044] In one of the particular examples of the implementation of the claimed solution, several automated devices (140) can be used, for example, a cleaning robot and a vacuum cleaner robot. In this case, when forming the route of their movement, both the density of the client flow is taken into account, on the basis of which the actual possibility of their movement simultaneously with people (101) in the cleaning area (10) and the order of their floor processing (102) will be determined based on the segments ( 201). As an example, more polluted segments (201) will first be subjected to wet cleaning, and then dry cleaning, which allows you to create a route for several devices (140) in sequence.

[0045] In FIG. 4 shows a general view of a computing device (400) suitable for performing the claimed method and on the basis of which a computing device (130) can be implemented. The device (400) may be, for example, a server or other type of computing device that can be used to implement the claimed technical solution. Including being part of a cloud computing platform.

[0046] In general, the computing device (400) contains one or more processors (401) connected by a common information exchange bus, memory means such as RAM (402) and ROM (403), input / output interfaces (404), input devices /output (405), and a device for networking (406).

[0047] The processor (401) (or multiple processors, multi-core processor) may be selected from a variety of devices currently widely used, such as Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, and etc. The processor (401) can also be a graphics processor such as Nvidia, AMD, Graphcore, etc.

[0048] RAM (402) is a random access memory and is designed to store machine-readable instructions executable by the processor (401) to perform the necessary data logical processing operations. The RAM (402) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.).

[0049] A ROM (403) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0050] Various types of I/O interfaces (404) are used to organize the operation of device components (400) and organize the operation of external connected devices. The choice of the appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, Fire Wire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro , mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.[0051] To ensure user interaction with the computing device (400), various means (405) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, a touchpad, trackball, speakers, microphone, augmented reality tools, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

[0052] The networking means (406) enables the communication of data by the device (400) via an internal or external computer network, such as an Intranet, the Internet, a LAN, and the like. As one or more means (406) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others

[0053] Additionally, satellite navigation tools in the device (400) can also be used, for example, GPS, GLONASS, BeiDou, Galileo.

[0054] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (24)

1. A method for cleaning a room using automated cleaning devices, comprising the steps of: using the processor of a computing device receiving at least images of the floor of the room from at least one video surveillance camera; a map of indoor pollution is formed based on the received images, and the map is generated using a machine learning model trained on images of indoor pollution, while the map is dynamically generated during periodic scanning of the degree of floor pollution, during the construction of which the images received from the cameras are processed to break down the floor image area into segments, for each of which the presence and degree of pollution is determined; carry out the analysis of the client flow at the time of formation of the pollution map; transmitting the generated pollution map and client flow data to at least one automated cleaning device; with at least one automated cleaning device receive data from the computing device; a route for cleaning the premises is formed on the basis of the obtained pollution map and client flow, while the route is formed on the basis of segments of the pollution map characterizing the cleaning zones; cleaning the premises. 2. The method according to claim 1, characterized in that, in addition, data on the contamination of objects in the room are received from the video surveillance camera. 3. The method according to claim 1, characterized in that at least one of the following is determined for the segments: the priority of the cleaning segment, the intensity of cleaning, the use of the detergent composition, the method of using the detergent or cleaning agent. 4. The method according to claim 3, characterized in that the route of movement of the automated cleaning device is formed based on the priority of cleaning the segments. 5. The method according to claim 1, characterized in that the beginning of the cleaning of the premises begins at a given threshold value of the client flow and the level of pollution. 6. The method according to claim 1, characterized in that for an automated cleaning device, a map of the room is additionally generated in advance. 7. The method according to claim 6, characterized in that the map of the room is formed on the basis of data from at least one optical sensor installed on the automated device. 8. The method according to claim 1, characterized in that several automated cleaning devices are simultaneously used for cleaning the premises. 9. The method according to claim 8, characterized in that each automated means calculates the cleaning route taking into account the movement of other automated devices. 10. Cleaning system using automated devices for cleaning, containing a computing device containing at least one processor and associated with at least one video surveillance camera; at least one cleaning device controlled by a data link with a computing device; moreover, using the processor of the computing device receive at least images of the floor of the room from at least one video surveillance camera; a map of indoor pollution is formed based on the received images, and the map is generated using a machine learning model trained on images of indoor pollution, while the map is dynamically generated by periodically scanning the degree of floor pollution, during the construction of which the images received from the cameras are processed to break down the floor image area into segments, for each of which the presence and degree of pollution is determined; carry out the analysis of the client flow at the time of formation of the pollution map; transmitting the generated pollution map and client flow data to control at least one automated cleaning device; with at least one automated cleaning device receive data from the computing device; a route for cleaning the premises is formed on the basis of the obtained pollution map and client flow, while the route is formed on the basis of segments of the pollution map characterizing the cleaning zones; cleaning the premises.

Images