KR102619900B1 - Cleaning robot system and controlling method thereof - Google Patents

Abstract

According to one embodiment, a cleaning robot; A control method of a cleaning robot system including a server that controls the cleaning robot. The server generating a first movement path of the cleaning robot based on drawing information and transmitting the first movement path to the cleaning robot; The cleaning robot performs cleaning based on the first movement path and travels the first movement path; transmitting a first image captured by the cleaning robot of the inside of the duct to the server; The server matching the drawing information and the first image and determining the location of the cleaning robot; Based on the position of the cleaning robot, the server generates a second movement path that returns to the movement path, and transmitting the second movement path to the cleaning robot; The cleaning robot transmits a second image captured inside the duct to the server based on the second movement path and travels the second movement path; and the server inputting the first image and the second image into an artificial neural network and evaluating cleanliness through the artificial neural network.

Description

Cleaning robot system and its control method {CLEANING ROBOT SYSTEM AND CONTROLLING METHOD THEREOF}

This embodiment relates to a cleaning robot system and its control method that determines the duct cleaning results of the cleaning robot based on artificial intelligence.

Republic of Korea Patent Publication KR 10-1190093 B1 discloses a remote control robot for duct cleaning with improved cleaning efficiency. Specifically, the prior literature includes a cart that is remotely operated and controlled to move within a duct; a high-pressure air injection nozzle installed in front of the cart to spray high-pressure air toward the front of the cart; a cleaning unit located in front of the cart and equipped with a cleaning brush that rotates to remove foreign substances from the inner wall of the duct; As the remote control robot for duct cleaning moves forward, high-pressure air is first blown toward the front floor by the high-pressure air injection nozzle, and dust clumps accumulated on the inner wall of the duct in the area to be cleaned are primarily moved forward. After moving, a remote control robot for duct cleaning is started that rotates the cleaning brush of the cleaning unit and secondarily cleans the inner wall of the duct with mechanical force.

Additionally, Republic of Korea Patent Publication KR 10-1072908 B1 discloses a remote control robot cleaning system that can easily check the cleaning status. Specifically, the prior literature includes a cart that is controlled and moves remotely inside a duct, a cleaning unit that is coupled to a link rotatably mounted on the cart to clean the inner surface of the duct, and the cart is A cleaning robot equipped with a front camera that photographs the front as it moves and a rear camera that photographs the rear that the cart has passed; a control unit that remotely controls movement of the cart and rotation of the link; a display unit that simultaneously displays a front image captured by the front camera and a rear image captured by the rear camera on two or more split screens; It is configured to include a storage unit that stores images displayed on the display unit, so that the worker controlling the cleaning robot can check the state before cleaning captured by the front camera and the state after cleaning photographed by the rear camera through the display unit. Since the duct cleaning is carried out while cleaning, not only can the inside of the duct that was not properly cleaned during cleaning be cleaned, but the storage unit stores the images taken by the cleaning robot's front and rear cameras on a recording medium, allowing the cleaning robot to be controlled. We are launching a remote control robot cleaning system that allows anyone, even if not the operator, to check the cleaning status of the entire duct as if it were being cleaned in real time even when the duct cleaning is completed.

However, these prior documents only provide simple images to control the robot's cleaning operation or check the cleaning status in real time, and do not disclose a technology for reviewing the cleaning results while moving the duct performed by the cleaning robot again. No.

Republic of Korea Patent Publication KR 10-1190093 B1 Republic of Korea Patent Publication KR 10-1072908 B1

The present invention is intended to solve the problems described above, and aims to provide a cleaning robot system and a control method thereof that compare images taken of the duct state while moving inside the duct after cleaning the inside of the duct, and evaluate the cleanliness. do.

Additionally, embodiments utilize tags that identify the inner area of the duct, so that images taken by cleaning robots can be clearly compared and the accuracy of evaluating cleanliness can be increased.

A cleaning robot according to an embodiment of the present invention to solve the above problems; A control method of a cleaning robot system including a server that controls the cleaning robot. The server generating a first movement path of the cleaning robot based on drawing information and transmitting the first movement path to the cleaning robot; The cleaning robot performs cleaning based on the first movement path and travels the first movement path; transmitting a first image captured by the cleaning robot of the inside of the duct to the server; The server matching the drawing information and the first image and determining the location of the cleaning robot; Based on the position of the cleaning robot, the server generates a second movement path that returns to the movement path, and transmitting the second movement path to the cleaning robot; The cleaning robot transmits a second image captured inside the duct to the server based on the second movement path and travels the second movement path; and the server inputting the first image and the second image into an artificial neural network and evaluating cleanliness through the artificial neural network.

Driving along the first movement path and the second movement path; recognizing, by the cleaning robot, tag information provided in the duct; It may include photographing the first image or the second image based on the tag information.

The step of evaluating the cleanliness includes matching and comparing the first image or the second image with each other based on the tag information; setting a region of interest in the first image based on pixel luminance; calculating pixel luminance of the second image in a corresponding region of interest of the second image; Comparing the difference between the pixel luminance of each region of interest in each image with a preset reference value; And, as a result of the comparison, inputting at least one of the first image or the second image into the artificial neural network, and evaluating the cleanliness based on the output value of the artificial neural network.

Driving the first movement path and the second movement path; includes the cleaning robot acquiring environmental information of the duct measured through a sensor group and transmitting the environmental information to the server. and evaluating the cleanliness; matching and comparing the first image and the second image based on the tag information; the server recognizing contaminants in the first image through the environmental information and the artificial neural network; the server performing labeling of the contaminant based on pixel information of the first image; and determining, by the server, the size of the contaminant and whether to remove it based on pixel information of the matched second image.

Driving the first movement path; determining, by the server, an area that needs additional cleaning based on the first image; Based on the area requiring additional cleaning, the server generates a cleaning control command and transmits the cleaning control command to the cleaning robot; It may include performing, by the cleaning robot, a predefined cleaning operation based on the control command.

According to another disclosed embodiment, a server that generates a movement path of the cleaning robot based on drawing information, generates a control command of the cleaning robot, and trains an artificial neural network; A cleaning robot including a camera, a moving part, and a cleaning part; and a user terminal that displays the image captured by the camera, wherein the cleaning robot travels inside the duct based on the first movement path of the cleaning robot generated by the server, and a second device that photographs the inside of the duct is provided. 1 Transmit an image to the server, drive again inside the duct based on the second movement path generated by the server, and transmit the second image taken on the second movement path to the server, and the server , the first image and the second image are input into an artificial neural network, cleanliness is evaluated through the artificial neural network, and the user terminal always displays the first image, the second image, or the cleanliness.

According to the present invention, the cleaning robot system and its control method can compare images taken of the duct state while moving inside the duct again after cleaning the inside of the duct, and evaluate cleanliness.

In addition, according to the present invention, the cleaning robot system and its control method can clearly compare images taken by the cleaning robot and increase the accuracy of evaluating cleanliness by utilizing a tag that identifies the inner area of the duct.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram for explaining the configuration of a cleaning robot system according to an embodiment.
Figure 2 is a flow chart to explain the operation of the server performed in the cleaning robot system.
Figure 3 is a flowchart for explaining the operation of the cleaning robot in the cleaning robot system.
FIG. 4 is a diagram illustrating a method of evaluating cleanliness by comparing a first image and a second image according to an embodiment.
Figure 5 is a flow chart for one embodiment of evaluating cleanliness.
Figure 6 is a flow chart for an embodiment of determining cleanliness through environmental information.
Figure 7 is a flowchart for explaining learning of an artificial neural network according to an embodiment.
Figure 8 is an exemplary diagram of the configuration of a device according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms. The present embodiments only serve to ensure that the disclosure of the present invention is complete and that common knowledge in the technical field to which the present invention pertains is not limited. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are the same as those commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. It has meaning. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the embodiments of the present invention, have an ideal or excessively formal meaning. It is not interpreted as

The shapes, sizes, proportions, angles, numbers, etc. disclosed in the drawings for explaining embodiments of the present invention are illustrative, and the present invention is not limited to the matters shown. Additionally, in describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. When 'includes', 'has', 'consists of', etc. mentioned in this specification are used, other parts may be added unless 'only' is used. When a component is expressed in the singular, the plural is included unless specifically stated otherwise.

When interpreting a component, it is interpreted to include the margin of error even if there is no separate explicit description.

In the case of a description of a positional relationship, for example, if the positional relationship of two parts is described as 'on top', 'on the top', 'on the bottom', 'next to', etc., 'immediately' Alternatively, there may be one or more other parts placed between the two parts, unless 'directly' is used.

When an element or layer is referred to as “on” another element or layer, it includes instances where the element or layer is directly on top of or intervening with another element. Like reference numerals refer to like elements throughout the specification.

The size and thickness of each component shown in the drawings are shown for convenience of explanation, and the present invention is not necessarily limited to the size and thickness of the components shown.

Each feature of the various embodiments of the present invention can be partially or fully combined or combined with each other, and as can be fully understood by those skilled in the art, various technical interconnections and operations are possible, and each embodiment may be implemented independently of each other. It may be possible to conduct them together due to a related relationship.

Hereinafter, the operating principle and embodiments of the present invention will be described with reference to the attached drawings.

1 is a diagram for explaining the configuration of a cleaning robot system according to an embodiment.

A cleaning robot system according to an embodiment may include a cleaning robot 1, a server 100 that controls the cleaning robot, and a user terminal 110. The cleaning robot system can function to precisely and safely clean the inside of ducts and pipes by controlling the cleaning robot (1).

First, the server 100 may be an own server owned by a user or organization that manages the cleaning robot 1, may be a cloud server, or may be a p2p (peer-to-peer) set of distributed nodes. there is. The server 100 may be configured to perform all or part of the calculation function, storage/reference function, input/output function, and control function of a typical computer. The server 100 may be equipped with at least one artificial neural network that performs an inference function. The server 100 may be linked with a web page or application for the user of the user terminal 110. The server 100 may be configured to communicate wired or wirelessly with the communication unit 40 of the cleaning robot 1 and the user terminal 110.

The server 100 may obtain drawing information of the duct or pipe to be cleaned. The server 100 may calculate the movement path of the cleaning robot 1 based on the drawing information and transmit the movement path to the cleaning robot. Through this, the cleaning robot 1 can determine the cleaning movement line. The cleaning robot 1 can clean while automatically traveling inside a duct or pipe.

The server 100 may control the cleaning robot 1 to photograph the inside of the duct through a movement path (hereinafter referred to as a first movement path) along which the cleaning robot 1 travels inside the duct before cleaning. The image captured by the cleaning robot 1 (hereinafter referred to as the first image) is transmitted to the server 100. After the cleaning robot 1 completes the first movement path, the server 100 controls the cleaning robot 1 to travel on a movement path (hereinafter referred to as a second movement path) that returns to the first movement path. The server 100 may determine the cleanliness within the duct or pipe by comparing the image (hereinafter referred to as the second image) transmitted by the cleaning robot 1 while traveling through the second movement path and the first image. A detailed description of this will be provided later through other drawings below.

The user terminal 110 may be a desktop computer, laptop, tablet, smartphone, etc. Alternatively, the user terminal 110 may be a professional computer device for controlling the cleaning robot 1. The user terminal 110 can access a web page operated by a user or organization that manages the cleaning robot 1. The user terminal 110 may be installed with an application developed and distributed by a user or organization that manages the cleaning robot 1. The user terminal 110 can communicate with the server 100 wired or wirelessly through a web page or application. A web page or application can display the situation inside the duct obtained through the cleaning robot (1). A web page or application may display a control command input UI (User Interface) for controlling the cleaning robot 1.

The cleaning robot 1 may have functions and configurations for cleaning the inside of ducts and pipes. Specifically, the cleaning robot 1 includes a moving part 10 that moves the cleaning robot inside the duct, cleaning parts 21 and 22 that function to clean the inside of the duct, a camera 31 that photographs the inside of the duct, and a camera 31 that takes pictures of the inside of the duct. It may include a sensor group 32 including a reader that measures the environment and collects tag information, and a communication unit 40 that transmits and receives data with the aforementioned server 10 or user terminal 110.

Specifically, the moving unit 10 provides power to the cleaning robot 1 and allows it to travel. The moving unit 10 may be composed of a plurality of crawler tracks and orbital motors.

If there is a subtle distance error inside the duct or pipe, the cleaning robot (1) detects whether the pressure is above or below a certain pressure through the plurality of pressure sensors (11) installed on the orbital motor of the moving part (10) and moves and rotates the section. can be corrected. Specifically, each pressure sensor 11 can measure the pressure of an area where the cleaning robot 1 contacts the duct or pipe. When the cleaning robot 1 rotates in the elbow section of a duct or pipe, the plurality of pressure sensors 11 may acquire pressure for each contact area. The cleaning robot 1 can transmit pressure for each contact area to the server 100. The server 100 may correct the first movement path of the cleaning robot 1 in the elbow section of the duct or pipe based on the difference in pressure for each contact area. Through this, the cleaning robot 1 can pass through the rotating/bending section without being swept or collided with one side of the duct or pipe.

Meanwhile, the moving unit 10 is not necessarily limited to the above-described configuration and may further include various moving means.

The cleaning units 21 and 22 may include a brush 21 and a blaster 22. The brush 21 may perform the function of wiping the inside of the duct. The blaster 22 can fire (blast) water and detergent into the duct. The water fired from the blaster 22 may be hot water. By blasting water from the blaster 22, the cleaning robot 1 can create a temperature environment suitable for cleaning inside the duct.

The camera 31 photographs the inside of the duct. The captured first and second images may be transmitted to the server 100. Specifically, the first image is the result of taking pictures of the inside of the duct before cleaning, so it may include various images such as contaminants, pests, grease, oil, etc. The server 100 may determine the presence or absence of cleaning based on the luminosity of each pixel of the first image, and may transmit a control command to the cleaning robot 1 to perform one or more blasting if insufficient. Additionally, the server 100 evaluates cleanliness by comparing the image (second image) and the first image of the inside of the duct taken while the cleaning robot 1 travels along the second movement path. As an example, the server 100 may set a region of interest based on the pixel luminance of the first image and calculate the luminance of pixels within the region of interest in the second image. The server 100 may evaluate the degree of cleaning, that is, cleanliness, of the cleaning robot 1 based on the calculated pixel luminance. Various methods of evaluating cleanliness through images are described later through other drawings below.

The server 100 may create an augmented reality layer to be added to the image based on the first image captured through the camera 31. The server 100 may transmit the actual image and augmented reality layer to the user terminal 110. The user of the user terminal 110 can more quickly and intuitively understand the environment inside the duct by checking the image inside the duct overlaid with the augmented reality layer.

The sensor group 32 of the cleaning robot 1 can measure the physical/chemical condition inside the duct. The sensor group 32 may be composed of a plurality of sensors. The sensor group 32 may include a temperature sensor, a fine dust sensor, a sensor that detects a specific chemical substance, and a reader that collects tag information. The server 100 determines pollutants such as toxic substances and pests (hereinafter referred to as environmental information) in the first image based on the detection value measured by the sensor group 32. The server 100 determines cleanliness by determining whether contaminants are removed from the first image and the second image.

The server 100 may create a translucent layer to be added to the image based on environmental information measured through the sensor group 32. The user of the user terminal 110 can more quickly and intuitively understand the internal environment of the duct by checking the image overlaid with a translucent layer.

The cleaning robot 1 and the server 100 can communicate wired or wirelessly through the communication unit 40. Specifically, the communication unit 40 may transmit the image captured by the camera 31 and the measured value of the sensor group 32 to the server 100 or the user terminal 110. The user can forcibly adjust the cleaning intensity of a specific area while viewing the image inside the duct based on the image transmitted through the communication unit 40. For example, the user may determine that the cleaning robot 1 enters the uncleaned area through the captured first image. If an uncleaned area enters the blasting radius, the user can slow down or stop and then enter a force control command to direct additional blasting of that area. The user terminal 110 transmits a control command to the cleaning robot 1, and the communication unit 40 can perform operations based on the received control command.

The server 100 may also obtain a control command of the user terminal 110. The server 100 may modify the first movement path based on the control command input from the user terminal 110 and transmit the modified first movement path to the cleaning robot 1.

The communication unit 40 may include one or more components that enable communication with the server 100 or the user terminal 110. For example, it may include at least one of a short-range communication module, a wired communication module, and a wireless communication module, and in addition to the examples described above, various data can be exchanged with the outside.

Meanwhile, the cleaning robot 1 may further include various configurations in addition to the configuration described above.

Figure 2 is a flow chart to explain the operation of the server performed in the cleaning robot system.

Referring to FIG. 2, the server 100 creates a first movement path based on duct drawing information (200).

Specifically, the server 100 may obtain drawing information from a duct drawing library that has been previously databased. The server 100 may receive a drawing file held by a user or organization requesting duct cleaning from an external network.

Upon receiving the drawing information, the server 100 calculates the distance that the cleaning robot 1 will go straight in the straight section based on the drawing information, and the cleaning robot 1 starts rotating in the rotation section such as the elbow section. You can calculate the starting point, radius of rotation, end point of rotation, etc. The server 100 may calculate the cleaning robot 1 to rotate gradually or continuously depending on the geometry of the elbow section. Based on the results of this calculation, the server 100 creates a first movement path along which the cleaning robot 1 travels inside the duct.

The server 100 may generate a first movement path by receiving a cleaning start point from a user's input command as well as drawing information on the inside of the duct. The user's input command is received from the user terminal 110. Additionally, the server 100 may transmit the first movement path generated based on the drawing information to the user terminal 110 and request whether to change the first movement path from the user.

The server 100 collects pressure for each contact area based on the detection value of the pressure sensor (210).

Specifically, the server 100 transmits the first movement path to the cleaning robot 1, and the cleaning robot 1 starts traveling based on the received first movement path. While driving, the cleaning robot (1) collects pressure for each contact area. Each pressure sensor 11 can measure the pressure of the area where the cleaning robot 1 contacts the duct. Each pressure sensor 11 may be located at a different location on the surface of the cleaning robot 1 where the cleaning robot 1 contacts the duct. Each pressure sensor 11 can measure the pressure for each contact area between the cleaning robot 1 and the duct. In a straight section, the pressure values measured through the plurality of pressure sensors 11 may differ only by less than a predefined range. In the elbow section, each pressure value measured through the plurality of pressure sensors 11 may differ by more than a predefined range. The cleaning robot 1 may transmit the pressure for each contact area measured through the plurality of pressure sensors 11 to the server 100.

The server 100 modifies the first movement path in the rotation section based on the pressure difference (220) and transmits the modified first movement path to the cleaning robot (230).

For example, based on the drawing information, the cleaning robot 1 may have previously received a first movement path from the server 100 to start turning to the left 1 m ahead. At this time, when the cleaning robot 1 has advanced 97 cm, the pressure per contact area of the pressure sensor located on the right may be reduced to a predefined range or more than the pressure per contact area of the pressure sensor located on the left. In this case, the cleaning robot 1 should turn to the left in front of 1 m, but in reality, it may be correct to move forward 97 cm in front. If the cleaning robot (1) goes 1m ahead and turns to the left, the right side of the cleaning robot (1) may hit or be swept by the duct. Frequent rubbing or collision with the duct may reduce the lifespan of the cleaning robot (1) or cause a breakdown of the cleaning robot (1). To solve this problem, the server 100 can calculate the difference in pressure for each contact area obtained in real time from the cleaning robot 1. When the pressure difference for each contact area begins to differ beyond a predefined range, the server 100 may transmit a control command to correct the movement path in the predefined rotation section to the cleaning robot 1. The predefined rotation section may be an elbow section of the duct.

The server 100 collects tag information and the first image from the cleaning robot 1 (240).

Specifically, tag information can be collected from an RFID (Radio Frequency IDentification) tag previously provided inside the duct through a reader provided in the sensor group 32 provided in the cleaning robot 1. Here, the RFID tag includes an IC chip, and the reader may be composed of an antenna that reads the identification code or information of the IC chip and other packaging. RFID tags can be provided at preset distances inside the duct or at rotation points. When the cleaning robot 1 reads the tag information, it photographs the inside of the duct through the camera 31. The cleaning robot 1 may transmit the captured first image and the read tag information to the server 100 or the user terminal 110 through the communication unit 40.

The server 100 determines the current location of the cleaning robot (250).

Specifically, the drawing information may include the location where the IC chip is installed, and the server 100 provides location information where the cleaning robot 1 is located in the duct based on the received tag information or the GPS signal transmitted by the cleaning robot 1. can be figured out.

The server 100 creates a second movement path using the modified first movement path, current location, and drawing information (260).

For example, the server 100 may determine that the cleaning robot 1 has currently completed the inside of the duct based on the transmitted tag information and location information. If the cleaning robot 1 completes the first movement path, the server 100 creates a second movement path that moves inside the duct again based on the tag information, the modified first movement path, and the drawing information. That is, the second movement path may be a movement path in which the cleaning robot 1 will not collide with the duct based on the modified first movement path, without the need to modify the movement path by the pressure sensor 11 again.

The server 100 again collects tag information and the second image from the cleaning robot 1 (270).

As described above, tag information is provided inside the duct. The cleaning robot 1 moves back through the duct based on the second movement path and takes pictures of the inside of the duct again through the camera 32. The image captured by the camera 32, that is, the second image, is transmitted to the server 100.

The server 100 matches the first image and the second image (280).

Each image is matched based on the area where the tag information transmitted by the cleaning robot 1 matches. If there is an image that does not clearly match from the tag information, the server 100 performs matching by referring to the location information of the cleaning robot 1, etc.

The server 100 evaluates cleanliness through an artificial neural network (290).

Cleanliness can be evaluated through various methods. Specifically, the server 100 evaluates cleanliness by comparing the matched first image and the second image. If the luminance of the pixel of the first image taken before cleaning is dark, the server 100 may evaluate that the inside of the duct is full of dust. If the pixel brightness of the same area in the matched second image becomes brighter, the server 100 evaluates the cleanliness as high based on the brightening degree. If the environmental information detected by the second image and sensor group 32 is changed compared to the environmental information detected by the first image and sensor group 32, the server 100 determines that the cleanliness has increased based on the degree of change. You can judge.

Meanwhile, the server 100 may use an artificial neural network to comprehensively consider evaluation factors of cleanliness with various levels of accumulation of insects, contaminants, toxic substances, and dust. The artificial neural network may be prepared as a Convolution Neural Network (CNN) or/and a Recurrent Neural Network (RNN) for recognizing the first image and the second image. Specifically, the artificial neural network may be composed of a multi-layer perceptron. The perceptron multiplies the input value by a weight, adds it with a bias, and then transfers the transformed input value. It can be composed of a function or activation function). In other words, the perceptron outputs only the transformed input value that satisfies the threshold value of the transfer function as an output value. The artificial neural network is trained to evaluate cleanliness by comparing the input first image and the second image, and if necessary, environmental information can be added to learn cleanliness for various situations. The artificial neural network learned in this way is stored in advance in the server 100, and the server 100 evaluates cleanliness based on the matched first image, second image, and environmental information. A detailed description of the artificial neural network will be described later in FIG. 7, etc.

Figure 3 is a flowchart for explaining the operation of the cleaning robot in the cleaning robot system. Among the matters described in FIG. 2, overlapping descriptions will be omitted or briefly explained.

Referring to FIG. 3, the cleaning robot 1 receives a first movement path from the server 100 (300).

As described above in FIG. 2, the first movement path may be generated from the server 100 based on drawing information inside the duct and an input command through the user terminal 110.

When the first movement path is received from the server 100, the cleaning robot 1 travels the first movement path (310).

The cleaning robot 1 receives the first movement path through the communication unit 40 and operates the movement unit 10 based on the first movement path.

The cleaning robot 1 collects environmental information or pressure through a sensor group (311), captures a first image through a camera 31 (312), and recognizes tag information (313).

Each of the above-described operations is an operation performed by the cleaning robot 1 while traveling along the first movement path, and the three operations may be performed simultaneously or in chronological order.

The cleaning robot 1 transmits the collected information, first image, and location information to the server (320).

The collected information, first image, and location information are converted into electrical signals and transmitted to the server 100 through the communication unit 40. In particular, the cleaning robot 1 can combine the time when the first image was captured, the time when tag information was collected, and location information and transmit it to the server 100 so that the server 100 can identify a plurality of information. there is.

The cleaning robot 1 continues to travel inside the duct based on the modified first movement path (330).

The server 100 transmits the pressure for each contact area to the server 100 based on the detection value collected through the pressure sensor 11 of the cleaning robot 1. The server 100 can correct the movement path of the cleaning robot 1 in the elbow section of the duct or pipe based on the difference in pressure for each contact area, and receives this corrected first movement path from the server 100. do.

The cleaning robot 1 determines whether it has completed traveling the first movement path (340).

Specifically, whether driving of the first movement path has been completed can be determined by whether the operation of the moving unit 10 has been completed according to the received first movement path. As another example, when receiving a control command to stop movement from the current location from the server 100 or the user terminal 110, the cleaning robot 1 may determine that the first movement path has been completed.

If the cleaning robot 1 does not complete traveling the first movement path (No in 340), the cleaning robot 1 continues traveling the first movement path and performs the cleaning operation and the operations of steps 311, 312, and 313. Perform.

If the cleaning robot 1 completes traveling the first movement path (example in 340), the cleaning robot 1 waits and then receives the second movement path from the server 100 (350).

The second movement path is a movement path that goes back inside the duct. The second movement path is created based on the tag information transmitted by the cleaning robot 1, the first movement path modified through the sense group 32, and drawing information.

The cleaning robot 1 travels a second movement path (360), captures a second image (361), and recognizes tag information (362).

Specifically, the cleaning robot 1 operates the moving unit 10 based on the second moving path, similar to driving on the first moving path, and photographs the inside of the duct through the camera 31. The captured second image is temporarily stored in the cleaning robot 1. Additionally, the cleaning robot 1 collects tag information through the sensor group 32. If an RFID tag is provided inside the duct, tag information can be read again on the second movement path.

The cleaning robot 1 transmits the collected information, second image, and location information to the server (370).

The server 100 may evaluate the cleanliness inside the duct based on the collected information. A detailed description of this will be provided later through other drawings below.

The cleaning robot 1 determines whether it has completed traveling the second movement path (380).

Specifically, whether the driving of the second movement path has been completed can be determined by whether the operation of the moving unit 10 has been completed according to the second movement path. Alternatively, when receiving a control command to stop movement from the current location from the server 100 or the user terminal 110, the cleaning robot 1 may determine that the second movement path has been completed.

If the cleaning robot 1 does not complete traveling the second movement path (No in 380), the cleaning robot 1 continues traveling the second movement path and performs the cleaning operation and the operations of steps 361 and 362. do.

If the cleaning robot 1 completes traveling the second movement path (example 380), the cleaning robot 1 stops operating and waits.

FIG. 4 is a diagram illustrating a method of evaluating cleanliness by comparing a first image and a second image according to an embodiment.

Referring to FIG. 4 , the first image 60 may be an example of a pre-cleaning image taken of the inside of a duct while the cleaning robot 1 travels the first movement path. Specifically, dust may accumulate on the ceiling, left side, and floor of the first image 60, and contaminants 52 may be captured.

The cleaning robot 1 can read the RFID tag 33 provided inside the duct through the reader included in the sensor group 32 and collect tag information. Additionally, the cleaning robot 1 can detect pollutants 52 provided inside the duct from environmental information through the sensor group 32.

The cleaning robot 1 may transmit the first image 60, environmental information, and tag information to the server 100 in the embodiment shown in FIG. 4.

The second image 61 is an image captured by the cleaning robot 1 at a position close to the position where the first image was captured while the cleaning robot 1 was traveling inside the duct based on the second movement path.

Since the cleaning robot 1 has performed cleaning through the cleaners 21 and 22, the second image 61 shows the dust removed and the contaminants in the area of interest 51 where the contaminants 52 were in the first image. This may include what has been removed.

The cleaning robot 1 can read the RFID tag 33 provided inside the duct through the reader included in the sensor group 32 and collect tag information. Additionally, the cleaning robot 1 can again detect the presence of contaminants 52 provided inside the duct through the sensor group 32 and transmit the detection result to the server 100.

The server 100 receives tag information, environment information, and the second image 61 from the cleaning robot 1, and matches the first image 60 and the second image 61 based on the tag information. After matching is performed, the server 100 may evaluate cleanliness based on environmental information, the first image 60, and the second image 61. A specific method for evaluating cleanliness will be described later through other drawings below.

Figure 5 is a flow chart for one embodiment of evaluating cleanliness.

Referring to FIG. 5, the server 100 matches the first image and the second image based on tag information (400).

The first image 60 and the second image 61 are images transmitted by the cleaning robot 1. The server 100 matches images to each other based on tag information. However, the first image 60 captured by the cleaning robot 1 on the first movement path and the second image 61 captured by the cleaning robot 1 on the second movement path are left and right opposite each other, as shown in FIG. 4. It may be in an inverted form, and may be a different image in which contaminants are removed by the operation of the cleaners 21 and 22. The server 1 recognizes the location of the cleaning robot 1 inside the duct through tag information, and corrects the first image 60 or the second image 61 through location information such as a GPS signal to create each image. Match each other.

The server 100 sets a region of interest in the first image based on the luminance of the pixel (410).

As shown in FIG. 4 , the region of interest 51 may be created in an area of the first image 60 where luminance values included within a preset standard are crowded together. Here, the preset standard can vary depending on the degree of dust accumulation inside the duct, and there is no limit to the size and shape of the area of interest.

The server 100 calculates the luminance of a pixel in the region of interest of the second image (420).

The second image 61 is an image captured by the cleaning robot 1 after cleaning has already been performed. Accordingly, the luminance value of each pixel in the second image 61 may be different from the luminance value of the first image 60. Unlike step 410, the server 100 generates the region of interest 52 in the second image 61 based on the location coordinates of the region of interest 51 in the matched first image 60. The shapes of the regions of interest 51 and 52 may be the same as those created in the first image, as shown in FIG. 4 . The server 100 calculates pixel luminance in the region of interest 52 created in the second image.

The server 100 compares the difference between the pixel brightness of each region of interest 51 and 52 for each image with a reference value (430).

Specifically, the server 100 compares the difference between the pixel luminance of each interest region 51 and 52 for each image with a reference value. Here, the standard value may vary and may change in various ways depending on the environment inside the duct. The reference value may be changed through the user terminal 110.

Meanwhile, the standard for classifying luminance values is not necessarily limited to the region of interest. The server 1 may evaluate cleanliness by calculating the overall luminance value of the first image 60 and the second image 61 without creating a region of interest.

The server 100 inputs the comparison result into the artificial neural network (440), and the server 100 evaluates the cleanliness inside the duct based on the output value output by the artificial neural network (450).

Even when comparing images inside a simple duct as shown in FIG. 4, regions of interest can be created in very different areas, and luminance values can be compared in very different ways. The server 100 may improve the accuracy of cleanliness evaluation by not simply comparing pixel luminance values, which can be performed in various ways, in areas of interest to be created in various positions and shapes, but by comparing them using an artificial neural network.

Figure 6 is a flow chart for an embodiment of determining cleanliness through environmental information.

The server 100 recognizes pollutants in the first image through environmental information and an artificial neural network (500).

Environmental information is various detection values collected through the sensor group 32 of the cleaning robot 1. As described above in FIG. 1, the sensor group 32 may include a temperature sensor, a fine dust sensor, a sensor that detects a specific chemical substance, etc., and the server 100 synthesizes the detection values detected by each sensor. Thus, environmental information can be generated.

The server 100 may recognize contaminants (510).

The server 100 can recognize pollutants through the generated environmental information. For example, the server 100 may detect contaminants inside the duct through a specific chemical sensor. As another example, there may be a situation where toxic chemicals have spread inside the duct due to a toxic liquid. A sensor that detects specific chemicals in the cleaning robot (1) can detect toxic chemicals. The server 100 may determine a situation where the concentration of toxic chemicals inside the duct is higher than a predefined concentration.

The server 100 performs labeling of contaminants in the first image (520).

Specifically, the server 100 may distinguish pollutants in the first image through an artificial neural network. The artificial neural network can learn an object recognition algorithm or recognize contaminants from the first image through CNN or RNN. If the server 100 first recognizes a contaminant through the first image, the server 100 can specifically distinguish from environmental information whether the contaminant is a substance that interferes with cleaning, such as a poisonous substance or an insect. That is, the server 100 recognizes pollutants in the first image and labels the first image with the type of pollutant collected through environmental information.

The server 100 determines the size of the contaminant or whether to remove it from the second image (530).

The server 100 compares the matched first image and the second image. Specifically, the server 100 first recognizes contaminants in the second image using an artificial neural network. If there are no contaminants recognized in the second image, the server 100 evaluates the cleanliness as high. If a contaminant is recognized in the second image, the server 100 recognizes the contaminant in the second image through environmental information. If the contaminant recognized in the second image is the same as the contaminant labeled in the first image, the server 100 evaluates the cleanliness as low. If the contaminant recognized in the second image is different from the contaminant labeled in the first image, the server 100 determines whether it is due to a cause other than cleaning based on the environmental information and sends the cleanliness determination result to the user terminal. Send to (110).

Figure 7 is a flowchart for explaining learning of an artificial neural network according to an embodiment.

The artificial neural network may be included in the server 100 and may be learned through the server 100 or a separate learning device.

The artificial neural network may include both a first artificial neural network that evaluates cleanliness by comparing the first image and the second image, and a second artificial neural network that recognizes contaminants based on the first image and environmental information.

The learning device according to one embodiment may acquire training data and a label (600).

For learning the first artificial neural network, the learning device can acquire actual images inside the duct as training data. Additionally, the learning device can obtain results regarding the cleanliness inside the duct actually cleaned by the expert as labels corresponding to each training data.

For learning the second artificial neural network, the learning device can acquire environmental information inside the duct measured through temperature, fine dust, and chemical sensors as training data. Additionally, the learning device can obtain recognition results of contaminants inside the duct actually inspected by an expert as labels corresponding to each training data.

The learning device may generate input for an artificial neural network from training data (610).

The learning device can use the training data as is as an input to the artificial neural network or generate the input of the artificial neural network after undergoing known processing such as removing unnecessary information.

The learning device may apply the input to an artificial neural network (620).

The artificial neural network included in the server may be an artificial neural network learned according to supervised learning. The artificial neural network may be a convolutional neural network (CNN) or recurrent neural network (RNN) structure suitable for learning through supervised learning.

The learning device may obtain output from the artificial neural network (630).

The output of the first artificial neural network may be an inference about how high the level of cleanliness was achieved by determining the change in the second image compared to the first image. Specifically, the first artificial neural network is a change in the brightness of pixels inside a duct where dust has accumulated, a change in toxic liquid that is dangerous if it scatters during cleaning, a change in cleaning inside a stuck duct that blocks the flow of water or air, rats, cockroaches, etc. If there are living creatures such as insects, patterns such as changes in the cleaning interior can be learned, and the cleanliness level can be output by inferring changes in the first and second images based on the learning results.

The output of the second artificial neural network may be an inference about recognizing the pollutant itself. Specifically, the second artificial neural network learns patterns such as differences in the first image according to various types of pollutants and whether pollutants matching environmental information are recognized in the first image, and based on the learning results, Contaminants can be inferred from the image and output.

Afterwards, the learning device can compare the output and the label (640). The process of comparing the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be accomplished by calculating a loss function. As a loss function, conventionally known mean squared error (MSE), cross entropy error (CEE), etc. may be used. However, it is not limited to this, and loss functions used in various artificial neural network models can be used as long as the deviation, error, or difference between the output of the artificial neural network and the label can be measured.

The learning device can optimize the artificial neural network based on the comparison value (650).

By updating the weight of the nodes of the artificial neural network so that the learning device comparison value becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched, and through this, the artificial neural network can It can be optimized to output inferences that are close to the correct answer. Specifically, the learning device can optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value is closer to the estimate of the minimum value. To optimize the artificial neural network, known backpropagation algorithms, stochastic gradient descent, etc. can be used. However, it is not limited to this, and weight optimization algorithms used in various neural network models can be used.

A learning device can learn an artificial neural network by repeating this process.

Through this, the learning device optimizes the artificial neural network, enables the second artificial neural network to accurately recognize contaminants in the matched first image, and allows the first artificial neural network to accurately recognize pollutants through the matched first image and second image. Enables evaluation of high cleanliness.

Figure 8 is an exemplary diagram of the configuration of a device according to an embodiment.

Device 701 according to one embodiment includes a processor 702 and memory 703. The processor 702 may include at least one device described above with reference to FIGS. 1 to 7 or may perform at least one method described above with reference to FIGS. 1 to 7 . Specifically, the device 701 may be a cleaning robot 1, a server 100, a user terminal 110, or an artificial neural network learning device.

The memory 703 may store information related to the above-described methods or store a program in which the above-described methods are implemented. Memory 703 may be volatile memory or non-volatile memory.

The processor 702 can execute programs and control the device 701. The code of the program executed by the processor 702 may be stored in the memory 703. The device 701 is connected to an external device (eg, a personal computer or a network) through an input/output device (not shown) and can exchange data through wired or wireless communication.

Device 701 may be used to train an artificial neural network or use a trained artificial neural network. Memory 703 may include an artificial neural network that is being trained or has been trained. The processor 702 may learn or execute the artificial neural network algorithm stored in the memory 703. The device 701 for training an artificial neural network and the device 701 for using the learned artificial neural network may be the same or may be separate.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using one or more general-purpose or special-purpose computers, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may perform an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. Computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. A hardware device may be configured to operate as one or more software modules to perform the operations of an embodiment, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made without departing from the technical spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

1: Cleaning robot 10: Moving part
11: pressure sensor 21, 22: cleaner
32: Sensor group 40: Communication department
100: server 110: user terminal

Claims (6)

cleaning robot; In a control method of a cleaning robot system including a server that controls the cleaning robot,
The server generating a first movement path of the cleaning robot based on drawing information and transmitting the first movement path to the cleaning robot;
The cleaning robot performs cleaning based on the first movement path and travels the first movement path;
transmitting a first image captured by the cleaning robot of the inside of the duct to the server;
The server determining the location of the cleaning robot based on the drawing information and the first image;
The server generating a second movement path that returns to the movement path based on the position of the cleaning robot, and transmitting the second movement path to the cleaning robot;
transmitting, by the cleaning robot, a second image of the inside of the duct to the server based on the second movement path, and traveling the second movement path; and
Comprising: the server inputting the first image and the second image into an artificial neural network, and evaluating cleanliness through the artificial neural network,
Driving the first movement path and the second movement path;
Recognizing tag information provided in the duct by the cleaning robot; and
A step of the cleaning robot photographing the first image and the second image based on the tag information,
The step of evaluating the cleanliness;
The server matching and comparing the first image and the second image based on the tag information;
setting, by the server, a region of interest in the first image based on pixel luminance of the first image;
the server calculating pixel luminance of the second image in a corresponding region of interest of the second image;
Comparing, by the server, the difference between pixel brightness of each region of interest in each image with a preset reference value; and
The server inputs the first image and the second image as a result of the comparison into the artificial neural network, and evaluating the cleanliness based on the output value of the artificial neural network. delete delete According to paragraph 1,
Driving the first movement path and the second movement path;
The cleaning robot further includes obtaining environmental information of the duct measured through a sensor group and transmitting the environmental information to the server,
The step of evaluating the cleanliness;
The server matching and comparing the first image and the second image based on the tag information;
the server recognizing contaminants in the first image through the environmental information and the artificial neural network;
the server performing labeling of the contaminant based on pixel information of the first image; and
A method of controlling a cleaning robot system comprising: the server determining the size of the contaminant and whether to remove it based on pixel information of the second image. According to paragraph 1,
Driving the first movement path;
The server determining an area requiring additional cleaning based on the first image;
The server generating a cleaning control command based on the area requiring additional cleaning, and transmitting the cleaning control command to the cleaning robot;
A method of controlling a robot system comprising: allowing the cleaning robot to perform a predefined cleaning operation based on the control command.
delete