KR102328595B1 - A moving-robot and control method thereof - Google Patents

Abstract

The present invention provides a sensing unit for acquiring image information around a mobile robot; a control unit generating a control signal by using the image information; and a communication unit for transmitting the control signal to a terminal connected to the mobile robot by a wired/wireless communication method and receiving a terminal signal from the terminal, wherein the control unit extracts characteristic information of an unidentified user from the image information, and It is characterized in that it is determined whether it is an intruder based on the characteristic information.

Description

{A MOVING-ROBOT AND CONTROL METHOD THEREOF}

The present invention relates to a security-related technology of a mobile robot.

Robots have been developed for industrial use and have been a part of factory automation. Recently, the field of application of robots has been further expanded, and medical robots and aerospace robots have been developed, and household robots that can be used in general households are also being made. Among these robots, those capable of driving by their own force are called mobile robots. A typical example of a mobile robot used at home is a robot vacuum cleaner.

Various technologies for detecting an environment and a user around the mobile robot through various sensors provided in the mobile robot are known. Also, there are known technologies in which a mobile robot learns and maps a driving area by itself, and recognizes a current location on a map.

In particular, in the patent document, the robot cleaner records the surrounding environment with a camera, recognizes it with a sensor, avoids surrounding obstacles, and discloses cleaning at a reserved time.

However, in the prior art, there are problems in that the robot cleaner simply performs only a function limited to cleaning, does not have a function to detect an external intrusion in the absence of the user, and does not have a means for notifying the user of external intrusion.

In addition, the robot cleaner may recognize an object or a person, and the user may notify the movement of the object or the presence of a person in the room. However, in this case, when the indoor and the outside are separated by a transparent glass, etc., when moving from the outside without entering the room from the outside, it is determined that an outsider has entered the room and a warning can be transmitted to the user. In this way, the user is inconvenient to receive a warning even though an outsider has not intruded.

In other words, when the robot cleaner analyzes the image captured by the camera and simply recognizes an object or a person, it is not possible to know exactly whether this is an intrusion of an outsider, and it is not possible to determine the exact state of the object, so the accuracy of the alarm is lowered. this exists

Korean Patent Publication No. 20190015930

The problem to be solved by the present invention is that the robot cleaner detects that there is an external intrusion into the user's house while the user is absent, and informs the user of this.

Another problem to be solved by the present invention is whether an external intrusion into the user's home during the user's absence is recognized by people and things, and whether it is an intrusion of an outsider based on the direction of movement of people and things, or whether it is a situation to output a warning it can be judged

The present invention is characterized in that the robot cleaner analyzes the direction of the movement of an object or person to determine a situation in which an alarm should be generated.

Specifically, a camera for photographing an image related to the cleaning area in which the main body is located at every preset cycle; a memory for storing information related to a preset algorithm for analyzing the image; and detecting a difference between the first image, the second image, and the third image continuously photographed by the camera, specifying a first subject based on the detected difference, and detecting movement direction information of the first subject. It is characterized in that it includes a control unit.

The identification of the first subject may include detecting a difference between the first image and the second image, generating a first difference image based on the detected difference, and analyzing the generated first difference image to determine the second image. 1 Information related to the subject can be detected.

The controller may determine a degree of similarity between a plurality of pieces of image information included in the training data and the first difference image, and detect information related to the first subject based on the determined degree of similarity.

The control unit detects a difference between the second image and the third image, generates a second difference image based on the detected difference, and analyzes the generated second difference image to move the first subject direction information can be detected.

The present invention may further include a communication unit for communicating with the outside.

When the first subject is a person and the moving direction of the first subject is a preset moving direction, the controller may control the communication unit to transmit information related to the first subject to at least one of a server and a terminal.

The controller may control the terminal to output an image of the first subject when the first subject is a person and the moving direction of the first subject is a preset moving direction.

The controller may control the communication unit to transmit a warning message to the terminal when the first subject is a person and the moving direction of the first subject is a preset moving direction.

When the first subject is a door and the moving direction of the first subject is a preset moving direction, the controller may control the communication unit to transmit information related to the first subject to at least one of a server and a terminal.

The controller may control the communication unit to transmit a warning message to the terminal when the first subject is a door and the moving direction of the first subject is a preset moving direction.

The control unit is a mobile robot that controls the terminal to activate a reference information input screen for receiving reference information, which is a reference for outputting an alarm.

The present invention may further include a driving unit for moving the main body.

The controller may control the driving unit so that the first image, the second image, and the third image are acquired at the same point in the same camera direction.

When the monitoring driving of the cleaner is started, the controller controls the driving unit so that the main body traverses a plurality of preset points within the cleaning area, and the camera takes the first to third images at each of the plurality of points. can control

The controller may analyze the second image to specify the location of the first subject.

When the first subject is a person, the position of the first subject is within a boundary area, and the moving direction of the first subject is a preset movement direction corresponding to the boundary area, the control unit may be configured to relate to the first subject. The communication unit may be controlled so that information is transmitted to at least one of a server and a terminal.

The control unit may include, when the first subject is a person, the position of the first subject is within a boundary area, and a movement direction of the first object is a preset movement direction corresponding to the boundary area, the terminal is configured to control the first object You can control to output an image of a subject.

If the first subject is a person, the position of the first subject is within a boundary area, and the movement direction of the first object is a preset movement direction to correspond to the boundary area, a warning message is sent to the terminal. It is possible to control the communication unit to be transmitted.

The reference information may include type information of the first object corresponding to each boundary area and information on a movement direction of the first object.

Through the above solution, the present invention can detect a dangerous situation in the house by using various sensors of the robot cleaner without adding other separate equipment when the user is absent, and the user can recognize the state of the house. There are advantages to being able to.

In addition, the present invention allows the robot cleaner to report to the user when an outsider enters the home, and provides an image of the situation when an outsider intrudes to the user. And there is an advantage of being able to issue a command corresponding to a dangerous situation to the home appliance.

In addition, the present invention detects the moving direction of objects and people in the absence, and in the case of a person, accurately determines whether an outsider moves only from the outside or invades the inside, and misjudging an outsider moving from the outside as an intruder can prevent

In addition, the present invention has the advantage of preventing the erroneous determination of the opening of the door by detecting the opening direction of the door, the image change around the door.

1 is a perspective view showing an example of a mobile robot according to the present invention.
FIG. 2 is a plan view of the mobile robot shown in FIG. 1 .
FIG. 3 is a side view of the mobile robot shown in FIG. 1 .
4 is a block diagram illustrating components of a mobile robot according to an embodiment of the present invention.
5 is a conceptual diagram illustrating a system of a mobile robot according to an embodiment of the present invention.
6 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.
7 is a conceptual diagram illustrating a method in which a mobile robot detects a difference between a plurality of images according to an embodiment of the present invention.
8 is a conceptual diagram illustrating a method in which a mobile robot detects a difference between a plurality of images according to an embodiment of the present invention.
9 is a diagram illustrating a control screen of a terminal according to an embodiment of the present invention.
10 is a flowchart illustrating a method for controlling a mobile robot according to another embodiment of the present invention.
11 is a conceptual diagram illustrating a method in which a mobile robot detects a difference between a plurality of images according to another embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between components and other components. Spatially relative terms should be understood as terms including different orientations of components in use or operation in addition to the orientation shown in the drawings. For example, when a component shown in the drawings is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" means that a referenced component, step and/or action excludes the presence or addition of one or more other components, steps and/or actions. I never do that.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly specifically defined.

In the drawings, the thickness or size of each component is exaggerated, omitted, or schematically illustrated for convenience and clarity of description. In addition, the size and area of each component do not fully reflect the actual size or area.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a perspective view showing an example of a robot cleaner 100 according to the present invention, FIG. 2 is a plan view of the robot cleaner 100 shown in FIG. 1 , and FIG. 3 is a robot cleaner 100 shown in FIG. 1 . is a side view of

The mobile robot of the present invention may include a robot cleaner, and the present invention will be described below based on the robot cleaner.

For reference, in this specification, a mobile robot, a robot cleaner, and a cleaner performing autonomous driving may be used as the same meaning.

1 to 3 , the robot cleaner 100 performs a function of cleaning the floor while traveling on its own in a predetermined area. The cleaning of the floor referred to herein includes sucking in dust (including foreign matter) on the floor or mopping the floor.

The robot cleaner 100 includes a cleaner body 110 , a suction unit 120 , a sensing unit 130 , and a dust container 140 .

The cleaner body 110 includes a control unit for controlling the robot cleaner 100 and a wheel unit 111 for driving the robot cleaner 100 . By the wheel unit 111 , the robot cleaner 100 may be moved or rotated forward, backward, left and right.

The wheel unit 111 includes a main wheel 111a and a sub wheel 111b.

The main wheel 111a is provided on both sides of the cleaner body 110, and is configured to be rotatable in one direction or the other according to a control signal of the controller. Each of the main wheels 111a may be configured to be driven independently of each other. For example, each of the main wheels 111a may be driven by different motors.

The sub-wheel 111b supports the cleaner body 110 together with the main wheel 111a, and is configured to assist the driving of the robot cleaner 100 by the main wheel 111a]. This sub wheel 111b may also be provided in the suction unit 120 to be described later.

As described above, as the controller controls the driving of the wheel unit 111 , the robot cleaner 100 autonomously travels on the floor.

Meanwhile, a battery (not shown) for supplying power to the robot cleaner 100 is mounted on the cleaner body 110 . The battery is configured to be rechargeable, and may be configured to be detachably attached to the bottom of the cleaner body 110 .

The suction unit 120 is disposed to protrude from one side of the cleaner body 110, and is configured to suck air containing dust. The one side may be the side on which the cleaner body 110 travels in the forward direction F, that is, the front side of the cleaner body 110 .

In this figure, it is shown that the suction unit 120 has a shape protruding from one side of the cleaner body 110 to both the front and left and right sides. Specifically, the front end of the suction unit 120 is disposed at a position spaced forward from one side of the cleaner body 110 , and both left and right ends of the suction unit 120 are spaced apart from one side of the cleaner body 110 to the left and right, respectively. placed in the designated location.

As the cleaner body 110 is formed in a circular shape, and both sides of the rear end of the suction unit 120 protrude from the cleaner body 110 to the left and right sides, respectively, there is an empty space between the cleaner body 110 and the suction unit 120 . A space, ie a gap, may be formed.

The empty space is a space between the left and right both ends of the cleaner body 110 and the left and right ends of the suction unit 120 , and has a shape recessed inside the robot cleaner 100 .

When an obstacle is caught in the empty space, a problem in that the robot cleaner 100 is caught by the obstacle and cannot move may occur. To prevent this, the cover member 129 may be disposed to cover at least a portion of the empty space. The cover member 129 may be provided in the cleaner body 110 or the suction unit 120 . In the present embodiment, it is shown that the cover members 129 are formed to protrude from both sides of the rear end of the suction unit 120 , and are arranged to cover the outer circumferential surface of the cleaner body 110 .

The cover member 129 is disposed to fill at least a part of the empty space, that is, the empty space between the cleaner body 110 and the suction unit 120 . Accordingly, a structure in which an obstacle is prevented from being caught in the empty space or can be easily separated from the obstacle can be implemented even if the obstacle is caught in the empty space.

The cover member 129 protruding from the suction unit 120 may be supported on the outer peripheral surface of the cleaner body 110 . If the cover member 129 is formed to protrude from the cleaner body 110 , the cover member 129 may be supported on the rear surface of the suction unit 120 . According to the structure, when the suction unit 120 collides with an obstacle and receives an impact, a portion of the impact is transmitted to the cleaner body 110 so that the impact may be dispersed.

The suction unit 120 may be detachably coupled to the cleaner body 110 . When the suction unit 120 is separated into the cleaner body 110 , a mop module (not shown) may be detachably coupled to the cleaner body 110 to replace the separated suction unit 120 . Accordingly, the user may mount the suction unit 120 on the cleaner body 110 to remove dust from the floor, and mount the mop module on the cleaner body 110 to wipe the floor.

When the suction unit 120 is mounted to the cleaner body 110 , the mounting may be guided by the above-described cover member 129 .

That is, since the cover member 129 is disposed to cover the outer circumferential surface of the cleaner body 110 , the relative position of the suction unit 120 with respect to the cleaner body 110 may be determined.

A sensing unit 130 is disposed on the cleaner body 110 . As shown, the sensing unit 130 may be disposed on one side of the cleaner body 110 where the suction unit 120 is located, that is, in front of the cleaner body 110 .

The sensing unit 130 may be disposed to overlap the suction unit 120 in the vertical direction of the cleaner body 110 . The sensing unit 130 is disposed on the suction unit 120, and the suction unit 120 positioned at the front of the robot cleaner 100 is configured to detect an obstacle or a feature in the front so that it does not collide with the obstacle.

The sensing unit 130 is configured to additionally perform a sensing function other than the sensing function. This will be described in detail later.

A dust container accommodating part 113 is provided in the cleaner body 110 , and a dust container 140 for separating and collecting dust in the suctioned air is detachably coupled to the dust container accommodating part 113 . As shown, the dust container accommodating part 113 may be formed on the other side of the cleaner body 110 , that is, at the back of the cleaner body 110 .

A part of the dust bin 140 may be accommodated in the dust bin accommodating part 113 , and the other part of the dust bin 140 may be formed to protrude in the reverse direction R opposite to the front direction F of the cleaner body 110 . have.

The dust container 140 has an inlet 140a through which air containing dust is introduced and an outlet 140b through which the air separated from dust is discharged. 140a) and the outlet 140b are configured to communicate with the first opening 110a and the second opening 110b formed in the inner wall of the dust container accommodating part 113, respectively.

The intake flow path inside the cleaner body 110 corresponds to the flow path from the inlet (not shown) communicating with the communication part 120b to the first opening 110a, and the exhaust flow path is from the second opening 110b to the exhaust port 112 ) up to Euro.

According to this connection relationship, the air containing the dust introduced through the suction unit 120 is introduced into the dust container 140 through the intake passage inside the cleaner body 110, and the filter or the cyclone of the dust container 140 is removed. As it passes, air and dust are separated from each other. Dust is collected in the dust container 140 , and after air is discharged from the dust container 140 , it passes through an exhaust passage inside the cleaner body 110 , and is finally discharged to the outside through the exhaust port 112 .

An embodiment related to the components of the robot cleaner 100 will be described with reference to FIG. 4 below.

The robot cleaner 100 or the mobile robot according to an embodiment of the present invention includes a communication unit 1100, an input unit 1200, a driving unit 1300, a sensing unit 1400, an output unit 1500, a power supply unit 1600, At least one of the memory 1700 and the controller 1800 or a combination thereof may be included.

In this case, since the components shown in FIG. 4 are not essential, a robot cleaner having more components or fewer components may be implemented as a matter of course. Hereinafter, each component will be described.

First, the power supply unit 1600 is provided with a battery that can be charged by an external commercial power supply to supply power to the mobile robot.

The power supply unit 1600 may supply driving power to each component included in the mobile robot to supply operating power required for the mobile robot to travel or perform a specific function.

In this case, the control unit 1800 may detect the remaining power of the battery, and if the remaining power is insufficient, control to move to a charging station connected to an external commercial power source to receive a charging current from the charging station to charge the battery. The battery may be connected to the battery detection unit so that the remaining battery amount and charging state may be transmitted to the control unit 1800 . The output unit 1500 may display the remaining battery amount on the screen by the control unit.

The battery may be located at the lower part of the center of the robot cleaner, or may be located at either one of the left and right sides. In the latter case, the mobile robot may further include a counterweight to eliminate the weight bias of the battery.

The controller 1800 serves to process information based on artificial intelligence technology, and may include one or more modules that perform at least one of information learning, information inference, information perception, and natural language processing. can

The control unit 1800 learns a vast amount of information (big data) such as information stored in the vacuum cleaner, environmental information around the mobile terminal, and information stored in a communicable external storage using machine learning technology, At least one of inference and processing may be performed. In addition, the controller 1800 predicts (or infers) an operation of at least one executable cleaner by using the information learned using the machine learning technique, and has the highest practicability among the at least one predicted operation. You can control the cleaner to perform the action.

Machine learning technology is a technology for collecting and learning large-scale information based on at least one algorithm, and judging and predicting information based on the learned information. Information learning is an operation of quantifying the relationship between information and information by identifying characteristics, rules, and judgment criteria of information, and predicting new data using the quantified pattern.

Algorithms used by machine learning technology can be statistics-based algorithms, for example, a decision tree that uses the tree structure as a predictive model, and an artificial neural network that mimics the structure and function of a neural network in an organism. (neural network), genetic programming based on the evolutionary algorithm of living things, clustering that distributes observed examples into subsets called clusters, and Monte Carlo method that calculates function values with probability through randomly drawn random numbers (Monter carlo method) and the like.

As a field of machine learning technology, deep learning technology is a technology that performs at least one of learning, judging, and processing information by using an artificial neural network (DNN) algorithm. An artificial neural network (DNN) may have a structure that connects layers and transmits data between layers. Such deep learning technology can learn a vast amount of information through an artificial neural network (DNN) using a graphic processing unit (GPU) optimized for parallel operation.

The controller 1800 uses training data stored in an external server or memory, and may be equipped with a learning engine that detects a characteristic for recognizing a predetermined object. In this case, the characteristics for recognizing the object may include the size, shape, and shadow of the object.

Specifically, when a part of an image acquired through a camera provided in the vacuum cleaner is input to the learning engine, the controller 1800 may recognize at least one object or living organism included in the input image.

In this way, when the learning engine is applied to the running of the cleaner, the controller 1800 can recognize whether there are obstacles such as chair legs, fans, and certain types of balcony gaps around the cleaner that interfere with the running of the cleaner. , it is possible to increase the efficiency and reliability of the vacuum cleaner running.

Meanwhile, the above learning engine may be mounted on the control unit 1800 or may be mounted on an external server. When the learning engine is mounted on the external server, the controller 1800 may control the communication unit 1100 to transmit at least one image to be analyzed to the external server.

The external server may recognize at least one object or living organism included in the image by inputting the image received from the cleaner into the learning engine. In addition, the external server may transmit information related to the recognition result back to the cleaner.

In this case, the information related to the recognition result may include information related to the number of objects included in the image to be analyzed and the name of each object.

Meanwhile, the driving unit 1300 may include a motor, and by driving the motor, the left and right wheels may be rotated in both directions to rotate or move the main body. The driving unit 1300 may move the main body of the mobile robot forward, backward, left and right, curvedly traveling, or rotated in place.

Meanwhile, the input unit 1200 receives various control commands for the robot cleaner from the user. The input unit 1200 may include one or more buttons, for example, the input unit 1200 may include a confirmation button, a setting button, and the like. The confirmation button is a button for receiving a command from the user to check detection information, obstacle information, location information, and map information, and the setting button is a button for receiving a command for setting the information from the user.

In addition, the input unit 1200 cancels the previous user input and receives an input reset button to receive a user input again, a delete button to delete a preset user input, a button to set or change an operation mode, and a command to return to the charging station It may include a button for receiving input, and the like.

In addition, the input unit 1200 may be installed on the upper part of the mobile robot as a hard key, soft key, touch pad, or the like. Also, the input unit 1200 may have the form of a touch screen together with the output unit 1500 .

Meanwhile, the output unit 1500 may be installed above the mobile robot. Of course, the installation location or installation type may vary. For example, the output unit 1500 may display a battery state or a driving method on the screen.

Also, the output unit 1500 may output information on the internal state of the mobile robot detected by the sensing unit 1400 , for example, the current state of each component included in the mobile robot. Also, the output unit 1500 may display external state information, obstacle information, location information, map information, etc. detected by the sensing unit 1400 on the screen. The output unit 1500 may be any one of a light emitting diode (LED), a liquid crystal display (LCD), a plasma display panel, and an organic light emitting diode (OLED). It can be formed as an element of

The output unit 1500 may further include a sound output means for aurally outputting an operation process or operation result of the mobile robot performed by the controller 1800 . For example, the output unit 1500 may output a warning sound to the outside according to the warning signal generated by the control unit 1800 .

In this case, the sound output means may be a means for outputting sound, such as a beeper or a speaker, and the output unit 1500 uses audio data or message data having a predetermined pattern stored in the memory 1700 to make sound. It can be output to the outside through the output means. Accordingly, the mobile robot according to an embodiment of the present invention may output environmental information about the driving area to the screen or output the sound through the output unit 1500 . According to another embodiment, the mobile robot may transmit map information or environment information to the terminal device through the communication unit 1100 so that the terminal device outputs a screen or sound to be output through the output unit 1500 .

On the other hand, the communication unit 1100 is connected to a terminal device and/or other device located in a specific area (in this specification, the term "home appliance" is used interchangeably) with one of wired, wireless, and satellite communication methods. to transmit and receive signals and data.

The communication unit 1100 may transmit/receive data to/from another device located within a specific area. In this case, the other device may be any device capable of transmitting and receiving data by being connected to a network, and may be, for example, an air conditioner, a heating device, an air purification device, a light fixture, a TV, or a device such as a car. In addition, the other device may be a device for controlling a door, a window, a water valve, a gas valve, and the like. In addition, the other device may be a sensor that detects temperature, humidity, atmospheric pressure, gas, or the like.

Meanwhile, the memory 1700 stores a control program for controlling or driving the robot cleaner and data corresponding thereto. The memory 1700 may store audio information, image information, obstacle information, location information, map information, and the like. Also, the memory 1700 may store information related to a driving pattern.

The memory 1700 mainly uses a non-volatile memory. Here, the non-volatile memory (NVM, NVRAM) is a storage device capable of continuously maintaining stored information even when power is not supplied, for example, a ROM, a flash memory, a magnetic computer. It may be a storage device (eg, a hard disk, a diskette drive, a magnetic tape), an optical disk drive, a magnetic RAM, a PRAM, and the like.

Meanwhile, the sensing unit 1400 may include at least one of an external signal detection sensor, a front detection sensor, a cliff detection sensor, a 2D camera sensor, and a 3D camera sensor.

The external signal detection sensor may detect an external signal of the mobile robot. The external signal detection sensor may be, for example, an infrared sensor, an ultra sonic sensor, a radio frequency sensor, or the like.

The mobile robot can check the position and direction of the charging station by receiving a guide signal generated by the charging station using an external signal detection sensor. In this case, the charging stand may transmit a guide signal indicating a direction and a distance so that the mobile robot can return. That is, the mobile robot may receive a signal transmitted from the charging station, determine its current position, set a moving direction, and return to the charging station.

On the other hand, the front detection sensor may be installed at regular intervals in front of the mobile robot, specifically along the outer peripheral surface of the side of the mobile robot. The forward detection sensor is located on at least one side of the mobile robot to detect an obstacle in the front. The forward detection sensor detects an object, particularly an obstacle, existing in the moving direction of the mobile robot, and transmits detection information to the controller 1800 . can transmit That is, the front detection sensor may detect protrusions, household appliances, furniture, walls, wall corners, etc. existing on the movement path of the mobile robot, and transmit the information to the controller 1800 .

The front detection sensor may be, for example, an infrared sensor, an ultrasonic sensor, an RF sensor, a geomagnetic sensor, and the like, and the mobile robot may use one type of sensor as the front detection sensor or use two or more types of sensors together as needed. have.

For example, the ultrasonic sensor may be mainly used to generally detect a distant obstacle. The ultrasonic sensor includes a transmitter and a receiver, and the controller 1800 determines the presence of an obstacle by whether the ultrasonic wave emitted through the transmitter is reflected by an obstacle and is received by the receiver, and determines the ultrasonic radiation time and ultrasonic reception time. can be used to calculate the distance to the obstacle.

In addition, the controller 1800 may detect information related to the size of an obstacle by comparing the ultrasonic waves emitted from the transmitter and the ultrasonic waves received by the receiver. For example, the controller 1800 may determine that the size of the obstacle increases as more ultrasound waves are received by the receiver.

In an embodiment, a plurality of (eg, five) ultrasonic sensors may be installed on the front side of the mobile robot along the outer circumferential surface.

In this case, preferably, the ultrasonic sensor may be installed in the front of the mobile robot by a transmitter and a receiver alternately.

That is, the transmitter may be disposed to be spaced apart on the left and right sides from the center of the front of the main body, and one or more transmitters may be disposed between the receiver to form a receiving area of the ultrasonic signal reflected from an obstacle or the like. With such an arrangement, the reception area can be expanded while reducing the number of sensors. The transmission angle of the ultrasonic wave may maintain an angle within a range that does not affect different signals to prevent a crosstalk phenomenon. Also, the reception sensitivities of the receivers may be set differently.

In addition, the ultrasonic sensor may be installed upward by a certain angle so that the ultrasonic waves transmitted from the ultrasonic sensor are output upward, and in this case, a predetermined blocking member may be further included to prevent the ultrasonic waves from being radiated downward.

On the other hand, as the front detection sensor, as described above, two or more types of sensors may be used together, and accordingly, the front detection sensor may use any one type of sensor such as an infrared sensor, an ultrasonic sensor, or an RF sensor. .

For example, the front detection sensor may include an infrared sensor as a sensor other than the ultrasonic sensor.

The infrared sensor may be installed on the outer peripheral surface of the mobile robot together with the ultrasonic sensor. The infrared sensor may also detect obstacles existing in front or on the side and transmit obstacle information to the controller 1800 . That is, the infrared sensor detects protrusions, household appliances, furniture, walls, wall corners, etc. existing on the movement path of the mobile robot, and transmits the information to the controller 1800 . Accordingly, the mobile robot can move its main body within a specific area without colliding with an obstacle.

On the other hand, the cliff detection sensor (or cliff sensor) may detect an obstacle on the floor supporting the main body of the mobile robot, mainly using various types of optical sensors.

That is, the cliff detection sensor is installed on the rear surface of the mobile robot on the floor, but may be installed at different positions depending on the type of the mobile robot. The cliff detection sensor is located on the rear surface of the mobile robot to detect obstacles on the floor. The cliff detection sensor is an infrared sensor, ultrasonic sensor, RF sensor, PSD (Position) having a light emitting unit and a light receiving unit like the obstacle detection sensor Sensitive Detector) sensor or the like.

For example, any one of the cliff detection sensors may be installed in front of the mobile robot, and the other two cliff detection sensors may be installed relatively backward.

For example, the cliff detection sensor may be a PSD sensor, but may also include a plurality of different types of sensors.

The PSD sensor detects the short and long-distance position of the incident light with one p-n junction using the semiconductor surface resistance. The PSD sensor includes a one-dimensional PSD sensor that detects light in only one axial direction and a two-dimensional PSD sensor that detects a light position on a plane, both of which may have a pin photodiode structure. The PSD sensor is a kind of infrared sensor and measures the distance by using infrared rays to transmit infrared rays and then measure the angle of infrared rays reflected back from obstacles. That is, the PSD sensor calculates the distance to the obstacle using the triangulation method.

The PSD sensor is provided with a light emitting unit that emits infrared rays to an obstacle and a light receiving unit that receives infrared rays reflected back from the obstacle, but is generally configured in the form of a module. When an obstacle is detected using the PSD sensor, a stable measurement value can be obtained regardless of the difference in reflectance and color of the obstacle.

The control unit 1800 may measure an infrared angle between an infrared light emitting signal emitted by the cliff detection sensor toward the ground and a reflected signal received by being reflected by an obstacle to detect the cliff and analyze its depth.

Meanwhile, the controller 1800 may determine whether to pass or not according to the ground state of the cliff sensed using the cliff sensor, and may determine whether to pass the cliff according to the determination result. For example, the control unit 1800 determines whether a cliff exists and the depth of the cliff through the cliff sensor, and then passes the cliff only when a reflection signal is detected through the cliff sensor.

As another example, the controller 1800 may determine the lift-up phenomenon of the mobile robot using the cliff detection sensor.

Meanwhile, the two-dimensional camera sensor is provided on one surface of the mobile robot to acquire image information related to the periphery of the main body during movement.

The optical flow sensor generates image data in a predetermined format by converting a downward image input from an image sensor provided in the sensor. The generated image data may be stored in the memory 1700 .

Also, one or more light sources may be installed adjacent to the optical flow sensor. One or more light sources irradiate light to a predetermined area of the floor surface photographed by the image sensor. That is, when the mobile robot moves in a specific area along the floor surface, a constant distance is maintained between the image sensor and the floor surface if the floor surface is flat. On the other hand, when the mobile robot moves on the floor surface of the non-uniform surface, it is moved away from it by a certain distance or more due to irregularities and obstacles on the floor surface. In this case, one or more light sources may be controlled by the controller 1800 to adjust the amount of light irradiated. The light source may be a light emitting device capable of controlling the amount of light, for example, a Light Emitting Diode (LED).

Using the optical flow sensor, the controller 1800 may detect the position of the mobile robot regardless of the sliding of the mobile robot. The controller 1800 may calculate a moving distance and a moving direction by comparing and analyzing image data captured by the optical flow sensor over time, and may calculate the position of the mobile robot based on this. By using the image information on the lower side of the mobile robot by using the optical flow sensor, the controller 1800 can make strong correction against sliding with respect to the position of the mobile robot calculated by other means.

The 3D camera sensor may be attached to one surface or a part of the body of the mobile robot to generate 3D coordinate information related to the circumference of the body.

That is, the three-dimensional camera sensor may be a three-dimensional depth camera (3D depth camera) for calculating the distance between the mobile robot and the object to be photographed.

Specifically, the 3D camera sensor may capture a 2D image related to the surroundings of the body, and may generate a plurality of 3D coordinate information corresponding to the captured 2D image.

In one embodiment, the three-dimensional camera sensor includes two or more cameras for acquiring an existing two-dimensional image, and combining two or more images obtained from the two or more cameras to generate three-dimensional coordinate information. can be formed in this way.

Specifically, the three-dimensional camera sensor according to the embodiment includes a first pattern irradiating unit for irradiating a first pattern of light downward toward the front of the body, and a second pattern of irradiating light of a second pattern upward toward the front of the body. It may include a second pattern irradiation unit and an image acquisition unit for acquiring an image of the front of the main body. Accordingly, the image acquisition unit may acquire an image of a region where the light of the first pattern and the light of the second pattern are incident.

In another embodiment, the three-dimensional camera sensor includes an infrared pattern emitter for irradiating an infrared pattern together with a single camera, and captures the shape of the infrared pattern irradiated from the infrared pattern emitter projected on the object to be photographed, so that the three-dimensional camera A distance between the sensor and the object to be photographed may be measured. The 3D camera sensor may be an IR (Infra Red) type 3D camera sensor.

In another embodiment, the three-dimensional camera sensor has a light emitting unit that emits light together with a single camera, receives a portion of the laser emitted from the light emitting unit reflected from the object to be photographed, and analyzes the received laser. A distance between the camera sensor and the object to be photographed may be measured. Such a three-dimensional camera sensor may be a three-dimensional camera sensor of a time of flight (TOF) method.

Specifically, the laser of the three-dimensional camera sensor as described above is configured to irradiate a laser having a form extending in at least one direction. In one example, the three-dimensional camera sensor may include first and second lasers, the first laser irradiates a laser beam of a straight line that intersects with each other, and the second laser irradiates a single straight laser beam. can do. According to this, the lowermost laser is used to detect an obstacle in the bottom part, the uppermost laser is used to detect an obstacle in the upper part, and the middle laser between the lowermost laser and the uppermost laser is used to detect the obstacle in the middle part. is used for

Referring to FIG. 5 below, a system including a cleaner performing autonomous driving according to an embodiment of the present invention will be described.

Referring to FIG. 5 , the robot cleaning system 50 according to an embodiment of the present invention includes a cleaner 100 that performs autonomous driving, an AP device 400 , a server 500 , a network 550 , and a mobile terminal. It may include the ones 200a and 200b.

Among them, the cleaner 100 , the AP device 400 , and the mobile terminals 600a may be disposed in the building 10 such as a house.

The vacuum cleaner 100 is a vacuum cleaner that automatically cleans, and may perform automatic driving and automatic cleaning. Meanwhile, the cleaner 100 includes, in addition to the driving function and the cleaning function, the communication unit 1100 therein, and transmits data with electronic devices in the internal network 10 or electronic devices connectable through the external network 550 . can be exchanged To this end, the communication unit 1100 may perform data exchange with the AP device 400 by wire or wirelessly.

The access point (AP) device 400 may provide the internal network 10 to an adjacent electric device. In particular, it is possible to provide a wireless network.

Meanwhile, the AP device 400 may allocate a wireless channel according to a predetermined communication method to electronic devices in the internal network 10 and perform wireless data communication through the corresponding channel. Here, the predetermined communication method may be a WiFi communication method.

In this case, the mobile terminal 600b located in the internal network 10 can perform monitoring and remote control of the cleaner 100 by accessing the cleaner 100 through the AP device 400 . .

Meanwhile, the AP device 400 may perform data communication with an external electronic device through the external network 550 in addition to the internal network 10 .

For example, the AP device 400 may perform wireless data communication with the mobile terminal 200a located outside through the external network 550 .

In this case, the mobile terminal 200a located in the external network 550 monitors and remotely controls the cleaner 100 by accessing the cleaner 100 through the external network 550 and the AP device 400 . etc. can be performed.

As another example, the AP device 400 may perform wireless data communication with the server 500 located outside through the external network 550 .

The server 500 may include a voice recognition algorithm. In addition, when receiving voice data, the received voice data may be converted into text data and output.

Meanwhile, the server 500 may store firmware information and driving information (course information, etc.) for the cleaner 100 , and register product information for the cleaner 100 . For example, the server 500 may be a server operated by the cleaner 100 manufacturer. As another example, the server 500 may be a server operated by an open application store operator.

Meanwhile, unlike the drawing, the server 500 is provided in the home and may be a home server that stores state information about home appliances or content shared by home appliances. When the server 500 is a home server, information related to a foreign material, for example, a foreign material image, may be stored.

Meanwhile, according to an embodiment of the present invention, the cleaner 100 captures an image containing foreign substances through a camera provided therein, and displays the image containing the foreign substances and image-related information, the mobile terminal 200 ) or transmitted to the server 500, based on the cleaning execution information for foreign substances, from the mobile terminal 200 or server 500, to clean around foreign substances, or based on cleaning avoidance information for foreign substances, foreign substances You may not be able to clean your surroundings. Accordingly, it is possible to selectively clean the foreign material.

Meanwhile, according to another embodiment of the present invention, the cleaner 100 captures an image containing a foreign material through a stereo camera provided therein, and performs signal processing for the image containing the foreign material obtained from the stereo camera. to check the object related to the foreign material in the image, and based on the identified object related to the foreign material, generate cleaning execution information or cleaning avoidance information for the foreign material, and clean around the foreign material based on the cleaning execution information, Based on the cleaning avoidance information, it is possible not to clean around the foreign matter. Accordingly, it is possible to selectively clean the foreign material.

The control method of the cleaner 100 according to the present invention will be described with reference to FIG. 6 below.

Referring to FIG. 6 , the cleaner 100 may start a monitoring driving ( S601 ).

For example, when receiving a control command for starting the monitoring driving from the user terminal, the cleaner 100 may start the monitoring driving. In another example, when a user input is applied to a button (not shown) provided on the cleaner body, the cleaner 100 may start monitoring driving. In another example, the cleaner 100 may start monitoring driving when entering a preset time band. In another example, when it is determined that a dangerous situation has occurred around the main body, the cleaner 100 may start monitoring driving.

Specifically, various embodiments in which the cleaner 100 detects a dangerous situation will be described as follows.

In an embodiment, when it is determined that an obstacle exists on the driving path of the cleaner 100 based on the detection result of the sensor 1400 , the controller 1800 determines that a dangerous situation has occurred in the cleaner 100 . can

In another embodiment, when it is determined that the wheel of the driving unit 1300 is restrained based on the detection result of the sensor 1400 , the controller 1800 may determine that a dangerous situation has occurred in the cleaner 100 .

In another embodiment, when it is determined that the wheel of the driving unit 1300 is idling based on the detection result of the sensor 1400 , the controller 1800 may determine that a dangerous situation has occurred in the cleaner 100 .

In another embodiment, when it is determined that one point of the main body of the cleaner 100 is spaced apart from the floor of the cleaning area by a predetermined distance or more based on the detection result of the sensor 1400, the controller 1800 ) can be considered as a dangerous situation.

In another embodiment, when it is determined that the main body deviated from the driving route based on the detection result of the sensor 1400 , the controller 1800 may determine that a dangerous situation has occurred in the cleaner 100 .

In another embodiment, when it is determined that the main body cannot escape from a specific area over a predetermined time interval based on the detection result of the sensor 1400, the control unit 1800 determines that a dangerous situation has occurred in the cleaner 100 can judge That is, when it is determined that the cleaner 100 is trapped under the bed or in a narrow area, the controller 1800 may determine that a dangerous situation has occurred in the cleaner 100 .

In another embodiment, the controller 1800 determines that a dangerous situation has occurred in the cleaner 100 when a significant difference is detected between a plurality of images captured at a predetermined time interval in a state in which the main body does not move. can Here, the controller 1800 may determine whether there is a significant difference between the plurality of images by comparing pixel values respectively included in the first image, the second image, and the third image among the plurality of images.

Specifically, the controller 1800 detects a difference between the first image, the second image, and the third image continuously captured by the camera, specifies the first subject based on the detected difference, and moves the first subject The direction information may be detected, and it may be determined that a dangerous situation has occurred in the type of the first subject and the moving direction of the first subject.

In another embodiment, the controller 1800 may determine that a dangerous situation has occurred in the cleaner 100 when an abnormal sound source is generated around the main body based on the detection result of the sensor 1400 .

Here, the memory 1700 of the cleaner 100 may store a sound source database including an abnormal sound source and a non-abnormal sound source.

For example, a situation where there is an intruder from the outside, such as the sound of various objects falling or breaking, the sound of glass breaking, the sound of a drill turning, the sound of a dog barking, an alarm sound generated by an alarm device with various sensors, etc. Various noises that can be determined to be an abnormal sound source can be stored as abnormal sound sources, and conversely, noise generated inside the robot cleaner 100, noise generated from home appliances such as refrigerators, washing machines, and water purifiers, etc. Various noises can be saved as non-ideal sound sources.

In this case, the controller 1800 determines whether the sound source acquired through the sound acquisition means included in the sensor 1400 is similar to at least one of the sound sources stored in the sound source database, so that the acquired sound source is an abnormal sound source or not. It can be determined whether it is a sound source or not. That is, the controller 1800 may determine whether the sound sources match and/or the degree of similarity between the sound sources acquired through the sound acquisition means and the sound sources stored in the sound source database by calculating the degree of similarity.

The controller 1800 may control the sensor 1400 so that information related to the dangerous situation described above can be periodically collected.

On the other hand, when the monitoring driving of the cleaner 100 is not started by a user input, but is automatically started by the decision of the control unit 1800, the control unit 1800 may initiate the monitoring driving with at least one of the user terminal and the server. The communication unit 1100 may be controlled to transmit a notification message. In particular, when the user terminal receives the above message from the cleaner 100, a button for executing an app related to monitoring driving or checking whether the app is running is pressed so that the monitoring screen is output on the display of the terminal. can be printed on

Referring to FIG. 6 , the controller 1800 may control the camera so that an image is captured every preset period ( S602 ). That is, when the monitoring driving is started, the controller 1800 may control the camera so that an image is captured every predetermined time interval.

In an embodiment, the controller 1800 may control the camera so that a plurality of images are captured while the main body of the cleaner 100 is positioned at one point.

In another embodiment, the controller 1800 may control the camera so that a plurality of images are captured in a state in which the direction the camera is directed is fixed.

In another embodiment, the controller 1800 may control the camera to capture the first to third images for each of the plurality of points while traversing a plurality of preset points within the cleaning area. Specifically, when the cleaner body is located at any one of the plurality of points, the controller 1800 may control the driving unit so that one surface of the main body faces a direction set at each of the plurality of points. When one surface of the main body faces a set direction, the controller 1800 may control the camera to capture an image every preset period.

That is, the controller 1800 may control the driver and the camera so that the first image, the second image, and the third image are acquired at the same point in the same camera direction.

Meanwhile, when the monitoring driving starts, the controller 1800 may control the driving unit to stop the main body of the cleaner 100 while the first image, the second image, and the third image included in the comparison target are photographed.

Also, the controller 1800 may control the driving of the camera so that the direction the camera faces is fixed while the first image, the second image, and the third image included in the comparison target are photographed. Also, the controller 1800 may control the driving of the camera so that the camera does not zoom in or zoom out while the first image, the second image, and the third image are captured.

That is, the controller 1800 stops the movement of the cleaner 100 or the rotation of the camera while the plurality of images included in the comparison target are being photographed, so that the movement of the cleaner generated between the first image and the second image difference can be minimized.

Next, the controller 1800 may determine whether a difference has occurred between the second image photographed in the current period and the first image photographed in the previous period ( S603 ).

In more detail, the controller 1800 may compare a plurality of color values for each pixel included in the first image and a color value for each pixel included in the second image, respectively.

When the point at which the first image is captured and the point at which the second image is captured are different, the controller 1800 may determine whether a difference occurs between a part of the first image and a part of the second image.

In addition, when the direction the camera is oriented at the time the first image is captured is different from the direction the camera is oriented at the time the second image is captured, the controller 1800 may control a part of the first image and a part of the second image. It can be determined whether there is a difference between them.

Also, when the first part of the cleaning area corresponding to the first image and the second part of the cleaning area corresponding to the second image are different, the controller 1800 is configured to perform a space between a part of the first image and a part of the second image. It can be determined whether a difference has occurred.

Specifically, the controller 1800 may select a part of the first image and a part of the second image, respectively, based on the information related to the movement of the cleaner 100 from the time the first image is captured to the time the second image is captured. can be extracted. The controller 1800 may determine whether a difference occurs between a part of the extracted first image and a part of the extracted second image.

The controller 1800 detects a difference between the first direction the camera is oriented at the time the first image is captured and the second direction the camera is oriented at the time the second image is captured, and based on the detected difference, A part of the first image and a part of the second image may be extracted, respectively.

That is, the first part of the cleaning area corresponding to the part extracted from the first image may correspond to the second part of the cleaning area corresponding to the part extracted from the second image.

In order to extract a part of the first image and a part of the second image as comparison objects, the control unit 1800 calculates the movement history of the cleaner 100 from the time the first image is photographed to the time the second image is photographed. Relevant information is available. In addition, in order to extract a part of the first image and a part of the second image as comparison objects, the control unit 1800 divides the first set value that is the set value of the camera at the time the first image is captured and the second image. A second setting value that is a setting value of the camera at the time of photographing may be compared.

In addition, a time point at which the first image is captured is defined as a first view point, and a time point at which the second image is captured is defined as a second view point.

When a plurality of images are photographed while traversing a plurality of points, the controller 1800 may allocate a plurality of images photographed at the same point to any one comparison target group.

Also, if it is determined that a difference has occurred between the first image and the second image, the controller 1800 may generate a first difference image based on the difference between the first image and the second image ( S604 ). .

In one example, the controller 1800 may generate a first difference image by cropping a portion of the second image in which a difference from the first image occurs. In this case, the controller 1800 may crop only the portion where the difference from the first image occurs among the second image, or may crop the smallest rectangle including the portion where the difference occurs.

In another example, the first difference image may correspond to the second image. That is, when a difference occurs between the second image and the first image, the controller 1800 may generate the first difference image by copying the second image.

On the other hand, if it is determined that there is no difference between the first image and the second image, the controller 1800 may again perform the step S602 of capturing an image at a preset period. In this case, the previously captured first and second images may be deleted.

The controller 1800 may analyze the first difference image by using a deep learning algorithm (S605). The controller 1800 may analyze the first difference image to specify the first subject. That is, the controller 1800 may detect information related to the first subject by using a deep learning algorithm. For example, the information related to the first subject may include information related to whether the subject is an object or a living organism.

In another example, when the subject is a living body, the information related to the first subject may include information related to a species corresponding to the subject.

In another example, when the subject is an object, the information related to the first subject may include information related to the name of the subject.

In another example, the information related to the first subject may include information related to the size and shape of the subject.

As another example, the information related to the first subject may include information related to the number of objects included in the first difference image.

Specifically, the controller 1800 may extract objects included in the first difference image by using a deep learning algorithm and recognize the extracted objects. For example, the extracted objects may include a person or thing, a place corresponding to the background, a time indicated by the background (eg, day or night), a location of a place corresponding to the background, and the like.

The controller 1800 may calculate the similarity between the training data and the first difference image by comparing the first difference image with a plurality of training data previously stored in the memory 1700 by using a deep learning algorithm.

In this case, the plurality of training data previously stored in the memory 1700 may include training data for image analysis.

For reference, the training data may be defined as data used by the controller 1800 when the controller 1800 drives a learning engine for analyzing a dangerous situation.

That is, the training data may include data collected by the cleaner 100 when a dangerous situation has occurred in a plurality of other cleaners in the past. Specifically, the training data may include coordinate information for the cleaning area, sensing values related to obstacles, and images captured around the body.

In an embodiment, the controller 1800 may determine the similarity between the plurality of image information included in the training data stored in the memory 1700 and the first difference image by using a deep learning algorithm. The controller 1800 may detect information related to the first subject based on the similarity determined as described above.

Meanwhile, the training data may be stored in the memory 1700 mounted on the cleaner or may be stored in the server 500 .

When analysis of the first difference image is required, the controller 1800 may request training data from the server 500 .

In addition, when training data is stored in the memory 1700 , the controller 1800 may control the communication unit 1100 to transmit a training data update request to the server 500 in order to update the training data.

In an embodiment, the controller 1800 may transmit a training data update request to the server 500 whenever a difference occurs between the first image and the second image.

In another embodiment, the controller 1800 may transmit a training data update request to the server 500 whenever a monitoring driving is started.

In another embodiment, when it is difficult to detect a subject corresponding to the first difference image, the controller 1800 may transmit a training data update request to the server 500 .

In another embodiment, the controller 1800 may transmit a training data update request to the server 500 so that the training data is updated at a predetermined period.

Specifically, the controller 1800 may control the communication unit 1100 to transmit a training data update request to the server 500 when a predetermined time interval elapses from the time of the latest update of the training data. When a dangerous situation is detected, the controller 1800 may check the latest update time of the training data or periodically check the latest update time of the training data regardless of the dangerous situation.

On the other hand, the training data update request, identification information of the cleaner 100, information related to the version of the training data stored in the memory 1700 of the cleaner 100, information related to the version of the learning engine mounted in the cleaner 100, It may include at least one of information related to a dangerous situation, identification information indicating a type of information related to a dangerous situation, and information related to a cleaning area in which the cleaner 100 is disposed.

The controller 1800 may determine whether the analysis result of the first difference image is a moving direction analysis target (S606).

Specifically, when it is determined that the first subject is a person, the controller 1800 may determine that the analysis result of the first difference image is a moving direction analysis target.

Also, when it is determined that the first subject is an entrance (window or door) existing in the cleaning area, the controller 1800 may determine that the analysis result of the first difference image is a moving direction analysis target.

In addition, if it is determined that the difference between the first image and the second image is due to a change in the amount of light irradiated to the cleaning area, the controller 1800 may determine that the analysis result of the first difference image is not included in the report. can

Also, when it is determined that the first subject is a shadow, the controller 1800 may determine that the analysis result of the first difference image is not included in the report target.

On the other hand, even if it is determined that the first subject is a person or an animal, the controller 1800 determines whether the first difference image corresponds to a pre-registered image, and according to the determination result, the analysis result of the first difference image is reported It can be judged that it is not included in the subject of

When it is determined that the analysis result of the first difference image is a movement direction analysis target, the controller 1800 may detect movement direction information of the first subject.

Specifically, the control unit 1800 stops the movement of the cleaner 100 or the rotation of the camera while a plurality of images included in the comparison object are captured, thereby preventing the movement of the cleaner generated between the second image and the third image. difference can be minimized.

Next, the controller 1800 may determine whether a difference has occurred between the third image and the third image captured in the previous period ( S607 ).

In more detail, the controller 1800 may compare a plurality of pixel-specific color values included in the second image and a plurality of pixel-specific color values included in the third image, respectively.

When the point at which the second image is captured and the point at which the third image is captured are different from each other, the controller 1800 may determine whether a difference occurs between a portion of the second image and a portion of the third image.

Also, when the direction the camera is oriented at the time the second image is captured is different from the direction the camera is oriented at the time the third image is captured, the controller 1800 may control a part of the second image and a part of the third image. It can be determined whether there is a difference between them.

In addition, when the second part of the cleaning area corresponding to the third image and the third part of the cleaning area corresponding to the third image are different, the controller 1800 is configured to perform a space between a part of the second image and a part of the third image. It can be determined whether a difference has occurred.

Specifically, the controller 1800 may select a part of the second image and a part of the third image, respectively, based on information related to the movement of the cleaner 100 from the time the second image is captured to the time the third image is captured. can be extracted. The controller 1800 may determine whether a difference occurs between a part of the extracted second image and a part of the extracted third image.

The controller 1800 detects a difference between the second direction the camera is oriented at the time the second image is captured and the third direction the camera is oriented at the time the third image is captured, and based on the detected difference, A part of the second image and a part of the third image may be extracted, respectively.

In order to extract a part of the first image and a part of the second image as comparison objects, the control unit 1800 calculates the movement history of the cleaner 100 from the time the first image is photographed to the time the second image is photographed. Relevant information is available. In addition, in order to extract a part of the first image and a part of the second image as comparison objects, the control unit 1800 divides the first set value that is the set value of the camera at the time the first image is captured and the second image. A second setting value that is a setting value of the camera at the time of photographing may be compared.

Also, if it is determined that a difference has occurred between the second image and the third image, the controller 1800 may generate a second difference image based on the difference between the second image and the third image ( S608 ). .

In one example, the controller 1800 may generate a second difference image by cropping a portion of the third image in which a difference from the second image occurs. In this case, the controller 1800 may crop only the portion where the difference from the second image occurs among the third image, or may crop the smallest rectangle including the portion where the difference occurs.

In another example, the second difference image may correspond to the third image. That is, when a difference occurs between the third image and the second image, the controller 1800 may generate the second difference image by copying the third image.

On the other hand, if it is determined that there is no difference between the second image and the third image, the controller 1800 may again perform the step S602 of capturing an image every preset period. In this case, the previously captured second and third images may be deleted.

The controller 1800 may analyze the second difference image by using a deep learning algorithm (S609). The controller 1800 may analyze the second difference image to detect movement direction information of the first subject. For example, the moving direction information of the first subject may include the moving direction of the living body when the first subject is an indoor and independent living body. The movement direction information of the first subject may be a rotation direction or an opening direction of the door when the first subject is an indoor-dependent object (eg, a door).

Specifically, the controller 1800 may extract movement directions of objects included in the second difference image by using a deep learning algorithm.

The controller 1800 may determine whether the analysis result of the second difference image is a report target (S616).

Specifically, when the first subject is a person and the moving direction of the first subject is a preset moving direction, the controller 1800 may determine that the analysis result of the second difference image is the subject of the report.

Also, when the first subject is a door and the moving direction of the first subject is a preset moving direction, the controller 1800 may determine that the analysis result of the second difference image is the subject of the report.

In addition, when it is determined that the difference between the second image and the third image is due to a change in the amount of light irradiated to the cleaning area, the controller 1800 determines that the analysis result of the second difference image is not included in the report. can

Specifically, when it is determined that the analysis result of the second difference image is the target of the report, the controller 1800 transmits at least one of the third image, the second difference image, and information related to the first subject to the server 500 and the user terminal ( 200b), it is possible to control the communication unit 1100 to transmit at least one.

In addition, in order to inform that a report target has occurred, the controller 1800 transmits a warning message to at least one of the server 500 and the user terminal 200b when it is determined that the analysis result of the second difference image is the target of the report. (1100) can be controlled.

Also, when it is determined that the analysis result of the second difference image is the subject of the report, the controller 1800 may control the communication unit 1100 to output the image of the first subject to the terminal 200b.

Meanwhile, when it is determined that the analysis result of the second difference image is not included in the report target, the controller 1800 may control the camera to capture a new image. That is, if it is determined that the analysis result of the second difference image is not included in the report target, the control unit 1800 may perform the step S602 of photographing the image every preset period again. In this case, the previously captured second and third images and the previously generated third difference image may be deleted.

Of course, the controller 1800 may control the terminal to activate a reference information input screen for receiving reference information, which is a reference for outputting an alarm. That is, through the terminal, it is possible to receive reference information for which the alarm output is a reference. Here, the reference information may be information on the type of the first object, which is an alarm output target, and the movement direction of the first object.

Also, the reference information may be information on the type of the first object corresponding to each boundary area and information on the movement direction of the first object. That is, the controller 1800 may receive information on the type of the first object and the movement direction information of the first object for each boundary area through the terminal.

The controller 1800 may analyze the second image or the third image to specify the position of the first subject. The position of the first subject is specified in consideration of the position at which the image is captured.

When the first subject is a person and the position of the first subject is within the boundary area, if the moving direction of the first subject is a preset movement direction to correspond to the boundary area, the control unit 1800 transmits information related to the first subject to the server and The communication unit may be controlled to be transmitted to at least one of the terminals.

Here, the boundary area is an area set by the user within the cleaning area, and is an area to be monitored by the cleaner when the user goes out or when the user commands a home guard. A plurality of boundary areas may be designated, and a type of the first object and a moving direction of the first object may be set corresponding to each boundary area. Accordingly, if the user predicts the direction of intrusion of a person for each boundary area and sets the moving direction of the first subject, there is an advantage in that an error in generating an alarm can be reduced.

As another example, if the first subject is a person, the position of the first subject is within the boundary area, and the movement direction of the first object is a preset movement direction to correspond to the boundary area, the terminal may control the first subject. You can control the output of an image.

As another example, if the first subject is a person, the position of the first subject is within the boundary area, and the movement direction of the first object is a preset movement direction to correspond to the boundary area, a warning message is sent to the terminal. may control the communication unit to transmit.

Hereinafter, a method of detecting a difference between a plurality of images by a cleaner according to an embodiment of the present invention is described with reference to FIG. 7 .

Referring to FIG. 7 , the camera of the cleaner 100 may capture a first image 701 , a second image 702 , and a third image 706 while the cleaner 100 performs a monitoring driving.

That is, during the monitoring driving, the controller 1800 may control the camera to capture an image every preset period. In one example, the set period may be changed by design. In another example, the set period may be changed by a user input. In another example, when a difference occurs between two previously captured images, the controller 1800 may decrease the period of capturing the images.

In another example, when it is determined that a difference generated in two previously captured images is the subject of a report, the controller 1800 may decrease the period of capturing an image.

Also, as shown in FIG. 7 , the controller 1800 compares the first image 701 with the second image 702 , and a difference generated between the first image 701 and the second image 702 is generated. (703) can be detected.

In addition, the controller 1800 may generate a first difference image 704 based on a difference generated between the first image 701 and the second image 702 . For example, the controller 1800 may generate the first difference image 704 by cropping a portion of the second image 702 that is different from the first image 701 .

The controller 1800 may include a deep learning algorithm unit for performing a deep learning algorithm and an image recognition unit. The controller 1800 may compare the first difference image 704 with a plurality of training data stored in the server 500 using a deep learning algorithm. The controller 1800 may detect at least one training data corresponding to the first difference image 704 among the plurality of training data. In addition, the controller 1800 may detect information related to the first subject of the first difference image 704 by using label information of the detected training data.

The controller 1800 may compare the second image 702 and the third image 706 to detect differences 705 and 707 generated between the second image 702 and the third image 706 . .

In addition, the controller 1800 may generate a second difference image 708 based on a difference generated between the second image 702 and the third image 706 . For example, the controller 1800 may generate the second difference image 708 by cropping a portion of the third image 706 that is different from the second image 702 .

The controller 1800 may analyze the second difference image 708 to detect movement direction information of the first subject.

As shown in FIG. 7 , the controller 1800 may recognize the first subject as a person and recognize the moving direction of the first subject from right to left. When the moving direction of the first subject does not match the preset direction, the controller 1800 may determine that the analysis result of the second difference image 708 is not a report target.

Here, the preset movement direction of the first subject is from the upper side to the lower side. Accordingly, the controller 1800 may determine that the analysis result of the second difference image 708 is not a report target.

8 is a conceptual diagram illustrating a method in which a mobile robot detects a difference between a plurality of images according to an embodiment of the present invention.

Referring to FIG. 8 , the camera of the cleaner 100 may capture a first image 701 , a second image 702 , and a third image 706 while the cleaner 100 performs a monitoring driving.

The controller 1800 may compare the first image 701 and the second image 702 to detect a difference 703 generated between the first image 701 and the second image 702 .

In addition, the controller 1800 may generate a first difference image 704 based on a difference generated between the first image 701 and the second image 702 . For example, the controller 1800 may generate the first difference image 704 by cropping a portion of the second image 702 that is different from the first image 701 .

The controller 1800 may include a deep learning algorithm unit for performing a deep learning algorithm and an image recognition unit. The controller 1800 may compare the first difference image 704 with a plurality of training data stored in the server 500 using a deep learning algorithm. The controller 1800 may detect at least one training data corresponding to the first difference image 704 from among the plurality of training data. In addition, the controller 1800 may detect information related to the first subject of the first difference image 704 by using label information of the detected training data.

The controller 1800 may compare the second image 702 and the third image 706 to detect a difference 705 generated between the second image 702 and the third image 706 .

In addition, the controller 1800 may generate a second difference image 708 based on a difference generated between the second image 702 and the third image 706 . For example, the controller 1800 may generate the second difference image 708 by cropping a portion of the third image 706 that is different from the second image 702 .

The controller 1800 may analyze the second difference image 708 to detect movement direction information of the first subject.

As shown in FIG. 7 , the controller 1800 may recognize the first subject as a person and recognize the moving direction of the first subject from upper side to lower side. When the moving direction of the first subject coincides with the preset direction, the controller 1800 may determine that the analysis result of the second difference image 708 is a report target.

Here, the preset movement direction of the first subject is from the upper side to the lower side. Accordingly, the controller 1800 may determine that the analysis result of the second difference image 708 is a report target.

9 illustrates a method of transmitting, by the cleaner, a result of analyzing a difference detected between a plurality of images to a user terminal according to an embodiment of the present invention.

Referring to FIG. 9 , when it is determined that the information related to the first subject of the second difference image 708 is the subject of the report, the controller 1800 selects at least one of the second image, the third image, and the second difference image. The communication unit 1100 may be controlled to be transmitted to the user terminals 200a and 200b.

The user terminals 200a and 200b may output the image received from the cleaner 100 on the display of the terminal. The display of the user terminals 200a and 200b may output a first window 301 displaying at least one of the images received from the cleaner 100 .

Hereinafter, a method of controlling the mobile robot 100 according to embodiments of the present invention will be described with reference to FIG. 10 . In each of the flowcharts, overlapping contents are denoted by the same reference numerals, and overlapping descriptions will be omitted.

Referring to FIG. 10 , the control method of the mobile robot includes a mode input step (not shown) for setting a home guard mode. In the mode input step, whether the home guard mode is On/Off may be selected.

Of course, the control method of the mobile robot may further include a command input step (not shown) of receiving a command to perform a task. In the command input step, the user may command the mobile robot 100 to perform the first operation.

The control method of the mobile robot 100 is performed only when the home guard mode is On. The mobile robot 100 may determine whether the home guard mode is On, and execute the following steps only when the home guard mode is On (S111).

In the step of detecting intrusion detection information (S113), the mobile robot 100 may detect intrusion detection information while patrolling a predetermined area or detect intrusion detection information at a predetermined location. For example, the mobile robot may collect image and sound information around the entrance while waiting around the entrance, and store it in a memory. Here, the intrusion detection information means all information when an external intrusion is suspected. The intrusion detection information may include the type of the first object and information on the moving direction of the first object.

In the step (S115) of delivering the intrusion detection information, the mobile robot determines whether there is a dangerous situation based on the intrusion detection information roll, and in case of a dangerous situation (for example, the second difference image analysis result is a report target), Transmits the corresponding intrusion detection information (control signal) to the terminal.

The terminal receiving the intrusion detection information may output the image received from the cleaner 100 on the display of the terminal based on the intrusion detection information. At this time, an icon for receiving an alarm generating command may be displayed on the display of the terminal.

Thereafter, the terminal may receive a user's control command (S119). The user's control command may be inputted by various input units, but it is preferable to receive the input through the control screen 301 .

Thereafter, the terminal may convert the control command into a terminal signal ( S121 ), and transmit the terminal signal to at least one of a mobile robot and a home appliance.

The mobile robot that has received the terminal signal from the terminal extracts an alarm generating command from the terminal signal transmitted from the terminal (S125, S127).

The mobile robot may generate a control signal so that at least one of the mobile robot and the home appliance outputs an alert according to an alert generating command (S126, S128). If the mobile robot and the home appliance together output an alarm, an alarm can be effectively given to an intruder from the outside, so there is an advantage in that the intruder can be expelled.

11 is a conceptual diagram illustrating a method in which a mobile robot detects a difference between a plurality of images according to another embodiment of the present invention.

Referring to FIG. 11 , according to another embodiment, the controller 1800 may detect the type of the first object and the movement direction information of the first object from the difference between the first image and the second image.

In detail, the camera of the cleaner 100 may capture the first image 701 and the second image 702 while the cleaner 100 performs a monitoring driving.

That is, during the monitoring driving, the controller 1800 may control the camera to capture an image every preset period.

The controller 1800 may compare the first image 701 and the second image 702 to detect a difference 703 generated between the first image 701 and the second image 702 .

In addition, the controller 1800 may generate a first difference image 704 based on a difference generated between the first image 701 and the second image 702 . For example, the controller 1800 may generate the first difference image 704 by cropping a portion of the second image 702 that is different from the first image 701 .

The controller 1800 may include a deep learning algorithm unit for performing a deep learning algorithm and an image recognition unit. The controller 1800 may compare the first difference image 704 with a plurality of training data stored in the server 500 using a deep learning algorithm. The controller 1800 may detect at least one training data corresponding to the first difference image 704 among the plurality of training data. In addition, the controller 1800 may detect information related to the first subject of the first difference image 704 by using label information of the detected training data.

The controller 1800 may analyze the first difference image 704 to detect movement direction related information of the first subject.

As shown in FIG. 11 , the controller 1800 may recognize the first subject as a door and recognize the moving direction of the first subject from right to left or left to right. When the moving direction of the first subject coincides with a preset direction, the controller 1800 may determine that the analysis result of the first difference image 704 is a report target.

Claims (19)

a camera for taking an image related to the cleaning area in which the main body is located at every preset cycle;
a memory for storing information related to a preset algorithm for analyzing the image;
A controller configured to detect a difference between a first image, a second image, and a third image continuously photographed by the camera, specify a first object based on the detected difference, and detect movement direction information of the first object ; and
Including a communication unit for performing communication with the outside,
The control unit is
When the first subject is a person and the moving direction of the first subject is a preset moving direction, controlling the communication unit to transmit information related to the first subject to at least one of a server and a terminal,
The control unit analyzes the second image to specify the position of the first subject,
The control unit is
If the first subject is a person and the position of the first subject is within the boundary area, if the moving direction of the first subject is a preset movement direction to correspond to the boundary area, the terminal displays the image of the first subject A mobile robot that controls to output. According to claim 1,
The first subject is specified,
detecting a difference between the first image and the second image, generating a first difference image based on the detected difference, and analyzing the generated first difference image to detect information related to the first subject mobile robot. 3. The method of claim 2,
The control unit is
A mobile robot that determines a degree of similarity between a plurality of pieces of image information included in training data and the first difference image, and detects information related to the first subject based on the determined degree of similarity. According to claim 1,
The control unit is
A difference between the second image and the third image is detected, a second difference image is generated based on the detected difference, and the generated second difference image is analyzed to detect movement direction information of the first subject. moving robot. delete delete According to claim 1,
The control unit is
When the first subject is a human and the moving direction of the first subject is a preset moving direction, the mobile robot controls the terminal to output an image of the first subject. According to claim 1,
The control unit is
When the first subject is a person and the moving direction of the first subject is a preset moving direction, the mobile robot controls the communication unit to transmit a warning message to a terminal. According to claim 1,
The control unit is
When the first subject is a door and the moving direction of the first subject is a preset moving direction, the mobile robot controls the communication unit to transmit information related to the first subject to at least one of a server and a terminal. According to claim 1,
The control unit is
When the first subject is a door and the moving direction of the first subject is a preset moving direction, the mobile robot controls the communication unit to transmit a warning message to a terminal. According to claim 1,
The control unit is
A mobile robot that controls the terminal to activate a reference information input screen that receives reference information that is a standard for outputting an alarm. According to claim 1,
The mobile robot further comprising a driving unit for moving the main body. 13. The method of claim 12,
The control unit is
A mobile robot that controls the driving unit so that the first image, the second image, and the third image are acquired at the same point in the same camera direction. 13. The method of claim 12,
The control unit is
When the monitoring driving of the cleaner is started, the main body controls the driving unit to traverse a plurality of preset points in the cleaning area,
A mobile robot for controlling the camera to photograph the first to third images at each of the plurality of points. delete According to claim 1,
The control unit is
When the first subject is a person, the location of the first subject is within the boundary area, and the moving direction of the first subject is a preset movement direction to correspond to the boundary area, information related to the first subject is transmitted to the server and a mobile robot for controlling the communication unit to be transmitted to at least one of the terminals. delete 17. The method of claim 16,
The control unit is
When the first subject is a person and the position of the first subject is within the boundary area, if the movement direction of the first object is a preset movement direction corresponding to the boundary area, the communication unit is configured to transmit a warning message to the terminal. Controlled mobile robot. 12. The method of claim 11,
The reference information is information on a type of the first object corresponding to each boundary area and information on a movement direction of the first object.

Images