KR102548936B1 - Artificial intelligence Moving robot and control method thereof - Google Patents

Abstract

An artificial intelligent mobile robot according to an aspect of the present invention includes a driving unit for moving the body, an image acquisition unit for acquiring an image around the body, a storage unit for storing the image obtained by the image acquisition unit, and one for detecting obstacles during movement. By including a sensor unit including the above sensors, and a control unit that controls to extract a part of the image obtained by the image acquisition unit in response to the direction of the obstacle detected by the sensor unit, extracting data effective for machine learning and obstacle attribute recognition can do.

Description

Artificial intelligence moving robot and control method thereof {Artificial intelligence Moving robot and control method thereof}

The present invention relates to a mobile robot and a control method thereof, and more particularly, to a mobile robot performing obstacle recognition and avoidance based on machine learning and a control method thereof.

Robots have been developed for industrial use and have been in charge of a part of factory automation. In recent years, the field of application of robots has been further expanded, and medical robots, space robots, etc. have been developed, and household robots that can be used at home are also being made. Among these robots, those capable of driving on their own are called mobile robots.

A representative example of a mobile robot used in a home is a robot cleaner. The robot cleaner is a device that cleans a corresponding area by sucking in dust or foreign substances while traveling in a certain area by itself.

The mobile robot is free to move as it can move by itself, and is equipped with a plurality of sensors for avoiding obstacles while driving so as to avoid obstacles.

In general, an infrared sensor or an ultrasonic sensor is used to detect an obstacle of a mobile robot. The infrared sensor determines the presence and distance of an obstacle through the amount of reflected light reflected from the obstacle or the time it is received, and the ultrasonic sensor emits ultrasonic waves having a predetermined cycle, and when there is ultrasonic waves reflected by obstacles, the ultrasonic emission time The distance to the obstacle is determined using the time difference between the reflection and the moment when the object is reflected and returned.

On the other hand, since obstacle recognition and avoidance greatly affect not only the driving performance of the mobile robot but also the cleaning performance, it is required to secure the reliability of the obstacle recognition capability.

The prior art (Patent Registration No. 10-0669892) discloses a technique for implementing highly reliable obstacle recognition technology by combining an infrared sensor and an ultrasonic sensor.

However, the prior art (Patent Registration No. 10-0669892) has a problem in that the attribute of the obstacle cannot be determined.

1 is a diagram referenced in a description of a method of detecting and avoiding an obstacle of a conventional mobile robot.

Referring to FIG. 1 , the robot cleaner cleans by sucking in dust and foreign substances while moving (S11).

The ultrasonic sensor detects the ultrasonic signal reflected by the obstacle to recognize the existence of the obstacle (S12), and determines whether the height of the recognized obstacle is a height that can be crossed (S13).

If it is determined that the height can be crossed, the robot cleaner moves straight ahead (S14), and if not, it may rotate 90 degrees (S15) and move.

For example, if the obstacle is a low threshold. The robot cleaner recognizes the threshold and, as a result of the recognition, determines that it can pass through, and moves beyond the threshold.

However, if the obstacle determined to be a height that can be crossed is a wire, the robot cleaner may get caught in the wire while crossing the wire and be constrained.

In addition, since the support of the fan has a height similar to or lower than the threshold, the robot cleaner may determine that the robot cleaner is an obstacle that can be overcome. In this case, the robot cleaner may be constrained while climbing on the pedestal of the fan while its wheels spin.

In addition, when the entire person is not recognized and only a part of human hair such as hair is detected, it may be determined that the height can exceed the human hair and go straight. However, in this case, the robot cleaner may suck in human hair and a safety accident may occur. there is.

Therefore, there is a need for a way to change the movement pattern to match the property by identifying the property of the front obstacle.

On the other hand, interest in machine learning, such as artificial intelligence and deep learning, has recently increased significantly.

Conventional machine learning has focused on classification, regression, and cluster models based on statistics. In particular, in the supervised learning of classification and regression models, humans pre-defined the characteristics of learning data and the learning model that distinguishes new data based on these characteristics. In contrast, in deep learning, computers find and discriminate characteristics on their own.

One of the factors that has accelerated the development of deep learning is open source deep learning frameworks. For example, deep learning frameworks include Theano from the University of Montreal, Canada, Torch from New York University, Caffe from the University of California, Berkeley, and TensorFlow from Google.

With the release of deep learning frameworks, in addition to deep learning algorithms, the learning process, learning method, and extraction and selection of data used for learning are becoming more important for effective learning and recognition.

In addition, research to use artificial intelligence and machine learning for various products and services is increasing.

Registered Patent Publication No. 10-0669892 (registration date: 2007. 1. 10.)

An object of the present invention is to provide a mobile robot and a control method thereof capable of performing highly reliable obstacle recognition and avoidance operations by determining the properties of obstacles and adjusting a driving pattern according to the properties of the obstacles.

An object of the present invention is to improve the stability of the mobile robot itself and the user's convenience by performing operations such as forward, backward, stop, and detour according to the result of recognizing an obstacle, and to improve driving efficiency and cleaning efficiency. And it is to provide a control method.

An object of the present invention is to provide a mobile robot capable of accurately recognizing an attribute of an obstacle based on machine learning and a control method thereof.

An object of the present invention is to provide a mobile robot capable of efficiently performing machine learning and extracting data usable for recognizing obstacle attributes and a control method thereof.

In order to achieve the above or other objects, a mobile robot according to an aspect of the present invention includes a driving unit for moving the body, an image acquisition unit for acquiring an image around the body, a storage unit for storing the image acquired by the image acquisition unit, and moving A sensor unit including one or more sensors for detecting obstacles, and a control unit for controlling extraction of a part of an image acquired by the image acquisition unit in response to the direction of the obstacle detected by the sensor unit, thereby enabling machine learning and obstacle properties Efficient data can be extracted for recognition.

In addition, in order to achieve the above or other objects, the mobile robot according to one aspect of the present invention is based on an obstacle recognition module for recognizing an obstacle pre-learned by machine learning from an extracted image and an attribute of the recognized obstacle Thus, stability, user's convenience, driving efficiency, and cleaning efficiency may be improved by including a driving control module for controlling driving of the driving unit.

In addition, in order to achieve the above or other objects, a control method of a mobile robot according to an aspect of the present invention includes detecting an obstacle during movement through a sensor unit, and obtaining an image acquisition unit in response to the direction of the obstacle detected by the sensor unit. Extracting a partial region of an image to be played, recognizing an obstacle pre-learned through machine learning in the extracted image, and controlling the driving of the driving unit based on the attribute of the recognized obstacle. can do.

According to at least one of the embodiments of the present invention, the mobile robot can determine the property of the obstacle and adjust the driving pattern according to the property of the obstacle, thereby performing highly reliable obstacle recognition and avoidance operations.

In addition, according to at least one of the embodiments of the present invention, the stability of the mobile robot itself and the user's convenience are improved by performing operations such as forward, backward, stop, and detour according to the recognition result of the obstacle, and driving efficiency and cleaning efficiency It is possible to provide a mobile robot and its control method that can improve the.

In addition, according to at least one of the embodiments of the present invention, it is possible to provide a mobile robot capable of accurately recognizing the property of an obstacle based on machine learning and a control method thereof.

In addition, according to at least one of the embodiments of the present invention, the mobile robot can efficiently perform machine learning and extract data usable for obstacle attribute recognition.

Meanwhile, other various effects will be disclosed directly or implicitly in detailed descriptions according to embodiments of the present invention to be described later.

1 is a diagram referenced in a description of a method of detecting and avoiding an obstacle of a conventional mobile robot.
2 is a perspective view showing a mobile robot and a charging stand for charging the mobile robot according to an embodiment of the present invention.
FIG. 3 is a diagram showing an upper surface of the mobile robot shown in FIG. 2;
FIG. 4 is a view showing the front of the mobile robot shown in FIG. 2;
FIG. 5 is a diagram illustrating a bottom portion of the mobile robot shown in FIG. 2;
6 and 7 are block diagrams showing a control relationship between major components of a mobile robot according to an embodiment of the present invention.
8 is an example of a simplified internal block diagram of a server according to an embodiment of the present invention.
9 to 12 are diagrams referenced for description of deep learning.
13 is a flowchart illustrating a control method of a mobile robot according to an embodiment of the present invention.
14 is a flowchart illustrating a control method of a mobile robot according to an embodiment of the present invention.
15 to 21 are views referenced for description of a control method of a mobile robot according to an embodiment of the present invention.
22 is a diagram referenced in a description of a method of operating a mobile robot and a server according to an embodiment of the present invention.
23 is a flowchart illustrating a method of operating a mobile robot and a server according to an embodiment of the present invention.
24 to 26 are views referenced for description of a control method of a mobile robot according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it goes without saying that the present invention is not limited to these embodiments and can be modified in various forms.

In the drawings, in order to clearly and concisely describe the present invention, parts not related to the description are omitted, and the same reference numerals are used for the same or extremely similar parts throughout the specification.

On the other hand, the suffixes "module" and "unit" for components used in the following description are simply given in consideration of the ease of preparation of this specification, and do not themselves give a particularly important meaning or role. Accordingly, the “module” and “unit” may be used interchangeably.

The mobile robot 100 according to an embodiment of the present invention refers to a robot capable of moving by itself using wheels, etc., and may be a home helper robot or a robot vacuum cleaner. Hereinafter, a robot cleaner having a cleaning function among mobile robots will be described as an example with reference to the drawings, but the present invention is not limited thereto.

2 is a perspective view showing a mobile robot and a charging stand for charging the mobile robot according to an embodiment of the present invention.

FIG. 3 is a view showing the top of the mobile robot shown in FIG. 2, FIG. 4 is a view showing the front of the mobile robot shown in FIG. 2, and FIG. 5 shows the bottom of the mobile robot shown in FIG. it is one degree

6 and 7 are block diagrams showing a control relationship between major components of a mobile robot according to an embodiment of the present invention.

Referring to FIGS. 3 to 7 , the mobile robots 100, 100a, and 100b include a body 110 and image acquisition units 120, 120a, and 120b that acquire images around the body 110.

Hereinafter, in defining each part of the main body 110, the part facing the ceiling in the driving area is defined as an upper part (see FIG. 3), and the part facing the floor in the driving area is defined as a bottom part (see FIG. 5). And, among the parts forming the circumference of the main body 110 between the upper and lower surfaces, the part facing the driving direction is defined as the front part (see FIG. 4).

The mobile robots 100, 100a, and 100b include a driving unit 160 that moves the main body 110. The driving part 160 includes at least one driving wheel 136 for moving the main body 110 . The driving unit 160 includes a driving motor (not shown) connected to the driving wheel 136 to rotate the driving wheel. The driving wheels 136 may be provided on the left and right sides of the body 110, respectively, and are hereinafter referred to as a left wheel 136(L) and a right wheel 136(R), respectively.

The left wheel 136(L) and the right wheel 136(R) may be driven by a single drive motor, but the left wheel drive motor and the right wheel 136(R) driving the left wheel 136(L) as needed ) may be provided respectively with a right wheel drive motor for driving. The running direction of the main body 110 can be switched to the left or right by making a difference between the rotation speeds of the left wheel 136 (L) and the right wheel 136 (R).

A suction port 110h through which air is sucked may be formed on the bottom of the main body 110, and a suction device (not shown) provides a suction force so that air can be sucked through the suction hole 110h in the main body 110. ), and a dust bin (not shown) for collecting dust sucked together with air through the inlet 110h may be provided.

The main body 110 may include a case 111 forming a space in which various parts constituting the mobile robots 100, 100a, and 100b are accommodated. An opening for inserting and removing the dust bin may be formed in the case 111 , and a dust bin cover 112 that opens and closes the opening may be rotatably provided with respect to the case 111 .

A roll-type main brush 134 having brushes exposed through the suction port 110h, and an auxiliary brush 135 located on the front side of the bottom surface of the main body 110 and having a brush made of a plurality of wings extending radially may be provided. By the rotation of these brushes 134 and 135, dust is separated from the floor in the driving area, and the dust separated from the floor is sucked through the suction port 110h and collected in the dust bin.

The battery 138 supplies power necessary for the overall operation of the mobile robots 100, 100a, and 100b as well as the driving motor. When the battery 138 is discharged, the mobile robots 100, 100a, and 100b may return to the charging station 200 for charging, and during the return travel, the mobile robots 100, 100a, and 100b can detect the position of the charging stand 200 by itself.

The charging station 200 may include a signal transmission unit (not shown) that transmits a predetermined return signal. The return signal may be an ultrasonic signal or an infrared signal, but is not necessarily limited thereto.

The mobile robots 100, 100a, and 100b may include a signal detector (not shown) for receiving a return signal. The charging station 200 may transmit an infrared signal through a signal transmission unit, and the signal detection unit may include an infrared sensor for detecting the infrared signal. The mobile robots 100, 100a, and 100b move to the location of the charging station 200 according to the infrared signal transmitted from the charging station 200 and dock with the charging station 200. Due to this docking, charging is performed between the charging terminals 133 of the mobile robots 100, 100a, and 100b and the charging terminals 210 of the charging base 200.

The image acquisition unit 120 captures a driving area and may include a camera module. The camera module may include a digital camera. A digital camera includes an image sensor (eg, CMOS image sensor) including at least one optical lens and a plurality of photodiodes (eg, pixels) forming images by light passing through the optical lens; A digital signal processor (DSP) for composing an image based on signals output from the photodiodes may be included. The digital signal processor can create not only still images, but also moving images composed of frames composed of still images.

Preferably, the image acquisition unit 120 is provided with a front camera 120a provided to acquire an image of the front of the main body 110 and an upper surface of the main body 110 to acquire an image of the ceiling in the driving area. Although an upper camera 120b is provided, the location and shooting range of the image acquisition unit 120 are not necessarily limited thereto.

In the case of this embodiment, a camera is installed in some parts (eg, front, rear, bottom) of the mobile robot, and captured images can be continuously acquired during cleaning. Several such cameras may be installed for each part for shooting efficiency. The image captured by the camera can be used to recognize the type of material such as dust, hair, floor, etc. present in the space, whether to clean, or to confirm the cleaning time.

The front camera 120a may capture an obstacle existing in front of the mobile robot 100, 100a, or 100b in the driving direction or a situation of the cleaning area.

According to an embodiment of the present invention, the image acquisition unit 120 may obtain a plurality of images by continuously photographing the surroundings of the main body 110, and the obtained plurality of images may be stored in the storage unit 150. can

The mobile robots 100, 100a, and 100b can increase the accuracy of obstacle recognition by using a plurality of images or by selecting one or more images from among the plurality of images and using effective data.

In addition, the mobile robots 100, 100a, and 100b may include a sensor unit 170 including sensors for sensing various data related to the operation and state of the mobile robot.

For example, the sensor unit 170 may include an obstacle detection sensor 131 for detecting an obstacle in front. In addition, the sensor unit 170 may further include a cliff detection sensor 132 for detecting whether a cliff exists on the floor in the driving area and a lower camera sensor 139 for obtaining an image of the floor.

Referring to FIGS. 2 and 4 , the obstacle detection sensor 131 may include a plurality of sensors installed at regular intervals on the outer circumferential surface of the mobile robot 100 .

For example, the sensor unit 170 may include a first sensor disposed on the front surface of the main body 110, and a second sensor and a third sensor disposed to be spaced apart from the first sensor in left and right directions. .

The obstacle detection sensor 131 may include an infrared sensor, an ultrasonic sensor, an RF sensor, a geomagnetic sensor, a Position Sensitive Device (PSD) sensor, and the like.

Meanwhile, the location and type of sensors included in the obstacle detection sensor 131 may vary depending on the type of mobile robot, and the obstacle detection sensor 131 may include more various sensors.

The obstacle detection sensor 131 is a sensor that detects a distance to an indoor wall or an obstacle, and the present invention is not limited to that type, but an ultrasonic sensor will be described below as an example.

The obstacle detection sensor 131 detects an object, particularly an obstacle, existing in the driving (moving) direction of the mobile robot and transmits obstacle information to the control unit 140. That is, the obstacle detection sensor 131 can detect the moving path of the mobile robot, protrusions in the front or side, furniture, furniture, wall surfaces, wall corners, etc., and transmit the information to the control unit.

In this case, the controller 140 may detect the position of the obstacle based on at least two or more signals received through the ultrasonic sensor, and control the movement of the mobile robot 100 according to the detected position of the obstacle.

Depending on the embodiment, the obstacle detection sensor 131 provided on the outer surface of the case 110 may include a transmitter and a receiver.

For example, in the ultrasonic sensor, at least one transmitting unit and at least two receiving units may be provided so that they are crossed with each other. Accordingly, signals can be radiated at various angles and signals reflected by obstacles can be received at various angles.

Depending on the embodiment, the signal received from the obstacle detection sensor 131 may undergo signal processing such as amplification and filtering, and then the distance and direction to the obstacle may be calculated.

Meanwhile, the sensor unit 170 may further include a motion detection sensor that detects motions of the mobile robots 100, 100a, and 100b according to the driving of the main body 110 and outputs motion information. As the motion detection sensor, a gyro sensor, a wheel sensor, an acceleration sensor, or the like may be used.

The gyro sensor detects a rotation direction and a rotation angle when the mobile robots 100, 100a, and 100b move according to the driving mode. The gyro sensor detects the angular velocity of the mobile robots 100, 100a, and 100b and outputs a voltage value proportional to the angular velocity. The controller 140 calculates the rotation direction and rotation angle using the voltage value output from the gyro sensor.

The wheel sensor is connected to the left wheel 136 (L) and the right wheel 136 (R) to detect the number of revolutions of the wheel. Here, the wheel sensor may be a rotary encoder. The rotary encoder detects and outputs the number of revolutions of the left wheel 136(L) and the right wheel 136(R).

The controller 140 may calculate the rotation speed of the left and right wheels using the number of revolutions. In addition, the controller 140 may calculate the rotation angle using a difference in rotation speeds between the left wheel 136 (L) and the right wheel 136 (R).

The acceleration sensor detects a change in the speed of the mobile robot 100, 100a, 100b, for example, a change in the mobile robot 100, 100a, 100b according to start, stop, direction change, collision with an object, and the like. The acceleration sensor is attached to a location adjacent to the main wheel or the auxiliary wheel, and can detect slippage or idling of the wheel.

In addition, the acceleration sensor may be built into the control unit 140 to detect a change in speed of the mobile robots 100, 100a, and 100b. That is, the acceleration sensor detects the amount of impact according to the speed change and outputs a corresponding voltage value. Accordingly, the acceleration sensor may perform a function of an electronic bumper.

The controller 140 may calculate a positional change of the mobile robots 100, 100a, and 100b based on motion information output from the motion detection sensor. This position becomes a relative position corresponding to the absolute position using image information. The mobile robot can improve the performance of location recognition using image information and obstacle information through relative location recognition.

Meanwhile, the mobile robots 100, 100a, and 100b may include a power supply unit (not shown) having a rechargeable battery 138 to supply power to the robot cleaner.

The power supply unit supplies driving power and operating power to each component of the mobile robots 100, 100a, and 100b, and when the remaining power is insufficient, it can be charged by receiving charging current from the charging station 200.

The mobile robots 100, 100a, and 100b may further include a battery detector (not shown) that detects the state of charge of the battery 138 and transmits the detection result to the controller 140. The battery 138 is connected to the battery detection unit, and the battery remaining amount and charging state are transmitted to the controller 140 . The remaining battery capacity may be displayed on a screen of an output unit (not shown).

In addition, the mobile robots 100, 100a, and 100b include a manipulation unit 137 capable of inputting on/off or various commands. Various control commands necessary for the overall operation of the mobile robot 100 may be received through the control unit 137 . In addition, the mobile robots 100, 100a, and 100b may display reservation information, battery status, operation mode, operation status, error status, and the like, including an output unit (not shown).

Referring to FIGS. 6 and 7 , the mobile robots 100a and 100b include a controller 140 that processes and determines various information such as recognizing a current location, and a storage unit 150 that stores various data. In addition, the mobile robots 100, 100a, and 100b may further include a communication unit 190 that transmits and receives data with an external terminal.

The external terminal has an application for controlling the mobile robots 100a and 100b, displays a map of the driving area to be cleaned by the mobile robots 100a and 100b through the execution of the application, and cleans a specific area on the map. area can be specified. The external terminal may include, for example, a remote control, a PDA, a laptop, a smart phone, a tablet, etc. loaded with an application for setting a map.

The external terminal may communicate with the mobile robots 100a and 100b to display the current location of the mobile robot along with a map, and information on a plurality of areas may be displayed. In addition, the external terminal updates and displays its location according to the movement of the mobile robot.

The control unit 140 controls the overall operation of the mobile robot 100 by controlling the image acquisition unit 120, the manipulation unit 137, and the driving unit 160 constituting the mobile robots 100a and 100b.

The storage unit 150 records various information necessary for controlling the mobile robot 100, and may include a volatile or non-volatile recording medium. Recording media stores data that can be read by a microprocessor, such as HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tapes, floppy disks, optical data storage devices, and the like.

Also, a map for a driving area may be stored in the storage unit 150 . The map may be input by an external terminal or server capable of exchanging information with the mobile robots 100a and 100b through wired or wireless communication, or may be generated by the mobile robots 100a and 100b through self-learning there is.

Locations of rooms within the driving area may be displayed on the map. Also, the current positions of the mobile robots 100a and 100b may be displayed on a map, and the current positions of the mobile robots 100a and 100b on the map may be updated during driving. The external terminal stores the same map as the map stored in the storage unit 150 .

The storage unit 150 may store cleaning history information. Such cleaning history information may be generated whenever cleaning is performed.

The map for the driving area stored in the storage unit 150 is a navigation map used for driving during cleaning, a SLAM (Simultaneous Localization and Mapping) map used for location recognition, and an obstacle when an obstacle is encountered. It may be a learning map used for learning and cleaning by storing information, a global location map used for global location recognition, an obstacle recognition map in which information on recognized obstacles is recorded, and the like.

Meanwhile, as described above, maps may be stored and managed by dividing them into the storage unit 150 for each purpose, but maps may not be clearly classified for each purpose. For example, a plurality of pieces of information may be stored in one map so that they can be used for at least two purposes.

The control unit 140 may include a driving control module 141, a location recognition module 142, a map generation module 143, and an obstacle recognition module 144.

3 to 7, the travel control module 141 controls the travel of the mobile robots 100, 100a, and 100b, and controls the driving of the travel unit 160 according to travel settings. In addition, the driving control module 141 may determine the driving path of the mobile robots 100, 100a, and 100b based on the operation of the driving unit 160. For example, the driving control module 141 can determine the current or past moving speed of the mobile robot 100, the distance traveled, etc. based on the rotational speed of the driving wheels 136, and each driving wheel 136 ( According to the rotation direction of L) and 136(R)), the current or past direction change process can also be grasped. Based on the driving information of the mobile robots 100, 100a, and 100b identified in this way, the location of the mobile robots 100, 100a, and 100b may be updated on the map.

The map generation module 143 may generate a map of the driving area. The map generation module 143 may create a map by processing the image obtained through the image acquisition unit 120 . That is, a cleaning map corresponding to the cleaning area may be created.

In addition, the map generation module 143 can recognize the global location by processing the image acquired through the image acquisition unit 120 at each location and linking it with a map.

The location recognition module 142 estimates and recognizes the current location. The location recognition module 142 identifies the location in conjunction with the map generation module 143 using the image information of the image acquisition unit 120, even when the location of the mobile robots 100, 100a, 100b suddenly changes. It can be recognized by estimating the current location.

The mobile robots 100, 100a, and 100b can perform location recognition during continuous driving through the location recognition module 142, and also use the map generation module 143 and the obstacle recognition module 144 without the location recognition module 142. Through this, the map can be learned and the current location can be estimated.

While the mobile robots 100, 100a, and 100b are driving, the image acquisition unit 120 acquires images around the mobile robot 100. Hereinafter, an image acquired by the image acquisition unit 120 is defined as an 'obtained image'.

The obtained image includes various features such as lights located on the ceiling, edges, corners, blobs, and ridges.

The map generating module 143 detects features from each of the acquired images. Various methods (Feature Detection) of detecting a feature from an image are well known in the field of computer vision technology. Several feature detectors suitable for detection of these features are known. For example, Canny, Sobel, Harris & Stephens/Plessey, SUSAN, Shi & Tomasi, Level curve curvature, FAST, Laplacian of Gaussian, Difference of Gaussians, Determinant of Hessian, MSER, PCBR, Gray-level blobs detector, etc.

The map generating module 143 calculates a descriptor based on each feature point. The map generating module 143 may convert a feature point into a descriptor using a Scale Invariant Feature Transform (SIFT) technique for feature detection. The descriptor may be expressed as an n-dimensional vector.

SIFT can detect features that are invariant with respect to changes in scale, rotation, and brightness of the subject to be photographed, so that even if the same area is photographed with different postures of the mobile robot 100, it is invariant (i.e., rotation invariant (Rotation)). -invariant)) features can be detected. Of course, it is not limited thereto, and various other techniques (eg, HOG: Histogram of Oriented Gradient, Haar feature, Fems, LBP: Local Binary Pattern, MCT: Modified Census Transform) may be applied.

The map generation module 143 classifies at least one descriptor for each acquired image into a plurality of groups according to a predetermined sub-classification rule, based on the descriptor information obtained through the acquired image of each location, and the same group according to a predetermined sub-representative rule. The descriptors included in may be converted into sub-representative descriptors, respectively.

As another example, all descriptors gathered from acquired images in a predetermined area, such as a room, are classified into a plurality of groups according to a predetermined sub-classification rule, and the descriptors included in the same group according to the predetermined sub-representation rule are respectively classified as sub-representative descriptors. can also be converted to

The map generation module 143 may obtain the feature distribution of each location through the above process. Each location feature distribution can be expressed as a histogram or an n-dimensional vector. As another example, the map generation module 143 may estimate an unknown current location based on a descriptor calculated from each feature point without going through a predetermined subclassification rule and a predetermined subrepresentation rule.

In addition, when the current location of the mobile robots 100, 100a, and 100b becomes unknown due to location jumping, etc., the current location can be estimated based on data such as a pre-stored descriptor or sub-representative descriptor.

The mobile robots 100, 100a, and 100b obtain an acquired image from an unknown current location through the image acquisition unit 120. Through the video, various features such as lights located on the ceiling, edges, corners, blobs, and ridges are confirmed.

The location recognition module 142 detects features from the acquired image. Descriptions of various methods for detecting features from images in the field of computer vision technology and various feature detectors suitable for detecting these features have been described above.

The location recognition module 142 calculates a recognition descriptor through a recognition descriptor calculation step based on each recognition feature point. At this time, the recognition feature points and recognition descriptors are for explaining the process performed by the obstacle recognition module 144, and are to be distinguished from terms describing the process performed by the map generation module 143. However, it is merely that the characteristics of the external world of the mobile robots 100, 100a, and 100b are defined in different terms.

The location recognition module 142 may convert the recognition feature point into a recognition descriptor using a Scale Invariant Feature Transform (SIFT) technique to detect this feature. The recognition descriptor may be expressed as an n-dimensional vector.

As described above, SIFT selects easily identifiable feature points such as corner points in the acquired image, and then determines the distribution characteristics of the brightness gradient of pixels belonging to a certain area around each feature point (direction of brightness change and rapid degree of change). ), it is an image recognition technique that obtains an n-dimensional vector with the rapid degree of change in each direction as a numerical value for each dimension.

The location recognition module 142 is based on at least one recognition descriptor information obtained through an acquired image of an unknown current location, and location information (eg, feature distribution of each location) to be compared according to a predetermined sub-conversion rule. It is converted into information (lower recognition feature distribution) that can be compared with .

According to a predetermined subcomparison rule, each location feature distribution may be compared with each recognition feature distribution to calculate each degree of similarity. A similarity (probability) is calculated for each location corresponding to each location, and a location where the highest probability is calculated among them can be determined as the current location.

In this way, the control unit 140 can divide the driving area and create a map consisting of a plurality of areas, or recognize the current location of the main body 110 based on a previously stored map.

When a map is generated, the control unit 140 may transmit the generated map to an external terminal or server through the communication unit 190 . Also, as described above, the controller 140 may store the map in the storage unit when the map is received from an external terminal or server.

In addition, when the map is updated while driving, the controller 140 transmits the updated information to the external terminal so that the external terminal and the maps stored in the mobile robots 100, 100a, and 100b are the same. As the maps stored in the external terminal and the mobile robots 100, 100a, and 100b remain the same, the mobile robots 100, 100a, and 100b can clean the designated area in response to a cleaning command from the mobile terminal, and also, the external terminal This is to enable the current location of the mobile robot to be displayed on the terminal.

In this case, the map may divide the cleaning area into a plurality of areas, include a connection passage connecting the plurality of areas, and include information about obstacles in the area.

When a cleaning command is input, the controller 140 determines whether the location on the map matches the current location of the mobile robot. The cleaning command may be input from a remote control, a control panel, or an external terminal.

If the current location does not match the location on the map, or if the current location cannot be confirmed, the controller 140 recognizes the current location and recovers the current location of the mobile robot 100, and then based on the current location The driving unit 160 may be controlled to move to a designated area.

If the current location does not match the location on the map or if the current location cannot be confirmed, the location recognition module 142 analyzes the acquired image input from the image acquisition unit 120 to estimate the current location based on the map. can do. In addition, the obstacle recognition module 144 or the map generation module 143 may also recognize the current location in the same way.

After recognizing the position and restoring the current position of the mobile robots 100, 100a, 100b, the driving control module 141 calculates a driving route from the current position to the designated area and controls the driving unit 160 to return to the designated area. move

When receiving cleaning pattern information from the server, the driving control module 141 may divide the entire driving area into a plurality of areas and set one or more areas as a designated area according to the received cleaning pattern information.

In addition, the driving control module 141 may calculate a driving route according to the received cleaning pattern information, drive along the driving route, and perform cleaning.

The controller 140 may store the cleaning record in the storage unit 150 when cleaning of the set designated area is completed.

In addition, the control unit 140 may transmit the operating state or cleaning state of the mobile robot 100 to an external terminal or server through the communication unit 190 at predetermined intervals.

Accordingly, the external terminal displays the location of the mobile robot along with a map on the screen of the running application based on the received data, and also outputs information on the cleaning status.

The mobile robots 100, 100a, and 100b according to an embodiment of the present invention move in one direction until an obstacle or wall is detected, and when the obstacle recognition module 144 recognizes the obstacle, it moves straight according to the attribute of the recognized obstacle. , rotation, etc. driving patterns can be determined.

For example, if the attribute of the recognized obstacle is a surmountable type of obstacle, the mobile robots 100, 100a, and 100b may continue to move forward. Alternatively, if the attribute of the recognized obstacle is a type of obstacle that cannot be crossed, the mobile robots 100, 100a, and 100b rotate and move a certain distance, and move again in the opposite direction to the initial movement direction to the distance at which the obstacle is detected, zigzag can run in the form

The mobile robots 100, 100a, and 100b according to an embodiment of the present invention may perform obstacle recognition and avoidance based on machine learning.

The controller 140 controls the driving of the driving unit 160 based on the obstacle recognition module 144 for recognizing an obstacle pre-learned by machine learning in an input image and the attribute of the recognized obstacle. A driving control module 141 for controlling may be included.

The mobile robots 100, 100a, and 100b according to an embodiment of the present invention may include an obstacle recognition module 144 in which attributes of obstacles are learned through machine learning.

Machine learning means that a computer learns through data without a person directly instructing the computer with logic, and through this, the computer solves the problem on its own.

Deep Learning is. Based on artificial neural networks (ANNs) to construct artificial intelligence, it is an artificial intelligence technology that allows computers to learn like humans on their own without being taught by humans as a method of teaching computers how to think.

The artificial neural network (ANN) may be implemented in the form of software or hardware such as a chip.

The obstacle recognition module 144 may include an artificial neural network (ANN) in the form of software or hardware in which attributes of obstacles are learned.

For example, the obstacle recognition module 144 uses a deep neural network (DNN) such as a convolutional neural network (CNN), a recurrent neural network (RNN), and a deep belief network (DBN) learned by deep learning. can include

Deep learning will be described later in detail with reference to FIGS. 9 to 12 and the like.

The obstacle recognition module 144 may determine attributes of obstacles included in input image data based on weights between nodes included in the deep neural network (DNN).

Meanwhile, when the sensor unit 170 detects an obstacle while the mobile robots 100, 100a, and 100b are moving, the control unit 140 responds to the direction of the obstacle detected by the sensor unit 170 to the image acquisition unit ( 120) can be controlled to extract a partial area of the acquired image.

The image acquisition unit 120, particularly the front camera 120a, may obtain an image within a predetermined angular range in the moving direction of the mobile robots 100, 100a, and 100b.

The control unit 140 may determine the attribute of the obstacle existing in the moving direction by using only a part of the image, not the entire image acquired by the image acquisition unit 120, particularly the front camera 120a.

Referring to the embodiment of FIG. 6 , the controller 140 performs image processing to extract a partial area of an image acquired by the image acquisition unit 120 in response to the direction of an obstacle detected by the sensor unit 170. A module 145 may be further included.

Alternatively, referring to the embodiment of FIG. 7 , the mobile robot 100b extracts a partial region of an image acquired by the image acquisition unit 120 in response to the direction of an obstacle detected by the sensor unit 170. A separate image processing unit 125 may be further included.

The mobile robot 100a according to the embodiment of FIG. 6 and the mobile robot 100b according to the embodiment of FIG. 7 have the same configuration other than the image processing module 145 and the image processing unit 125.

Alternatively, depending on embodiments, unlike the embodiments of FIGS. 6 and 7 , the image acquisition unit 120 may directly extract a partial region of the image.

Due to its characteristics, the obstacle recognition module 144 learned through machine learning has a higher recognition rate as the learning target occupies a larger portion of the input image data.

Therefore, according to the present invention, the recognition rate can be increased by extracting different regions from images acquired by the image acquisition unit 120 according to the direction of the obstacle detected by the sensor unit 170, such as an ultrasonic sensor, and using them as data for recognition.

The obstacle recognition module 144 may recognize an obstacle based on pre-learned data from the extracted image through machine learning.

Also, the driving control module 141 may control the driving of the driving unit 160 based on the attribute of the recognized obstacle.

Meanwhile, when the obstacle is detected in the right direction of the front of the main body, the controller 140 extracts the lower right area of the image acquired by the image acquisition unit, and detects the obstacle in the left direction of the front of the main body. control to extract the lower left area of the image acquired by the image acquisition unit, and extract the lower center area of the image acquired by the image acquisition unit when the obstacle is sensed from the front direction of the main body. can do.

In addition, the control unit 140 may control extraction by shifting the region to be extracted in the image acquired by the image acquisition unit to correspond to the direction of the detected obstacle.

A configuration of cropping and extracting a portion of an image obtained by the image acquisition unit 120 will be described later in detail with reference to FIGS. 15 to 21 and the like.

In addition, when the sensor unit 170 detects an obstacle while the mobile robots 100, 100a, and 100b are moving, the control unit 140, based on the moving direction and moving speed of the main body 110, the image acquisition unit ( 120) may control to select an image of a specific viewpoint prior to the obstacle detection time of the sensor unit 170 among a plurality of consecutive images acquired.

When the image acquisition unit 120 acquires an image using the obstacle detection time of the sensor unit 170 as a trigger signal, the mobile robot continues to move, so the obstacle is not included in the acquired image or is included in a small amount. can

Therefore, in one embodiment of the present invention, the sensor unit 170 detects obstacles among a plurality of consecutive images acquired by the image acquisition unit 120 based on the moving direction and moving speed of the main body 110. An image of a specific viewpoint prior to the viewpoint can be selected and used as data for obstacle recognition.

Meanwhile, the obstacle recognition module 144 may recognize attributes of obstacles included in the selected specific viewpoint image based on pre-learned data through machine learning.

Meanwhile, the storage unit 150 may store input data for obstacle attribute determination and data for learning the deep neural network (DNN).

The storage unit 150 may store the original image obtained by the image acquisition unit 120 and the extracted images from which a predetermined area is extracted.

Also, depending on embodiments, weights and biases constituting the deep neural network (DNN) structure may be stored in the storage unit 150 .

Alternatively, according to embodiments, weights and biases constituting the deep neural network structure may be stored in an embedded memory of the obstacle recognition module 144 .

On the other hand, whenever the obstacle recognition module 144 extracts a partial region of the image acquired by the image acquisition unit 120, it performs a learning process using the extracted image as training data, or a predetermined number of After the above extracted images are obtained, a learning process may be performed.

That is, the obstacle recognition module 144 updates a deep neural network (DNN) structure such as a weight by adding a recognition result whenever an obstacle is recognized, or obtained after a predetermined number of training data is secured. A deep neural network (DNN) structure such as weights may be updated by performing a learning process with training data.

Alternatively, the mobile robots 100, 100a, and 100b may transmit the extracted image to a predetermined server through the communication unit 190 and receive data related to machine learning from the predetermined server.

In this case, the mobile robots 100, 100a, and 100b may update the obstacle recognition module 141 based on data related to machine learning received from the predetermined server.

8 is an example of a simplified internal block diagram of a server according to an embodiment of the present invention.

Referring to FIG. 8 , the server 70 may include a communication unit 820, a storage unit 830, a learning module 840, and a processor 810.

The processor 810 may control overall operations of the server 70 .

Meanwhile, the server 70 may be a server operated by a home appliance manufacturer, such as the mobile robots 100, 100a, and 100b, or a server operated by a service provider, or may be a kind of cloud server.

The communication unit 820 may receive various types of data, such as status information, operation information, and operation information, from home appliances such as portable terminals and mobile robots 100, 100a, and 100b, and gateways.

Further, the communication unit 820 may transmit data corresponding to various types of received information to home appliances such as portable terminals and mobile robots 100, 100a, and 100b, and gateways.

To this end, the communication unit 820 may include one or more communication modules such as an internet module and a mobile communication module.

The storage unit 830 may store received information and may have data for generating result information corresponding thereto.

Also, the storage unit 830 may store data used for machine learning, result data, and the like.

The learning module 840 may serve as a learner for home appliances such as the mobile robots 100, 100a, and 100b.

The learning module 840 may include an artificial neural network, for example, a deep neural network (DNN) such as a convolutional neural network (CNN), a recurrent neural network (RNN), and a deep belief network (DBN). Neural networks can be trained.

As the learning method of the learning module 840, both unsupervised learning and supervised learning may be used.

Meanwhile, the controller 810 may control to update the artificial neural network structure of home appliances such as the mobile robots 100, 100a, and 100b to the learned artificial neural network structure after learning according to settings.

9 to 12 are diagrams referenced for description of deep learning.

Deep learning technology, a type of machine learning, learns by going down to a multi-level deep level based on data.

Deep learning may represent a set of machine learning algorithms that extract core data from a plurality of data as the level increases.

The deep learning structure may include an artificial neural network (ANN). For example, the deep learning structure is composed of a deep neural network (DNN) such as a convolutional neural network (CNN), a recurrent neural network (RNN), and a deep belief network (DBN). can be

Referring to FIG. 9 , an artificial neural network (ANN) may include an input layer, a hidden layer, and an output layer. Each layer includes a plurality of nodes, and each layer is connected to the next layer. Nodes between adjacent layers may be connected to each other with a weight.

Referring to FIG. 10 , a computer (machine) discovers a certain pattern from input data 1010 and forms a feature map. The computer (machine) may extract from the low-level feature 1020, the middle-level feature 1030, and the high-level feature 1040, recognize the object, and output 1050 the result.

Artificial neural networks can be abstracted into higher-level features as they go to the next layer.

Referring to FIGS. 9 and 10 , each node may operate based on an activation model, and an output value corresponding to an input value may be determined according to the activation model.

An output value of an arbitrary node, eg, a low-level feature 1020 , may be input to a node of a next layer connected to the corresponding node, eg, a node of a mid-level feature 1030 . A node of the next layer, for example, a node of the mid-level feature 1030 may receive values output from a plurality of nodes of the low-level feature 1020 as input.

In this case, an input value of each node may be a value to which a weight is applied to an output value of a node of a previous layer. A weight may mean connection strength between nodes.

Also, the deep learning process can be seen as a process of finding an appropriate weight.

Meanwhile, an output value of an arbitrary node, for example, the mid-level feature 1030, may be input to a next layer connected to the corresponding node, for example, a node of a higher-level feature 1040. A node of the next layer, for example, a node of the high-level feature 1040 may receive values output from a plurality of nodes of the mid-level feature 1030 as input.

The artificial neural network may extract feature information corresponding to each level using a learned layer corresponding to each level. The artificial neural network can recognize a predetermined object by sequentially abstracting and utilizing feature information of the highest level.

For example, looking at the face recognition process by deep learning, the computer distinguishes bright pixels from dark pixels according to the brightness of the pixels from the input image, and after distinguishing simple shapes such as borders and edges, a little more complex shapes and can differentiate things. Finally, the computer can figure out the shape that defines the human face.

The deep learning structure according to the present invention may use various known structures. For example, the deep learning structure according to the present invention may be a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), and the like.

RNN (Recurrent Neural Network) is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes over time. .

DBN (Deep Belief Network) is a deep learning structure composed of multiple layers of RBM (Restricted Boltzman Machine), a deep learning technique. By repeating RBM (Restricted Boltzman Machine) learning, when a certain number of layers is obtained, a DBN (Deep Belief Network) having a corresponding number of layers can be configured.

A Convolutional Neural Network (CNN) is a structure widely used, especially in the field of object recognition, and will be described with reference to FIGS. 11 and 12 .

CNN (Convolutional Neural Network) is a model that simulates human brain functions based on the assumption that when a person recognizes an object, the basic features of the object are extracted, and then the object is recognized based on the result of complex calculation in the brain. am.

11 is a diagram illustrating a structure of a Convolutional Neural Network (CNN).

A convolutional neural network (CNN) may also include an input layer, a hidden layer, and an output layer.

A predetermined image 1100 is input to the input layer.

Referring to FIG. 11 , a hidden layer is composed of a plurality of layers and may include a convolution layer and a sub-sampling layer.

In CNN (Convolutional Neural Network), various filters for extracting features of an image through convolution operation are basically combined with pooling or non-linear activation functions to add nonlinear characteristics. used

Convolution is mainly used for filter calculation in the field of image processing and is used to implement a filter for extracting features from an image.

For example, as shown in FIG. 12, if the convolution operation is repeatedly performed on the entire image while moving a 3X3 window, an appropriate result can be obtained according to the weight value of the window.

Referring to (a) of FIG. 12 , when a convolution operation is performed on a predetermined area 1210 of the entire image using a 3X3 window, a result value 1201 is obtained.

Referring to (b) of FIG. 12, if a result is obtained again for the area 1220 in which the 3X3 window is moved to the right by 1, a predetermined result value 1202 is obtained.

That is, as shown in (c) of FIG. 12, if a calculation is performed on the entire image while moving a predetermined window, a final result can be obtained.

A convolution layer may be used to perform convolution filtering for filtering information extracted from a previous layer using a filter having a predetermined size (eg, a 3X3 window illustrated in FIG. 12 ).

A convolution layer performs a convolution operation on the input image data (1100, 1102) using a convolution filter, and generates feature maps (1101, 1103) expressing the characteristics of the input image (1100). do.

As a result of convolution filtering, as many filtered images as the number of filters may be generated according to the number of filters included in the convolution layer. A convolution layer may be composed of nodes included in filtered images.

In addition, a sub-sampling layer paired with a convolution layer may include the same number of feature maps as the paired convolution layer.

A sub-sampling layer reduces the dimensions of the feature maps 1101 and 1103 through sampling or pooling.

An output layer recognizes the input image 1100 by combining various features expressed in the feature map 1104 .

The obstacle recognition module of the mobile robot according to the present invention may use the various deep learning structures described above. For example, although the present invention is not limited, a Convolutional Neural Network (CNN) structure widely used in object recognition in an image may be used.

On the other hand, learning of the artificial neural network can be performed by adjusting the weight of the connection line between nodes so that a desired output is produced for a given input. In addition, the artificial neural network may continuously update a weight value by learning. In addition, a method such as back propagation may be used to learn the artificial neural network.

13 and 14 are flowcharts illustrating a control method of a mobile robot according to an embodiment of the present invention.

Referring to FIGS. 2 to 7 and 13 , first, the mobile robots 100, 100a, and 100b may perform cleaning while moving according to commands or settings (S1310).

The sensor unit 170 includes an obstacle detection sensor 131 and can detect an obstacle in front.

When an obstacle is detected through the sensor unit 170 during movement (S1320), the image acquisition unit 120 may obtain an image of the front of the main body 110 (S1330).

In this case, the controller 140 may control to cut and extract a partial region of the image obtained by the image acquisition unit 120 in response to the direction of the obstacle detected by the sensor unit 170 (S1340).

Depending on the embodiment, the image processing module 145 in the control unit 140 may extract a partial region of the image. Alternatively, the control unit 140 may control the separately provided image processing unit 125 to extract a partial area. Alternatively, the control unit 140 may control the image acquisition unit 120 to extract a partial area.

Meanwhile, the obstacle recognition module 144 may recognize an obstacle based on data pre-learned through machine learning in the extracted image (S1350).

The obstacle recognition module 144 may include an artificial neural network trained to recognize attributes such as types of obstacles through machine learning, and may recognize attributes of obstacles based on pre-learned data (S1350).

For example, the obstacle recognition module 144 is equipped with a convolutional neural network (CNN), which is one of the deep learning structures, and the pre-learned convolutional neural network (CNN) recognizes the attributes of obstacles included in the input data, and as a result can output

Due to its characteristics, the obstacle recognition module 144 learned through machine learning has a higher recognition rate as the learning target occupies a larger portion of the input image data.

Therefore, according to the present invention, the recognition rate can be increased by extracting different regions from images acquired by the image acquisition unit 120 according to the direction of the obstacle detected by the sensor unit 170 and using them as data for recognition.

A configuration of cropping and extracting a portion of an image obtained by the image acquisition unit 120 will be described later in detail with reference to FIGS. 15 to 21 and the like.

Meanwhile, the driving control module 141 may control the driving of the driving unit 160 based on the recognized attribute of the obstacle (S1360).

For example, when the recognized obstacle is an obstacle of a height that cannot be overcome, the driving control module 141 may control the vehicle to drive while bypassing the obstacle.

In addition, the driving control module 141 may control the vehicle to continuously drive straight when the recognized obstacle is an obstacle of a height that can be crossed, such as a low-height barrier.

In addition, the driving control module 141 may control the vehicle to drive while bypassing the obstacle when an obstacle of low height, such as a pedestal of a fan, human hair, a power strip, or an electric wire, is recognized.

Referring to FIGS. 2 to 7 and 14 , first, the mobile robots 100, 100a, and 100b may move according to commands or settings (S1410).

When the sensor unit 170 includes an ultrasonic sensor, an obstacle may be recognized by detecting a reflected ultrasonic signal (S1420).

On the other hand, the image acquisition unit 120 may acquire images by continuously photographing the front and surroundings of the mobile robots 100, 100a, and 100b, or by photographing the front and surroundings according to the detection of obstacles by the sensor unit 170. .

The control unit 140 may control to extract an area corresponding to the area where the ultrasound signal is detected from among the images acquired through the image acquisition unit 120 (S1430). For example, when the area where the ultrasound signal is detected is the front right area, the controller 140 may control to extract the right area from among the images acquired through the image acquisition unit 120. In addition, the controller 140 may control to extract a central region from among images obtained through the image acquisition unit 120 when the region where the ultrasonic signal is detected is the front center region.

Meanwhile, the controller 140 may determine the attribute of the obstacle detected in the extracted image based on data pre-learned through machine learning.

In addition, the controller 140 may determine whether the detected obstacle is at a height that can be crossed (S1440).

If the recognized obstacle is an obstacle of an insurmountable height (S1440), the controller 140 may rotate 90 degrees and then control the vehicle to drive around the obstacle (S1455).

On the other hand, if the recognized obstacle is an obstacle of a height that can be overcome (S1440), the fisherman 140 may determine attribute information of the detected obstacle (S1445). That is, the controller 140 may determine whether the recognized obstacle is an obstacle that may proceed because the possibility of restraint is low.

If it is determined that the obstacle is acceptable to proceed, the controller 140 may control the object to continuously move in a straight line (S1450).

Conventionally, it is determined whether or not the detected obstacle is a height that can be crossed, and in the case of a low-height obstacle, straight driving is performed.

However, in the case of obstacles such as electric wires, there have been cases where the mobile robot is wrapped around the electric wires and is constrained.

In addition, when the mobile robot is restrained, it tries to get out of the restraint state by applying a left/right shaking motion, etc., but a safety accident such as peeling off the coating of the wire may occur.

However, the present invention can improve reliability by recognizing obstacle attribute information using machine learning and image information and determining a driving pattern according to the recognized obstacle attribute.

15 to 21 are diagrams referred to in the description of the method for controlling a mobile robot according to an embodiment of the present invention, and show specific examples of cropping and extracting a part of an image obtained by the image acquisition unit 120. it is depicted

Referring to FIG. 15 , the mobile robot 100 may detect an obstacle 1500 such as a fan while moving through a sensor unit 170 . 15 illustrates a case where an obstacle 1500 is detected in the center of the movement direction of the mobile robot 100.

When the mobile robot 100 detects the presence of an obstacle 1500 while moving, as shown in FIG. An image 1600 including a part thereof may be obtained.

The controller 140 may control to extract a partial area 1610 from the image 1600 obtained by the image acquisition unit 120 in response to the direction of the obstacle 1500 detected by the sensor unit 170.

Referring to FIGS. 15 and 16 , the controller 140 controls the lower center area of the image obtained by the image acquisition unit 120 when the obstacle 1500 is detected from the front direction of the main body 110. (1610) can be controlled to extract.

Meanwhile, the controller 140 may use the image 1620 extracted as shown in (b) of FIG. 16 as input data for recognizing the property of the obstacle.

The obstacle recognition module 144 may recognize that the obstacle 1500 is an electric fan based on data pre-learned through machine learning. For example, the obstacle recognition module 144 is equipped with a convolutional neural network (CNN), which is one of the deep learning structures, and the pre-learned convolutional neural network (CNN) recognizes the attributes of obstacles included in the input data, and as a result can output

Meanwhile, the driving control module 141 may control the driving unit 160 to perform an avoidance operation, such as moving after turning, when the detected obstacle is not an obstacle that may be passed.

Meanwhile, the extracted image 1620 may be stored in the storage unit 150 . In addition, the original image 1600 acquired by the image acquisition unit 120 may also be stored in the storage unit 150 .

The extracted image 1620 stored in the storage unit 150 may be used as training data.

Meanwhile, when an obstacle is detected from the right side of the front of the main body 110, the control unit 140 extracts the lower right area of the image acquired by the image acquisition unit 120, and the obstacle is located in the front side of the main body 110. When it is sensed in the left direction of , it can be controlled to extract the lower left area of the image acquired by the image acquisition unit 120.,

17 illustrates a case in which an obstacle 1500 is detected on the right side of the movement direction of the mobile robot 100.

Referring to (a) of FIG. 18 , when an obstacle 1500 is detected from the right side of the front of the main body 110, the controller 140 controls the right side of the image 1800 acquired by the image acquisition unit 120. It is possible to control to extract the lower area 1810.

The controller 140 may use the image 1820 extracted as shown in (b) of FIG. 18 as input data for recognizing the property of the obstacle.

In the present invention, the center, left, and right regions of the image may be extracted based on the direction in which an obstacle is detected, rather than simply cropping the central region of the entire image to a preset size.

Accordingly, as many obstacles as possible can be included in the input data for recognition. Since the machine recognizes the highest weight in the image, the attribute recognition rate of the obstacle can be improved.

Meanwhile, since the mobile robot 100 travels on the floor of a predetermined space, it is possible to include more obstacles in the input data for recognition by extracting the lower part of the acquired image.

Meanwhile, referring to FIG. 19 (a), FIG. 20 (a), and FIG. 21 (a), the sensor unit 170 according to an embodiment of the present invention is located on the front of the main body of the mobile robot 100. It may include a first sensor (S1) disposed, and a second sensor (S2) and a third sensor (S3) disposed to be spaced apart from the first sensor (S1) to the left and right.

In this case, the first sensor S1 may operate as a transmitting unit, and the second sensor S2 and the third sensor S3 may operate as receiving units. For example, the first sensor S1 may emit an ultrasonic signal, and the second sensor S2 and the third sensor S3 may receive signals reflected from an obstacle. When a signal reflected by an obstacle is received, the direction of the obstacle and the distance to the obstacle may be determined using known ultrasonic recognition methods.

19(a) illustrates a case in which the obstacle X1 is detected in the center of the front direction of the mobile robot 100. The distance (l1) between the detected obstacle (X1) and the second sensor (S2) and the distance (l2) between the detected obstacle (X1) and the third sensor (S3) are the same (l1 = l2) If so, it can be determined that the obstacle X1 is detected in the center of the front direction of the mobile robot 100.

In this case, as shown in (b) of FIG. 19, a predetermined area 1910 having a size of a2Xb2 is located at the lower center of the entire original image 1900 acquired by the image acquisition unit 120 having a size of a1Xb1. can be extracted.

Meanwhile, the controller 140 determines the region to be extracted from the image acquired by the image acquisition unit 120 as the distance l1 between the detected obstacle X1 and the second sensor S2 and the detected It can be controlled to extract by shifting in proportion to the difference in the distance l2 between the obstacle X1 and the third sensor S3.

(a) of FIG. 20 illustrates a case in which an obstacle X1 is detected from the front right side of the mobile robot 100. When the distance l1 between the detected obstacle X1 and the second sensor S2 is greater than the distance l2 between the detected obstacle X1 and the third sensor S3 (l1 > l2) , it may be determined that the obstacle X1 is detected in the front right direction of the mobile robot 100.

In this case, as shown in (b) of FIG. 20, a predetermined area 1920 having a size of a2Xb2 may be extracted from the lower right corner of the entire original image 1900 acquired by the image acquisition unit 120 having a size of a1Xb1. there is.

Comparing (b) of FIG. 19 with (b) of FIG. 20 , the center point CP2 and the starting point SP2 of the extraction target area 1920 when the obstacle X1 is detected from the right side are the obstacle X1 It can be seen that the region to be extracted 1910 is shifted to the right by a predetermined value d1 from the central point CP1 and the starting point SP1 of the region to be extracted 1910 when detected at the center.

In this case, the shifted predetermined value d1 is the distance l1 between the detected obstacle X1 and the second sensor S2 and the detected obstacle X1 and the third sensor ( S3) may be proportional to the difference (l1-l2) of the distance l2.

21(a) illustrates a case in which an obstacle X1 is sensed from the front left side of the mobile robot 100. If the distance l1 between the detected obstacle X1 and the second sensor S2 is smaller than the distance l2 between the detected obstacle X1 and the third sensor S3 (l1 < l2 ), it can be determined that the obstacle X1 is detected from the front left side of the mobile robot 100.

In this case, as shown in (b) of FIG. 21, a predetermined area 1930 having a size of a2Xb2 may be extracted from the lower left of the entire original image 1900 acquired by the image acquisition unit 120 having a size of a1Xb1. there is.

Comparing (b) of FIG. 19 with (b) of FIG. 21 , the center point CP3 and the starting point SP3 of the extraction target region 1930 when the obstacle X1 is detected from the left side are the obstacle X1 It can be confirmed that the region to be extracted 1910 is shifted to the left by a predetermined value d2 from the central point CP1 and the starting point SP1 of the region to be extracted 1910 when detected in the center.

In this case, the shifted predetermined value d2 is the distance l1 between the detected obstacle X1 and the second sensor S2 and the detected obstacle X1 and the third sensor ( S3) may be proportional to the difference (l2-l1) of the distance (l2) between them.

Meanwhile, the mobile robot according to an embodiment of the present invention performs a learning process using the extracted image as training data, thereby continuously updating an artificial neural network (ANN) and a deep neural network (DNN) structure. can

Alternatively, the extracted image may be transmitted to a predetermined server, and data related to machine learning may be received from the predetermined server. Thereafter, the mobile robot may update the obstacle recognition module based on data related to machine learning received from the predetermined server.

22 is a diagram referenced in the description of a method of operating a mobile robot and a server according to an embodiment of the present invention, and FIG. 23 is a flowchart illustrating a method of operating a mobile robot and a server according to an embodiment of the present invention. .

Referring to FIGS. 22 and 23 , a deep neural network (DNN) structure 144a such as a Convolutional Neural Network (CNN) may be installed in the obstacle recognition module 144 of the mobile robot 100.

The pre-learned deep neural network (DNN) structure 144a may receive input data for recognition (S2310), recognize attributes of an obstacle included in the input data (S2320), and output the result (S2330).

Unknown data that is not recognized by the deep neural network (DNN) structure 144a may be stored in the storage unit 150 or its own storage space 144b in the obstacle recognition module 144 (S2340).

Meanwhile, data that the obstacle recognition module 144 cannot recognize (unknown data) may be transmitted to the server 70 through the communication unit 190 (S2341). In addition, data successfully recognized by the obstacle recognition module 144 may also be transmitted to the server 70 .

The server 70 may create a configuration of learned weights, and the server 70 may learn a deep neural network (DNN) structure using training data.

The server 70 may learn the deep neural network (DNN) based on the received data (S2342), and then transmit the updated deep neural network (DNN) structure data to the mobile robot 100 to update.

24 to 26 are views referenced for description of a control method of a mobile robot according to an embodiment of the present invention.

The area for detecting the obstacle of the mobile robot according to the embodiment of the present invention can be identified by dividing it into three areas of the center, left, and right areas, and the areas can be divided into more than that number.

24 illustrates an example of detecting an obstacle by dividing it into six areas.

Referring to (a) of FIG. 24 , the obstacle sensing area 2400 is an upper center area (CU), a lower center area (CD), an upper right area (RU), a lower right area (RD), and an upper left area (LU). ), an obstacle can be detected by dividing it into six areas of the lower left area LD.

In addition, the mobile robot according to an embodiment of the present invention may extract a partial region corresponding to the region where the obstacle is detected from among the entire image acquired by the image acquisition unit 120 .

Referring to (b) of FIG. 24 , an obstacle 2410 may be detected over the central upper area CU and the right upper area RU. In this case, of the entire image obtained by the image acquisition unit 120, the upper center region and the upper right region may be cropped.

Referring to (c) of FIG. 24 , an obstacle 2420 may be detected across the lower center area CD, the lower right area RD, and the lower left area LD. In this case, the lower regions (CD, RD, LD) of the entire image obtained by the image acquisition unit 120 may be cropped.

Alternatively, instead of cropping for each area allocated to the obstacle detection area, the area may be further subdivided and cropped.

For example, referring to (c) of FIG. 24 , when the obstacle 2420 is detected across the lower center area CD, the lower right area RD, and the lower left area LD, the lower center area ( The lower region 2325 in the CD, lower right region RD, and lower left region LD may be cropped.

25 and 26 are diagrams referenced for description of the obstacle recognition of the obstacle recognition module 144.

Referring to FIG. 25 , the obstacle recognition module 144 classifies and classifies obstacles into classes such as electric fans, home theaters, power strips, lamp supports, human hair, and bumps, and can recognize them.

In addition, the obstacle recognition module 144 classifies and distinguishes classes such as electric fans, home theaters, power strips, lamp supports, and human hair into dangerous obstacle super-classes as a higher level concept and can recognize them.

In addition, the obstacle recognition module 144 classifies obstacles that can be driven straight, such as obstacles, into non-hazardous obstacle super-class, and classifies and recognizes them.

Referring to (a) of FIG. 26, the obstacle recognition module 144 recognizes an input image and obtains a recognition result having a confidence value of 0.95 for electric fans and a confidence value of 0.7 for home theaters. . In this case, the obstacle recognition module 144 may output an electric fan, which is a recognition result having a higher reliability value, as a recognition result of the input image.

Meanwhile, the confidence value may be normalized in the range of 0.0 to 1.0.

Referring to (b) of FIG. 26 , the obstacle recognition module 144 recognizes an input image and obtains a recognition result having a reliability value of 0.35 for electric fans and a reliability value of 0.4 for home theaters.

For example, when it is set not to recognize a confidence value of 0.6 or less, the obstacle recognition module 144 does not select a specific recognition result because both of the confidence values of the two recognition results are lower than the reference value, and may determine it as unknown data. there is.

Referring to (c) of FIG. 26 , the obstacle recognition module 144 recognizes an input image and obtains a recognition result having a reliability value of 0.95 for a fan and a reliability value of 0.9 for a home theater.

For example, when it is set to select a recognition result having a confidence value of 0.9 or more as the final recognition result, the obstacle recognition module 144 does not select a specific recognition result because both the confidence values of the two recognition results are higher than the reference value. , it can be judged as a high-level concept, a dangerous obstacle.

Alternatively, even when the recognition result with the highest confidence value is recognized when the difference between the confidence values is 0.15 or more, it may be determined as a high-order concept, a dangerous obstacle.

Meanwhile, even when it is determined as a dangerous obstacle, the driving control module 141 may control the driving unit 160 to move while avoiding the dangerous obstacle.

According to at least one of the embodiments of the present invention, the mobile robot can determine the property of the obstacle and adjust the driving pattern according to the property of the obstacle, thereby performing highly reliable obstacle recognition and avoidance operations.

In addition, according to at least one of the embodiments of the present invention, the stability of the mobile robot itself and the user's convenience are improved by performing operations such as forward, backward, stop, and detour according to the recognition result of the obstacle, and driving efficiency and cleaning efficiency It is possible to provide a mobile robot and its control method that can improve the.

In addition, according to at least one of the embodiments of the present invention, it is possible to provide a mobile robot capable of accurately recognizing the property of an obstacle based on machine learning and a control method thereof.

In addition, according to at least one of the embodiments of the present invention, the mobile robot can efficiently perform machine learning and extract data usable for obstacle attribute recognition.

The mobile robot according to the present invention is not limited to the configuration and method of the embodiments described above, but all or part of each embodiment is selectively combined so that various modifications can be made. may be configured.

Meanwhile, the method for controlling a mobile robot according to an embodiment of the present invention can be implemented as a processor-readable code on a processor-readable recording medium. The processor-readable recording medium includes all types of recording devices in which data readable by the processor is stored. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include those implemented in the form of carrier waves such as transmission through the Internet. . In addition, the processor-readable recording medium is distributed in computer systems connected through a network, so that the processor-readable code can be stored and executed in a distributed manner.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Body: 110
Image acquisition department: 120
Front camera: 120A
Upper camera: 120b
Controls: 137
Control: 140
Reservoir: 150
Driving part: 160
Sensor part: 170
Department of Communications: 190

Claims (17)

a driving unit for moving the main body;
an image acquisition unit acquiring an image around the main body;
a storage unit for storing the image acquired by the image acquisition unit;
a sensor unit including one or more sensors for detecting an obstacle during the movement; and
Including; a control unit for controlling to extract a partial area of the image acquired by the image acquisition unit in response to the direction of the obstacle detected by the sensor unit;
The sensor unit,
a first sensor disposed on the front surface of the main body;
A second sensor and a third sensor disposed to be spaced apart from the first sensor to the left and right;
The control unit determines an area to be extracted from an image obtained by the image acquisition unit in proportion to a difference between a distance between the detected obstacle and the second sensor and a distance between the detected obstacle and the third sensor. A mobile robot controlling to extract the extraction target region by shifting a part from the central region. According to claim 1,
The control unit,
An obstacle recognition module for recognizing an obstacle based on data pre-learned by machine learning from the extracted image; and
and a driving control module for controlling driving of the driving unit based on the recognized attribute of the obstacle. According to claim 2,
The control unit,
The mobile robot, characterized in that it further comprises an image processing module for extracting a partial region of the image acquired by the image acquisition unit in response to the direction of the obstacle detected by the sensor unit. According to claim 2,
The obstacle recognition module,
Each time a partial region of an image acquired by the image acquisition unit is extracted, a learning process is performed using the extracted image as training data, or a learning process is performed after a predetermined number or more of the extracted images are acquired. A mobile robot that does. According to claim 2,
The mobile robot further comprising a communication unit for transmitting the extracted image to a predetermined server and receiving data related to machine learning from the predetermined server. According to claim 5,
The mobile robot characterized in that the obstacle recognition module is updated based on data related to machine learning received from the predetermined server. According to claim 1,
The mobile robot further comprising: an image processing unit that extracts a partial area of the image acquired by the image acquisition unit in response to the direction of the obstacle detected by the sensor unit. According to claim 1,
The control unit,
When the obstacle is detected in the right direction of the front of the main body, extracting the lower right area of the image acquired by the image acquisition unit,
When the obstacle is detected in the left direction of the front of the main body, extracting the lower left area of the image acquired by the image acquisition unit;
The mobile robot characterized in that when the obstacle is detected in the front direction of the main body, control is performed to extract a lower center area of the image acquired by the image acquisition unit. delete delete Sensing an obstacle during movement through a sensor unit including a first sensor disposed on the front surface of the main body and a second sensor and a third sensor disposed to be left and right spaced apart from the first sensor;
extracting a partial region of an image acquired by an image acquisition unit in response to the direction of the obstacle detected by the sensor unit;
Recognizing an obstacle based on data pre-learned through machine learning in the extracted image; and,
Based on the attribute of the recognized obstacle, controlling the driving of the driving unit; includes,
In the step of extracting a partial region of the image, the image obtained by the image acquisition unit is proportional to a difference between a distance between the detected obstacle and the second sensor and a distance between the detected obstacle and the third sensor. A method of controlling a mobile robot for extracting a partial area by shifting a portion from a central area. According to claim 11,
A method for controlling a mobile robot further comprising performing a learning process using the extracted image as training data. According to claim 11,
transmitting the extracted video to a predetermined server;
The method of controlling a mobile robot further comprising receiving data related to machine learning from the predetermined server. According to claim 13,
The method of controlling a mobile robot further comprising updating the pre-learned data based on data related to machine learning received from the predetermined server. According to claim 11,
The step of extracting a partial region from the image acquired by the image acquisition unit,
When the obstacle is detected in the right direction of the front of the main body, extracting the lower right area of the image acquired by the image acquisition unit,
When the obstacle is detected in the left direction of the front of the main body, extracting the lower left area of the image acquired by the image acquisition unit;
The control method of the mobile robot, characterized in that, when the obstacle is detected in the front direction of the main body, a lower center area of the image acquired by the image acquisition unit is extracted. According to claim 11,
The method of controlling a mobile robot further comprising storing the extracted image. According to claim 11,
In the driving control step,
A method of controlling a mobile robot, characterized in that controlling to perform an avoidance operation when the detected obstacle is not an obstacle that can be advanced.

Images