KR102290983B1 - Controlling method for Artificial intelligence Moving robot - Google Patents

Abstract

The artificial intelligence mobile robot according to an aspect of the present invention can utter a voice to a user, so that it can talk and interact with the user through voice, and actively provide information before making a request, service, function, etc. By recommending , it is possible to increase the user's reliability, preference, and product utilization.

Description

Control method of artificial intelligence moving robot {Controlling method for Artificial intelligence Moving robot}

The present invention relates to a mobile robot and a method for controlling the same, and more particularly, to a mobile robot capable of actively providing information and services to a user and a method for controlling the same.

Robots have been developed for industrial use and have been a part of factory automation. Recently, the field of application of robots has been further expanded, and medical robots and aerospace robots have been developed, and household robots that can be used in general households are also being made. Among these robots, those capable of driving by their own force are called mobile robots.

A representative example of a mobile robot used at home is a robot cleaner, which is a device that cleans a corresponding area by sucking dust or foreign substances around it while driving in a certain area by itself.

The mobile robot is free to move because it can move by itself, and is provided with a plurality of sensors for avoiding obstacles while driving, so that it can run while avoiding obstacles.

Meanwhile, in order to improve user convenience, voice recognition technology is being applied to various devices, and research on a method of controlling a mobile robot using the voice recognition technology is increasing.

For example, in Prior Document 1 (Korean Patent Application Laid-Open No. 10-2012-0114670, publication date October 17, 2012), a robot cleaner has a voice recognition unit and recognizes a user's voice signal to generate a corresponding control command. is running

In Prior Document 1, the voice input is made unidirectionally from the user to the robot cleaner, so it stays as an additional means of control operation by pressing a button or operating with a remote control. Accordingly, there is a problem in that it is difficult for the voice recognition function to give a meaning beyond simple control to the user, and other functions and services other than the addition of a control input means cannot be provided.

It is an object of the present invention to provide a mobile robot capable of interacting with a user through voice and a control method thereof, beyond the limitation of the prior document 1, which only uses voice recognition as a control input means.

An object of the present invention is to provide a mobile robot with various information and services to the user actively.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a mobile robot capable of user-friendly operation and a method for controlling the same to improve product reliability and preference of a user.

In order to achieve the above or other objects, the mobile robot according to an aspect of the present invention may utter a voice for guiding a user with predetermined information and services, and may communicate and interact with the user through the voice. .

In order to achieve the above or other objects, the mobile robot according to an aspect of the present invention actively provides predetermined information before making a request and recommends a predetermined service, function, etc., thereby increasing the user's reliability, preference, and product utilization. can

In order to achieve the above or other object, an operating method of a mobile robot according to an aspect of the present invention includes the steps of performing cleaning while moving, determining a special area based on existing data obtained by performing previous cleaning, And, when arriving at the determined special area, by including the step of outputting a voice guidance message for the special area, before the user's request, it is possible to provide information about the special area that needs to be actively provided by voice. there is.

Here, the special area may be a dangerous area in which the driving disabled state occurred, or a low-efficiency area in which the running speed or the cleaning rate was lower than the average value, or the number of rotations was greater than the reference number.

In addition, the existing data obtained through the previous cleaning performance may include driving history data and cleaning history data.

In addition, the existing data obtained by performing previous cleaning may further include information about obstacles registered in the special area, in this case, the voice guidance message may include a guidance message for obstacles existing in the special area. there is.

Meanwhile, the voice guidance message may include a message requesting confirmation of whether to clean the special area. In this case, the voice guidance message may further include at least one of a description of the reason for determining the special area or a guidance on an example of a command that the user can input by voice.

A method of operating a mobile robot according to an aspect of the present invention includes the steps of receiving a user's voice feedback for the voice guidance message, identifying a voice command included in the received voice feedback, and responding to the identified voice command. The method may further include performing a corresponding operation.

In this case, the identification of the voice command may be performed in the mobile robot itself, in the server, or in steps in the mobile robot and the server.

In order to achieve the above or other object, a method of operating a mobile robot according to an aspect of the present invention includes receiving a user's voice command and recommending at least one of a function used less than a reference number based on a usage record. Outputting a voice guidance message, when a user's voice feedback for the voice guidance message is received, identifying a feedback voice command included in the received voice feedback, and performing a predetermined function based on the identified feedback voice command By including the setting step, useful functions among functions that are not frequently used by the user can be actively recommended.

Here, the usage record is the number of times of general cleaning completed, the number of returning to the charging station, the number of cleaning functions completed by area, operating time, average cleaning intensity by area, amount of dust by area, frequency of use of cleaning mode by area, ratio of cleaning mode used by area, cleaning It may include at least one of the number of uses for each function and additional function.

Also, the mobile robot may select and recommend at least one of the functions used less than a reference number of times according to a preset priority. In this case, the preset priority may be an order in which importance for each preset function is high or an order in which the number of times of use is small.

The method of operating a mobile robot according to an aspect of the present invention may further include identifying a user's voice command. In this case, an operation corresponding to the identified user's voice command may be performed, and a function related to the identified user's voice command may be selected and recommended among functions used less than a reference number of times.

Meanwhile, the identification of the voice command and the feedback voice command may be performed by the mobile robot itself, by the server, or by the mobile robot and the server in stages.

According to at least one of the embodiments of the present invention, the mobile robot may utter a voice to the user, and may talk and interact with the user through the voice.

In addition, according to at least one of the embodiments of the present invention, the mobile robot actively provides information before making a request and recommends services, functions, etc., thereby increasing user's reliability, preference, and product utilization.

In addition, according to at least one of the embodiments of the present invention, by actively guiding the mobile robot to a dangerous dangerous area and a low-efficiency area that is not easy to clean, the stability and user convenience of the mobile robot are improved, and operation efficiency and cleaning efficiency are improved. etc. can be improved.

In addition, according to at least one of the embodiments of the present invention, the user's reliability, preference, and product utilization may be increased by actively recommending useful functions among functions not frequently used by the user.

In addition, according to at least one of the embodiments of the present invention, by recommending another function related to the user's voice command, the user can easily set and use the related function without additional effort.

In addition, according to at least one of the embodiments of the present invention, effective voice recognition may be performed by performing voice recognition in the mobile robot itself, in the server, or in stages in the mobile robot and the server.

In addition, according to at least one of the embodiments of the present invention, it is possible to provide an evolving user experience (ux) by utilizing artificial intelligence and machine learning for voice recognition, obstacle recognition, product data analysis, and the like.

On the other hand, various other effects will be directly or implicitly disclosed in the detailed description according to the embodiment of the present invention to be described later.

1 is a configuration diagram of a home appliance network system according to an embodiment of the present invention.
2 is a perspective view illustrating a mobile robot according to an embodiment of the present invention.
3 is a plan view of the mobile robot of FIG. 2 .
4 is a side view of the mobile robot of FIG. 2 .
5 is a block diagram illustrating a control relationship between main components of a mobile robot according to an embodiment of the present invention.
6 is a diagram referenced in the description of learning using product data according to an embodiment of the present invention.
7 is an example of a simplified internal block diagram of a server according to an embodiment of the present invention.
8 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.
9 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.
10 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.
11 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.
12 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.
13 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.
14 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it goes without saying that the present invention is not limited to these embodiments and may be modified in various forms.

In the drawings, in order to clearly and briefly describe the present invention, the illustration of parts irrelevant to the description is omitted, and the same reference numerals are used for the same or extremely similar parts throughout the specification.

On the other hand, the suffixes "module" and "part" for the components used in the following description are given only considering the ease of writing the present specification, and do not give a particularly important meaning or role by themselves. Accordingly, the terms “module” and “unit” may be used interchangeably.

The mobile robot 100 according to an embodiment of the present invention means a robot capable of moving by itself using wheels, etc., and may be a home helper robot, a robot cleaner, or the like. Hereinafter, a robot cleaner having a cleaning function among mobile robots will be described as an example with reference to the drawings, but the present invention is not limited thereto.

1 is a configuration diagram of a home appliance network system according to an embodiment of the present invention.

Referring to FIG. 1 , a home appliance network system may include home appliances that are provided with a communication module to communicate with other devices and servers 70 or access a network.

For example, the home appliance may include an air conditioner 10 having a communication module, a cleaner 20 , a refrigerator 31 , a washing machine 32 , and the like.

Meanwhile, the air conditioner 10 may include at least one of an air conditioner 11 , air cleaners 12 and 13 , a humidifier 14 , and a hood 15 .

Also, the cleaner 20 may be a vacuum cleaner 21 , a robot cleaner 22 , or the like.

On the other hand, the communication module provided in the home appliance (10, 20, 31, 32) may be a Wi-Fi (wi-fi) communication module, the present invention is not limited to the communication method.

Alternatively, the home appliances 10 , 20 , 31 , and 32 may include different types of communication modules or a plurality of communication modules. For example, the home appliances 10 , 20 , 31 , and 32 may include an NFC module, a Zigbee communication module, a Bluetooth™ communication module, and the like.

The home appliances 10 , 20 , 31 , and 32 may be connected to a predetermined server 70 through a Wi-Fi communication module or the like, and may support smart functions such as remote monitoring and remote control.

The home appliance network system according to an embodiment of the present invention may include a mobile terminal 50 such as a smart phone and a tablet PC.

The user may check information on the home appliances 10 , 20 , 31 , 32 in the home appliance network system or control the home appliances 10 , 20 , 31 , 32 through the portable terminal 50 .

Meanwhile, the home appliance network system according to an embodiment of the present invention may be configured to include a plurality of Internet of Things (IoT) devices (not shown). Accordingly, the home appliance network system may include the home appliances 10 , 20 , 31 , 32 , the portable terminal 50 , and Internet of Things (IoT) devices.

The home appliance network system according to an embodiment of the present invention is not limited to a communication method constituting a network.

For example, the home appliances 10 , 20 , 31 , and 32 , the portable terminal 50 , and Internet of Things (IoT) devices may be communicatively connected through a wired/wireless router (not shown).

In addition, the devices in the home appliance network system may configure a mesh topology in which each of the devices is individually communicatively connected.

The home appliances 10 , 20 , 31 , and 32 in the home appliance network system may communicate with the server 70 or the mobile terminal 50 via a wired/wireless router (not shown). In addition, the home appliances 10 , 20 , 31 , and 32 in the home appliance network system may communicate with the server 70 or the mobile terminal 50 through Ethernet.

FIG. 2 is a perspective view illustrating a mobile robot according to an embodiment of the present invention, FIG. 3 is a plan view of the mobile robot of FIG. 2 , and FIG. 4 is a side view of the mobile robot of FIG. 2 .

2 to 4 , the mobile robot 100 may travel by itself in a predetermined area. The mobile robot 100 may perform a function of cleaning the floor. The cleaning of the floor referred to herein includes sucking in dust (including foreign matter) on the floor or mopping the floor.

The mobile robot 100 includes a body 110 . The body 110 includes a cabinet forming an exterior. The mobile robot 100 may include a suction unit 130 and a dust container 140 provided in the main body 110 . The mobile robot 100 includes an image acquisition unit 120 that detects information related to the environment around the mobile robot. The mobile robot 100 includes a traveling unit 160 that moves the main body. The mobile robot 100 includes a controller 150 for controlling the mobile robot 100 . The control unit 150 is provided in the main body 110 .

The driving unit 160 includes a wheel unit 111 for driving the mobile robot 100 . The wheel unit 111 is provided in the body 110 . By the wheel unit 111, the mobile robot 100 may be moved or rotated forward, backward, left and right. As the controller controls the driving of the wheel unit 111 , the mobile robot 100 may autonomously drive on the floor. The wheel unit 111 includes a main wheel 111a and a sub wheel 111b.

The main wheel 111a is provided on both sides of the main body 110, and is configured to be rotatable in one direction or the other according to a control signal of the controller. Each of the main wheels 111a may be configured to be driven independently of each other. For example, each of the main wheels 111a may be driven by different motors.

The sub wheel 111b supports the main body 110 together with the main wheel 111a, and is configured to assist the movement of the mobile robot 100 by the main wheel 111a. Such a sub wheel 111b may also be provided in the suction unit 130 to be described later.

The suction unit 130 may be disposed to protrude from the front F of the body 110 . The suction unit 130 is provided to suck air containing dust.

The suction unit 130 may have a shape protruding from the front of the main body 110 to both left and right sides. The front end of the suction unit 130 may be disposed at a position spaced forward from one side of the body 110 . Both left and right ends of the suction unit 130 may be disposed at positions spaced apart from the main body 110 in both left and right sides, respectively.

The body 110 is formed in a circular shape, and as both sides of the rear end of the suction unit 130 protrude from the body 110 to the left and right sides, respectively, there is an empty space between the body 110 and the suction unit 130 , that is, A gap may be formed. The empty space is a space between the left and right ends of the main body 110 and the left and right ends of the suction unit 130 , and has a shape recessed into the mobile robot 100 .

The suction unit 130 may be detachably coupled to the body 110 . When the suction unit 130 is separated from the main body 110 , a mop module (not shown) may be detachably coupled to the main body 110 to replace the separated suction unit 130 .

The image acquisition unit 120 may be disposed on the main body 110 . The image acquisition unit 120 may be disposed in the front (F) of the main body (110). The image acquisition unit 120 may be disposed to overlap the suction unit 130 in the vertical direction of the main body 110 . The image acquisition unit 120 may be disposed on the suction unit 130 .

The image acquisition unit 120 may detect obstacles around the mobile robot 100 . The image acquisition unit 120 may detect an obstacle or a feature in the front so that the suction unit 130 positioned at the front of the mobile robot 100 does not collide with the obstacle. The image acquisition unit 120 may additionally perform other sensing functions, which will be described later, in addition to these sensing functions.

The main body 110 may be provided with a dust container accommodating part (not shown). A dust container 140 that separates and collects dust in the sucked air is detachably coupled to the dust container accommodating part. The dust container accommodating part may be formed in the rear (R) of the main body 110 . A part of the dust container 140 may be accommodated in the dust container accommodating part, and the other part of the dust container 140 may be formed to protrude toward the rear R of the main body 110 .

In the dust container 140 , an inlet (not shown) through which air containing dust is introduced and an outlet (not shown) through which dust-separated air is discharged are formed. When the dust bin 140 is mounted on the dust bin accommodating part, the inlet and the outlet of the dust bin 140 are configured to communicate with a first opening (not shown) and a second opening (not shown) formed on the inner wall of the dust bin accommodating part, respectively. .

A suction flow path (not shown) for guiding air from the suction port of the suction unit 130 to the first opening is provided. An exhaust flow path (not shown) for guiding air from the second opening to an exhaust port (not shown) opened toward the outside is provided.

The air containing dust introduced through the suction unit 130 passes through the intake passage inside the body 110 and flows into the dust container 140 , and the air and dust pass through the filter or cyclone of the dust container 140 . separated from each other Dust is collected in the dust container 140 , and after the air is discharged from the dust container 140 , it passes through the exhaust passage inside the main body 110 and is finally discharged to the outside through the exhaust port.

5 is a block diagram illustrating a control relationship between main components of a mobile robot according to an embodiment of the present invention.

2 to 5 , the mobile robot 100 includes a main body 110 and an image acquisition unit 120 that acquires images around the main body 110 .

The mobile robot 100 includes a traveling unit 160 that moves the main body 110 . The driving unit 160 includes at least one wheel unit 111 for moving the body 110 . The driving unit 160 includes a driving motor (not shown) connected to the wheel unit 111 to rotate the wheel unit 111 .

The image acquisition unit 120 captures the driving area, and may include a camera module. The camera module may include a digital camera. A digital camera includes at least one optical lens and an image sensor (eg, CMOS image sensor) including a plurality of photodiodes (eg, pixels) that are imaged by light passing through the optical lens; It may include a digital signal processor (DSP) that configures an image based on signals output from the photodiodes. The digital signal processor may generate a still image as well as a moving picture composed of frames composed of still images.

A plurality of such cameras may be installed for each part for photographing efficiency. The image captured by the camera may be used to recognize the type of material such as dust, hair, floor, etc. present in the space, whether to clean it, or to check the cleaning time.

The camera may photograph the situation of an obstacle or a cleaning area existing in the front of the moving direction of the mobile robot 100 .

According to an embodiment of the present invention, the image acquisition unit 120 may acquire a plurality of images by continuously photographing the periphery of the main body 110 , and the acquired plurality of images may be stored in the storage unit 105 . can

The mobile robot 100 increases the accuracy of spatial recognition, position recognition, and obstacle recognition by using a plurality of images, or selects one or more images among a plurality of images and uses effective data to perform spatial recognition, position recognition, and obstacle recognition. accuracy can be increased.

In addition, the mobile robot 100 may include a sensor unit 170 including sensors for sensing various data related to the operation and state of the mobile robot.

For example, the sensor unit 170 may include an obstacle detection sensor that detects an obstacle in front. In addition, the sensor unit 170 may further include a cliff detection sensor for detecting the presence of a cliff on the floor in the driving zone, and a lower camera sensor for acquiring an image of the floor.

The obstacle detection sensor may include an infrared sensor, an ultrasonic sensor, an RF sensor, a geomagnetic sensor, a Position Sensitive Device (PSD) sensor, and the like.

On the other hand, the position and type of the sensor included in the obstacle detection sensor may vary depending on the model of the mobile robot, and the obstacle detection sensor may include more various sensors.

Meanwhile, the sensor unit 170 may further include a motion detection sensor that detects the motion of the mobile robot 100 according to the driving of the main body 110 and outputs motion information. For example, as the motion detection sensor, a gyro sensor, a wheel sensor, an acceleration sensor, or the like may be used.

The gyro sensor detects a rotation direction and a rotation angle when the mobile robot 100 moves according to a driving mode. The gyro sensor detects the angular velocity of the mobile robot 100 and outputs a voltage value proportional to the angular velocity. The controller 150 calculates a rotation direction and a rotation angle using a voltage value output from the gyro sensor.

The wheel sensor is connected to the wheel unit 111 to detect the number of rotations of the wheel. Here, the wheel sensor may be a rotary encoder.

The acceleration sensor detects a change in speed of the mobile robot 100 , for example, a change in the mobile robot 100 according to start, stop, direction change, collision with an object, and the like.

Also, the acceleration sensor may be embedded in the controller 150 to detect a change in speed of the mobile robot 100 .

The controller 150 may calculate a change in the position of the mobile robot 100 based on the motion information output from the motion detection sensor. This position becomes a relative position corresponding to the absolute position using the image information. The mobile robot can improve the performance of position recognition using image information and obstacle information through such relative position recognition.

Meanwhile, the mobile robot 100 may include a power supply unit (not shown) that includes a rechargeable battery and supplies power to the mobile robot.

The power supply unit supplies driving power and operating power to each component of the mobile robot 100, and when the remaining power is insufficient, it may be charged by receiving power from a charging station (not shown).

The mobile robot 100 may further include a battery detection unit (not shown) that detects the state of charge of the battery and transmits the detection result to the control unit 150 . The battery is connected to the battery detection unit, and the remaining battery level and charging state are transmitted to the control unit 150 . The remaining battery level may be displayed on the display 182 of the output unit 180 .

In addition, the mobile robot 100 includes an input unit 125 capable of inputting on/off or various commands. The input unit 125 may include a button, a dial, a touch screen, or the like. The input unit 125 may include a microphone for receiving a user's voice instruction. Various control commands necessary for the overall operation of the mobile robot 100 may be input through the input unit 125 .

In addition, the mobile robot 100 may include the output unit 180 to display reservation information, battery state, operation mode, operation state, error state, etc. as an image or output the sound.

The output unit 180 may include a sound output unit 181 for outputting an audio signal. The sound output unit 181 may output a notification message such as a warning sound, an operation mode, an operation state, an error state, etc. in sound according to the control of the controller 150 . The sound output unit 181 may convert the electrical signal from the control unit 150 into an audio signal and output it. To this end, a speaker or the like may be provided.

In addition, the output unit 180 may further include a display 182 for displaying reservation information, battery state, operation mode, operation state, error state, etc. as an image.

Referring to FIG. 5 , the mobile robot 100 includes a control unit 150 that processes and determines various types of information such as recognizing a current location, and a storage unit 105 that stores various data. In addition, the mobile robot 100 may further include a communication unit 190 for transmitting and receiving data with an external terminal.

The external terminal has an application for controlling the mobile robot 100, displays a map for the driving area to be cleaned by the mobile robot 100 through the execution of the application, and can designate an area to clean a specific area on the map there is. The external terminal may include, for example, a remote controller, a PDA, a laptop, a smart phone, a tablet, etc. equipped with an application for setting a map.

The external terminal may communicate with the mobile robot 100 to display the current location of the mobile robot together with a map, and information on a plurality of areas may be displayed. In addition, the external terminal updates and displays the position according to the movement of the mobile robot.

The control unit 150 controls the image acquisition unit 120, the input unit 125, the traveling unit 160, the suction unit 130, etc. constituting the mobile robot 100 to control the overall operation of the mobile robot 100. Control.

The controller 150 may process a user's voice input signal received through the microphone of the input unit 125 and perform a voice recognition process. According to an embodiment, the mobile robot 100 may include a voice recognition module that performs voice recognition inside or outside the control unit 150 .

According to an embodiment, simple voice recognition may be performed by the mobile robot 100 by itself, and high-level voice recognition such as natural language processing may be performed by the server 70 .

The storage unit 105 records various types of information necessary for controlling the mobile robot 100 , and may include a volatile or nonvolatile recording medium. A recording medium stores data that can be read by a microprocessor, such as HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic may include tapes, floppy disks, optical data storage devices, and the like.

Also, the storage unit 105 may store a map for the driving area. The map may be input by an external terminal or server capable of exchanging information with the mobile robot 100 through wired or wireless communication, or may be generated by the mobile robot 100 learning itself.

The map may display the locations of the rooms in the driving area. In addition, the current position of the mobile robot 100 may be displayed on the map, and the current position of the mobile robot 100 on the map may be updated during the driving process. The external terminal stores the same map as the map stored in the storage unit 105 .

The storage unit 105 may store cleaning history information. Such cleaning history information may be generated whenever cleaning is performed.

The map for the driving area stored in the storage unit 105 includes a navigation map used for driving while cleaning, a Simultaneous localization and mapping (SLAM) map used for location recognition, and an obstacle when the information is stored. Thus, it may be a learning map used for learning and cleaning, a global location map used for global location recognition, an obstacle recognition map in which information about recognized obstacles is recorded, and the like.

Meanwhile, as described above, although maps can be stored and managed in the storage unit 105 by use, the maps may not be clearly classified by use. For example, a plurality of pieces of information may be stored in one map to be used for at least two or more purposes.

The control unit 150 may include a driving control module 151 , a map generation module 152 , a location recognition module 153 , and an obstacle recognition module 154 .

1 to 5 , the driving control module 151 controls the driving of the mobile robot 100 and controls the driving of the driving unit 160 according to the driving setting. Also, the driving control module 151 may determine the driving path of the mobile robot 100 based on the operation of the driving unit 160 . For example, the driving control module 151 may determine the current or past moving speed, the distance traveled, etc. of the mobile robot 100 based on the rotational speed of the wheel unit 111. Based on the driving information of 100 ), the position of the mobile robot 100 on the map may be updated.

The map generation module 152 may generate a map of the driving area. The map generation module 152 may create a map by processing the image acquired through the image acquisition unit 120 . That is, a cleaning map corresponding to the cleaning area may be created.

In addition, the map generation module 152 may process the image acquired through the image acquisition unit 120 at each location and link it with the map to recognize the global location.

The location recognition module 153 estimates and recognizes the current location. The position recognition module 153 uses the image information of the image acquisition unit 120 to determine the position in connection with the map generation module 152, thereby estimating the current position even when the position of the mobile robot 100 is suddenly changed. can be recognized by

In addition, the location recognition module 153 can recognize the property of the area currently located, that is, the location recognition module 153 can recognize the space.

The mobile robot 100 can recognize the position during continuous driving through the position recognition module 153 and, through the map generation module 152 and the obstacle recognition module 154 without the position recognition module 153, create a map. You can learn and estimate your current location, etc.

While the mobile robot 100 is running, the image acquisition unit 120 acquires images around the mobile robot 100 . Hereinafter, an image acquired by the image acquisition unit 120 is defined as an 'acquired image'.

The acquired image includes various features such as lights located on the ceiling, edges, corners, blobs, and ridges.

The map generation module 152 detects a feature from each of the acquired images, and calculates a descriptor based on each feature point.

The map generation module 152 classifies at least one descriptor for each acquired image into a plurality of groups according to a predetermined sub-classification rule based on descriptor information obtained through the acquired image of each location, and classifies the same group according to a predetermined sub-representative rule. Descriptors included in ? can be converted into sub-representative descriptors, respectively.

As another example, all descriptors collected from acquired images within a predetermined area such as a room are classified into a plurality of groups according to a predetermined sub-classification rule, and the descriptors included in the same group according to the predetermined sub-representative rule are respectively classified as sub-representative descriptors. can also be converted to

The map generation module 152 may obtain the feature distribution of each location through this process. Each location feature distribution may be expressed as a histogram or an n-dimensional vector. As another example, the map generation module 152 may estimate an unknown current location based on descriptors calculated from each feature point without going through a predetermined sub-classification rule and a predetermined sub-representative rule.

Also, when the current position of the mobile robot 100 becomes unknown due to a position jump or the like, the current position may be estimated based on data such as a pre-stored descriptor or a lower representative descriptor.

The mobile robot 100 acquires an acquired image through the image acquisition unit 120 at an unknown current position. Various features such as lights located on the ceiling, edges, corners, blobs, and ridges are identified through the image.

The location recognition module 153 detects features from the acquired image and calculates a descriptor.

The location recognition module 153 is configured to compare location information (eg, feature distribution of each location) to be compared according to a predetermined sub-conversion rule based on at least one descriptor information obtained through an image acquired through an unknown current location. It is converted into comparable information (lower recognition feature distribution).

According to a predetermined sub-comparison rule, each positional feature distribution may be compared with each recognized feature distribution to calculate each similarity. A degree of similarity (probability) may be calculated for each location corresponding to each location, and a location at which the greatest probability is calculated may be determined as the current location.

In this way, the controller 150 may divide the driving zone and generate a map composed of a plurality of areas, or recognize the current location of the main body 110 based on a pre-stored map.

When the map is generated, the control unit 150 may transmit the generated map to an external terminal, a server, or the like through the communication unit 190 . Also, as described above, when a map is received from an external terminal, a server, or the like, the controller 150 may store the map in the storage unit 105 .

In this case, the map may divide the cleaning area into a plurality of areas, may include a connection passage connecting the plurality of areas, and may include information on obstacles in the area.

When a cleaning command is input, the control unit 150 determines whether the location on the map matches the current location of the mobile robot. The cleaning command may be input from a remote control, an input unit, or an external terminal.

When the current location does not match the location on the map, or when the current location cannot be confirmed, the controller 150 recognizes the current location and restores the current location of the mobile robot 100, and then based on the current location. The driving unit 160 may be controlled to move to a designated area.

When the current location does not match the location on the map or when the current location cannot be confirmed, the location recognition module 153 analyzes the acquired image input from the image acquisition unit 120 to estimate the current location based on the map can do. In addition, the obstacle recognition module 154 or the map generation module 152 may also recognize the current location in the same way.

After recognizing the position and restoring the current position of the mobile robot 100, the driving control module 151 calculates a driving route from the current position to the designated area and controls the driving unit 160 to move to the designated area.

When receiving the cleaning pattern information from the server, the driving control module 151 may divide the entire driving zone into a plurality of regions according to the received cleaning pattern information, and set one or more regions as a designated region.

In addition, the driving control module 151 may calculate a driving route according to the received cleaning pattern information, travel along the driving route, and perform cleaning.

When cleaning of the set designated area is completed, the controller 150 may store the cleaning record in the storage unit 105 .

In addition, the control unit 150 may transmit the operation state or the cleaning state of the mobile robot 100 to an external terminal or a server at a predetermined cycle through the communication unit 190 .

Accordingly, the external terminal displays the location of the mobile robot along with a map on the screen of the running application based on the received data, and also outputs information on the cleaning status.

The mobile robot 100 according to an embodiment of the present invention moves in one direction until an obstacle or a wall is detected, and when the obstacle recognition module 154 recognizes an obstacle, it travels straight ahead, rotates, etc. according to the properties of the recognized obstacle. pattern can be determined.

Meanwhile, the controller 150 may control to perform the avoidance driving in a different pattern based on the recognized attribute of the obstacle. The controller 150 may control the avoidance driving in a different pattern according to the properties of the obstacle, such as a non-hazardous obstacle (a general obstacle), a dangerous obstacle, a movable obstacle, and the like.

For example, the controller 150 may control the dangerous obstacle to be bypassed and avoided while securing a longer safety distance.

In addition, in the case of a movable obstacle, if the obstacle does not move even after a predetermined waiting time, the controller 150 may control to perform an evasive driving corresponding to a general obstacle or an evasive driving corresponding to a dangerous obstacle. Alternatively, when an avoidance driving pattern corresponding to a movable obstacle is set separately, the controller 150 may control the vehicle to travel according to the setting.

The mobile robot 100 according to an embodiment of the present invention may perform machine learning-based obstacle recognition and avoidance.

The controller 150 controls the driving of the driving unit 160 based on the obstacle recognition module 154 for recognizing an obstacle previously learned by machine learning from the input image and the properties of the recognized obstacle. It may include a driving control module 151 to control.

Meanwhile, although FIG. 5 illustrates an example in which the plurality of modules 151 , 152 , 153 , and 154 are separately provided in the control unit 150 , the present invention is not limited thereto.

For example, the location recognition module 153 and the obstacle recognition module 154 may be integrated as one recognizer to constitute one recognition module 155 . In this case, the recognizer is trained using a learning technique such as machine learning, and the learned recognizer can recognize properties such as regions and objects by classifying input data thereafter.

According to an embodiment, the map generation module 152 , the location recognition module 153 , and the obstacle recognition module 154 may be configured as one integrated module.

Hereinafter, the location recognition module 153 and the obstacle recognition module 154 are integrated as one recognizer to focus on an embodiment composed of one recognition module 155, but the location recognition module 153 and the obstacle recognition module Even when each module 154 is provided, it may operate in the same manner.

The mobile robot 100 according to an embodiment of the present invention may include a recognition module 155 in which properties of objects and spaces are learned through machine learning.

Machine learning means that a computer learns from data without a human instructing logic directly to the computer and allows the computer to solve problems on its own.

Deep Learning is. Based on Artificial Neural Networks (ANN) for constructing artificial intelligence, it is a method of teaching a computer how to think of a person. .

The artificial neural network (ANN) may be implemented in the form of software or hardware such as a chip.

The recognition module 155 may include an artificial neural network (ANN) in the form of software or hardware in which properties of objects such as spatial properties and obstacles are learned.

For example, the recognition module 155 is a deep neural network (DNN) such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and a Deep Belief Network (DBN) trained by deep learning. may include

The recognition module 155 may determine the properties of a space and an object included in the input image data based on weights between nodes included in the deep neural network (DNN).

Meanwhile, the driving control module 151 may control the driving of the driving unit 160 based on the recognized properties of the space and obstacles.

Meanwhile, the recognition module 155 may recognize the properties of a space and an obstacle included in the selected specific viewpoint image based on data previously learned through machine learning.

Meanwhile, the storage unit 105 may store input data for determining space and object properties, and data for learning the deep neural network (DNN).

The storage unit 105 may store the original image acquired by the image acquisition unit 120 and the extracted images from which a predetermined region is extracted.

In addition, according to an embodiment, the storage unit 105 may store weights and biases constituting the structure of the deep neural network (DNN).

Alternatively, according to an embodiment, weights and biases constituting the deep neural network structure may be stored in an embedded memory of the recognition module 155 .

Meanwhile, the recognition module 155 performs a learning process by using a predetermined image as training data whenever the image acquisition unit 120 acquires an image or extracts a partial region of the image, or After the image is acquired, a learning process may be performed.

Alternatively, the mobile robot 100 may receive data related to machine learning from the predetermined server through the communication unit 190 .

In this case, the mobile robot 100 may update the recognition module 155 based on the machine learning-related data received from the predetermined server.

6 is a diagram referenced in the description of learning using product data according to an embodiment of the present invention.

Referring to FIG. 6 , product data obtained by the operation of a predetermined device such as the mobile robot 100 may be transmitted to the server 70 .

For example, the mobile robot 100 may transmit space, object, and usage related data to the server 70 to the server 70 .

Here, the space and object related data are data related to the recognition of a space and an object recognized by the mobile robot 100 or a space acquired by the image acquisition unit 120 . and image data for objects.

In addition, usage-related data (Data) is data acquired according to the use of a predetermined product, for example, the mobile robot 100 , and includes usage history data, sensing data acquired by the sensor unit 170 , and the like. can be

Meanwhile, a deep neural network structure (DNN) such as a Convolutional Neural Network (CNN) may be mounted on the control unit 150 of the mobile robot 100 , more specifically, the recognition module 155 .

The learned deep neural network structure (DNN) may receive input data for recognition, recognize the properties of objects and spaces included in the input data, and output the result.

In addition, the learned deep neural network structure (DNN) receives input data for recognition, analyzes and learns usage-related data of the mobile robot 100 to recognize usage patterns, usage environments, etc. there is.

Meanwhile, space, object, and usage related data may be transmitted to the server 70 through the communication unit 190 .

The server 70 may generate a configuration of learned weights, and the server 70 may learn a deep neural network (DNN) structure using training data.

After the server 70 trains a deep neural network (DNN) based on the received data, the updated deep neural network (DNN) structure data may be transmitted to the mobile robot 100 to be updated.

Accordingly, home appliance products such as the mobile robot 100 become smarter and can provide a user experience (UX) that evolves as it is used.

7 is an example of a simplified internal block diagram of a server according to an embodiment of the present invention;

Referring to FIG. 7 , the server 70 may include a communication unit 720 , a storage unit 730 , a learning module 740 , and a processor 710 .

The processor 710 may control the overall operation of the server 70 .

Meanwhile, the server 70 may be a server operated by a home appliance manufacturer such as the mobile robot 100 or a server operated by a service provider, or may be a kind of cloud server.

The communication unit 720 may receive various data, such as state information, operation information, and operation information, from a home appliance such as a mobile terminal and a mobile robot 100 , a gateway, and the like.

In addition, the communication unit 720 may transmit data corresponding to various types of received information to a mobile terminal, a home appliance such as the mobile robot 100, a gateway, or the like.

To this end, the communication unit 720 may include one or more communication modules such as an Internet module and a mobile communication module.

The storage unit 730 may store received information and include data for generating result information corresponding thereto.

Also, the storage unit 730 may store data used for machine learning, result data, and the like.

The learning module 740 may serve as a learner of a home appliance such as the mobile robot 100 .

The learning module 740 may include an artificial neural network, for example, a deep neural network (DNN) such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or a Deep Belief Network (DBN). Neural networks can be trained.

Meanwhile, the processor 710 may control to update the artificial neural network structure of the home appliance such as the mobile robot 100 to the learned artificial neural network structure after learning according to a setting.

Also, the learning module 740 may receive input data for recognition, recognize properties of objects and spaces included in the input data, and output the result. In this case, the communication unit 720 may transmit the recognition result to the mobile robot 100 .

In addition, the learning module 740 may analyze and learn usage-related data of the mobile robot 100 to recognize a usage pattern, a usage environment, and the like, and output the result. In this case, the communication unit 720 may transmit the recognition result to the mobile robot 100 .

Accordingly, home appliance products such as the mobile robot 100 may receive the recognition result from the server 70 and operate using the received recognition result.

In addition, since the server 70 learns using product data and becomes smarter, it is possible to provide a user experience (UX) that evolves as the home appliance product is used.

Meanwhile, the mobile robot 100 and the server 70 may also use external information.

For example, the mobile robot 100 and the server 70 may be obtained from internal information such as spatial information, object information, usage pattern, etc. of a specific home appliance product such as the mobile robot 100 and other products, or the server 70 . can provide an excellent user experience by comprehensively using external information obtained from other linked service servers.

According to an embodiment of the present invention, it is possible to optimize the operation order of products in the home appliance network system connected by Wi-Fi through artificial intelligence, and present the optimal operation order to the user.

For example, house cleaning is performed with the mobile robot 100 , dust generated during cleaning is performed with the air purifiers 12 and 13 , humidity and temperature of the house are managed with the air conditioner 11 , and the washing machine 32 ) may perform the washing so that the washing ends according to the time when the user arrives at the house.

The server 70 may perform voice recognition by receiving a voice input signal uttered by a user. To this end, the server 70 may include a voice recognition module, and the voice recognition module may include an artificial neural network trained to perform voice recognition on input data and output a voice recognition result.

Meanwhile, the server 70 may include a voice recognition server for voice recognition. Also, the voice recognition server may include a plurality of servers that share and perform a predetermined process among the voice recognition processes. For example, the speech recognition server may include an Automatic Speech Recognition (ASR) server that receives speech data and converts the received speech data into text data, and the text from the automatic speech recognition server. It may include a natural language processing (NLP) server that receives data and analyzes the received text data to determine a voice command. In some cases, the voice recognition server may further include a text-to-speech (TTS) server that converts the text-to-speech recognition result output by the natural language processing server into voice data and transmits it to another server or home appliance. .

According to the present invention, the mobile robot 100 and/or the server 70 can perform voice recognition, so that the user's voice can be used as an input for controlling the mobile robot 100 .

In the conventional robot cleaner supporting voice recognition, voice recognition is merely a means for control. As a result, the robot vacuum cleaner performs only the recognized control commands, so the analysis of which functions the user currently uses the most and which parts are less cleaned has a disadvantage in that it is necessary to find out through a method other than voice.

However, according to the present invention, the mobile robot 100 may ignite and provide a simpler interaction function than simple terminal control.

In addition, according to the present invention, the mobile robot 100 can provide more diverse and active control functions to the user by actively providing information first or outputting a voice for recommending a function or service.

In addition, according to the present invention, the mobile robot 100 can learn and understand the user's usage pattern. Accordingly, the mobile robot 100 can interact while first suggesting a predetermined function to the user, thereby performing more efficient and user-friendly operation.

8 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention, and FIG. 9 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.

Referring to FIG. 8 , the mobile robot 100 may start cleaning while moving according to a command or setting ( S810 ). For example, the mobile robot 100 may move based on a navigation map or a SLAM (Simultaneous localization and mapping) map stored in the storage unit 105 according to a cleaning start command or a cleaning reservation setting.

The mobile robot 100 may perform cleaning while moving, and may store sensing data obtained from the sensor unit 170 , image data obtained from the image acquisition unit 120 , and the like in the storage unit 105 .

In addition, the mobile robot 100 may store data obtained by performing cleaning, such as driving history data and cleaning history data, in the storage unit 105 .

Here, the data obtained by performing cleaning may be a usage record of the mobile robot 100 . For example, the number of times of general cleaning completed, the number of returning to the charging station, the number of cleaning functions completed by area, operating time, average cleaning intensity by area, amount of dust by area, cleaning mode by area (fine cleaning, quick cleaning, general cleaning, etc.) ) frequency and rate, monitoring, designated cleaning, intensive cleaning, the number of times of use of functions such as virtual wall cleaning, and concentration by time of the above-mentioned record.

In addition, not only the cleaning function performed by the mobile robot 100 but also usage records related to other functions may be stored.

For example, cleaning usage history such as designated cleaning, intensive cleaning, virtual wall cleaning, etc., monitoring for a designated area, recognizing a moving object, photographing a recognized object and sending it to a designated device Functions that mobile robots can provide, such as home guard function and mapping, and environmental sensing and driving strategies such as finding objects, measuring temperature, humidity, and air quality dust can be stored as usage records.

Using the data obtained through such driving, the mobile robot 100 and/or the server 70 provides the user's mobile robot 100 usage pattern, spatial information for each area in the use environment such as home, and objects existing in the use environment. Analyze and learn information.

In addition, the mobile robot 100 may determine a special area based on existing data obtained by performing previous cleaning (S820). The mobile robot 100 may identify a current location and space based on the image obtained from the image acquisition unit 120 and may determine whether there is a special area. Also, the mobile robot 100 may determine whether a special area exists by additionally checking information for each area of the corresponding space based on the currently identified location and space information.

Here, the special area may be a dangerous area in which the driving disabled state occurred. For example, the dangerous area may be an area in which a driving impossible situation such as a stuck, a trap, a fall, or a power off occurs.

Alternatively, the special area may be a low-efficiency area in which the running speed or the cleaning rate is lower than the average value or the number of rotations is greater than the reference number. For example, the low-efficiency area may correspond to an area in which a certain area within the area is driven below a specified average speed, an area where the number of turns is frequent, an area where all corresponding motions occur more than a predetermined number of times, and the like.

When the mobile robot 100 arrives at the determined special area while moving (S830), it may output notification information for the special area (S840).

Preferably, the sound output unit 181 of the mobile robot 100 may output a voice guidance message for the special area (S840). Accordingly, it is possible to attract the user's attention and to naturally check the situation of the special area when the user turns or moves his/her head in the direction in which the sound is heard.

On the other hand, the existing data obtained by performing the previous cleaning may further include obstacle information registered in the special area. In this case, the voice guidance message uttered by the mobile robot 100 may include a guidance message for an obstacle existing in the special area. Accordingly, the user can immediately determine the type of obstacle and the degree of danger, and instruct the next operation of the mobile robot 100 .

The voice guidance message may include a message requesting confirmation of whether to clean the special area.

In addition, the voice guidance message may further include at least one of an explanation of the reason for determining the special area, or a guidance on an example of a command that the user can input by voice.

By providing an explanation of the reason for determining the special area, it can help the user to more quickly understand the special area.

By providing guidance on examples of commands that the user can input by voice, it can help the user to accurately input a feedback voice. In addition, when the user utters the exemplified command, the mobile robot 100 and/or the server 70 may recognize the user's voice command more quickly and accurately.

In addition, information such as why it is a dangerous area and what commands can be used can be simultaneously guided by voice. For example, the mobile robot 100 may utter, "This is an area where operation has been stopped in the past. If you want to clean, please answer "Clean".

The mobile robot 100 may determine a special area such as a dangerous area or a low-efficiency area (S820). When arriving at a special area (S830), it is possible to first inquire whether to perform cleaning for the special area (S840).

Referring to FIG. 9 , the mobile robot 100 that arrives at a dangerous area 910 in which many wires exist may ignite a voice guidance message 920 for guiding the dangerous area.

The voice guidance message 920 may include a content 921 that the wire is complicated and a content 922 that it is difficult to clean.

The mobile robot 100 according to an embodiment of the present invention, recognized through a plurality of cleaning, when entering a special area, such as a cleaning danger area where dangerous obstacles exist, a cleaning failure area with a cleaning failure history, etc., through a voice It can notify the danger area of cleaning, etc.

For example, there is a possibility that the movement of the mobile robot 100 is stuck in an indoor driving environment in which dangerous obstacles such as electric wires exist, and there is a high probability that a continuous restraint situation occurs in a place where the restraint situation occurs once.

Continuous occurrence of restraints may cause increased user dissatisfaction and lower product reliability.

Accordingly, the mobile robot 100 according to an embodiment of the present invention may notify the user that it is aware of a dangerous situation through utterance when entering the danger section. Accordingly, it is possible to increase the user's reliability and preference.

In addition, in areas that are too complex to drive, the user may be encouraged to clean up on their own, providing additional assistance to complete the cleaning.

For example, when entering a special area, the mobile robot 100 ignites a voice guidance message such as “It is difficult for me to clean here”, and the user can check whether there are complicated wires or toys in the area and remove it. Thereafter, the mobile robot 100 can safely perform cleaning of the corresponding area.

Meanwhile, according to the user's feedback response, cleaning of a special area may be skipped, so that cleaning of other areas may not be disturbed or an overall cleaning time may be reduced. For example, when the user says “Do not clean there”, the mobile robot 100 does not clean the corresponding area.

On the other hand, when the user does not understand the voice guidance message due to the cleaning noise during utterance and says “what?”, the mobile robot 100 may repeat the voice guidance message just output.

On the other hand, the mobile robot 100 may record a place where a driving impossible situation such as restraint or power off occurs during cleaning, and when a driving impossible situation occurs more than a certain number of times, the corresponding area may be classified as a dangerous area.

In this case, it is possible to increase the guidance strength of the voice guidance message and the like according to the frequency of occurrence of the driving impossible situation.

For example, when starting cleaning, the mobile robot 100 responds to a dangerous area in which driving is habitually impossible, saying, “Master, come here. It's really hard for me to clean here. If you clean it up a little, I will clean it harder." You can increase the level of expression intensity of the guidance.

In addition, the mobile robot 100 interacts with the user while first asking by voice whether it is necessary to avoid inefficient actions in the section in which the mobile robot 100 performs inefficient operations (the running speed or cleaning rate is lower than the average value, or the number of rotations is greater than the reference number). can

The special area is a dangerous area that may adversely affect the cleaning performance of other areas, such as the movement of the mobile robot 100 being restricted, or a low-efficiency area in which it is difficult to perform cleaning.

Accordingly, the user may be asked whether to clean the special area (S840), and when user feedback is received (S850), an operation may be performed according to the received feedback (S860).

The mobile robot 100 may receive a user's voice feedback for the voice guidance message during a predetermined voice waiting time (S850),

When the user's voice feedback is received, a voice command included in the received voice feedback may be identified and an operation corresponding to the identified voice command may be performed ( S860 ).

The user's feedback response may determine that cleaning is desired if positive words such as cleaning, proceeding, continuing, okay, uh, yes, etc. are used.

When the user's feedback response includes negative words such as no, do not clean, do not do, it may be determined that cleaning is desired.

According to an embodiment, the identification of the voice command for the user's voice feedback may be performed by the mobile robot 100 itself.

For example, the mobile robot 100 may include an artificial neural network trained to recognize a voice included in input data, and the artificial neural network may recognize a voice command included in the received voice feedback.

Alternatively, the identification of the voice command for the user's voice feedback may be performed by the server 70 .

In this case, the step of identifying the voice command included in the received voice feedback includes transmitting the received voice feedback data to a voice recognition server including an artificial neural network trained to recognize a voice included in the input data. and receiving a voice recognition result of the received voice feedback data from the voice recognition server.

Here, the voice recognition server may be configured as a part of the server 70 , but may be configured as a dedicated server for voice recognition separately from the server 70 .

Alternatively, it may be performed step by step in the mobile robot 100 and the server 70 . In the case of a simple voice command, the mobile robot 100 primarily recognizes it, and when the mobile robot 100 does not recognize it, the server 70 may be used.

In this case, the step of identifying the voice command may include: when the received voice feedback includes a preset keyword, identifying a voice command corresponding to the keyword; transmitting the received voice feedback data to a voice recognition server including an artificial neural network trained to recognize a voice included in input data when it does not include a set keyword, and the voice recognition server It may include receiving a voice recognition result of the received voice feedback data from the.

That is, the mobile robot 100 may recognize a simple keyword, such as a positive vocabulary, a negative vocabulary, or a specific vocabulary guided by an example of a command, and operate according to the recognized keyword.

Also, if the keyword is not recognized, the mobile robot 100 may transmit the user's voice feedback data to the server 70 and receive the voice recognition result from the server 70 .

Meanwhile, if the user's feedback is not received for a predetermined time (S850), the mobile robot 100 may perform a preset avoidance operation (S870).

According to an embodiment, if the user's feedback is not received for a predetermined time ( S850 ), the mobile robot 100 may be set to force cleaning.

According to an embodiment of the present invention, the mobile robot 100 may start to talk to the user after a predetermined time has elapsed after cleaning without saying anything at first.

That is, the mobile robot 100 may perform deep learning with data obtained by performing a plurality of cleaning, recognize a dangerous obstacle or a situation of its location through deep learning, and explain the situation in a complex manner.

In addition, it is possible to communicate with users by continuously updating hazardous areas and low-efficiency areas in the changing home environment.

According to the present invention, it is possible to improve the satisfaction of the voice recognition function and the overall product through interaction with a user as well as simple control of the mobile robot 100 through voice.

In addition, it controls difficult-to-clean areas to receive user input by voice. By cleaning or not cleaning according to a user's voice input, cleaning efficiency may also be improved.

10 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention.

Referring to FIG. 10 , the mobile robot 100 according to an embodiment of the present invention may receive a user's voice command ( S1010 ).

A conventional voice recognition mobile robot recognizes a received voice command and immediately performs a corresponding operation.

However, the mobile robot 100 according to an embodiment of the present invention may check and confirm existing data stored, such as a usage record, according to a user's voice command reception (S1010) (S1020).

Here, the usage records include the number of times of general cleaning completed, the number of returning to the charging station, the number of cleaning functions completed by area, operating time, average cleaning intensity by area, amount of dust by area, frequency of use of cleaning mode by area, rate of use of cleaning mode by area, It may include at least one of a cleaning function and the number of times of use for each additional function. In addition, it may further include a concentration level for each time period of the items included in the above-mentioned usage record.

In addition, not only the cleaning function performed by the mobile robot 100 but also usage records related to other functions may be stored and confirmed later.

For example, cleaning usage history such as designated cleaning, intensive cleaning, virtual wall cleaning, etc., monitoring for a designated area, recognizing a moving object, photographing a recognized object and sending it to a designated device Functions that mobile robots can provide, such as home guard function and mapping, and environmental sensing and driving strategies such as finding objects, measuring temperature, humidity, and air quality dust can be stored as usage records.

Using the data obtained through such driving, the mobile robot 100 and/or the server 70 provides the user's mobile robot 100 usage pattern, spatial information for each area in the use environment such as home, and objects existing in the use environment. Analyze and learn information.

The mobile robot 100 may output a voice guidance message for recommending at least one of the functions used less than a reference number of times based on the usage record ( S1030 ).

According to an embodiment of the present invention, the mobile robot 100 may induce other functions to be used by suggesting functions not normally used to the user by voice.

In addition, a user who has input a predetermined voice is located in an appropriate short distance to communicate with the mobile robot 100 . Accordingly, the mobile robot 100 can recommend a predetermined function or provide information to a user located in a short distance before performing a command corresponding to the user's voice, and natural interaction is possible.

In addition, it is possible to provide an improved cleaning service while the mobile robot 100 first asks about a function and a case that the user considers necessary.

The controller 150 may control the sound output unit 181 to output a recommended voice guidance message by selecting at least one of the functions used less than a reference number of times according to a preset priority.

Here, the preset priority may be an order of increasing importance for each preset function. For example, when the SLAM success rate is low, the mapping function is given the 1st priority, the uncleaned area cleaning function is the 2nd priority, and the functions according to the schedule such as absent cleaning and home guard are the 3rd priority, and the non-clean area virtual wall is the 3rd priority. Addition and dangerous area setting function can be set as 4th priority. In this case, a function having the highest priority among functions not frequently used by the user may be recommended.

Alternatively, the preset priority may be in the order of the number of times of use being small. In other words, it is possible to recommend features that are not used or are less frequently used. In this case, in the case of a function having the same number of uses, it may be randomly recommended, or a predetermined recommended function pool may be set and functions included in the pool may be preferentially recommended.

In addition, the control unit 150 may determine the recommended function by checking and confirming necessary data according to the preset priority.

Meanwhile, according to the reception of the user's voice command ( S1010 ), the mobile robot 100 and/or the server 70 may identify the user's voice command.

In this case, it is possible to select and recommend a function related to the identified user's voice command from among the functions used less than the reference number of times.

To this end, the mobile robot 100 and/or the server 70 may recognize the received voice command before or after checking the usage record ( S1020 ).

According to an embodiment of the present invention, a similar cleaning function or a helpful function may be recommended based on the control command determined by voice recognition. For example, if the user's voice command is determined as a thorough cleaning command, a function for performing intensive cleaning in an area with a large amount of dust may be recommended, and if it is determined as a quick cleaning command, a skip of an area that is not frequently cleaned may be recommended.

On the other hand, when the user's voice feedback for the voice guidance message is received within a predetermined time (S1040), a feedback voice command included in the received voice feedback is identified, and a predetermined function is based on the identified feedback voice command. can be set (S1050).

Thereafter, the mobile robot 100 may perform an operation corresponding to the identified user's voice command in response to the reception of the user's voice command (S1010) (S0160).

Meanwhile, as described with reference to FIG. 8 , the identification of the voice command for the user's voice feedback may be performed by the mobile robot 100 itself.

For example, the mobile robot 100 may include an artificial neural network trained to recognize a voice included in input data, and the artificial neural network may recognize a voice command included in the received voice feedback.

Alternatively, the identification of the voice command for the user's voice feedback may be performed by the server 70 .

In this case, the step of identifying the voice command included in the received voice feedback includes transmitting the received voice feedback data to a voice recognition server including an artificial neural network trained to recognize a voice included in the input data. and receiving a voice recognition result of the received voice feedback data from the voice recognition server.

According to the present invention, the mobile robot 100 can communicate with the user while recognizing and analyzing the user's usage pattern and suggesting functions not normally used first.

According to the present invention, it is possible to improve the voice recognition function and product satisfaction by not only controlling the mobile robot 100 through voice, but also interacting with the user.

In addition, when the mobile robot 100 determines that a better cleaning strategy (cleaning mode, schedule) is determined, the determined cleaning strategy or the like may be recommended by voice.

And according to the user's feedback voice input, the corresponding function can be set and used.

Hereinafter, various embodiments in which the mobile robot 100 actively interacts with a user will be described with reference to the drawings.

11 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention, and FIG. 12 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.

Referring to FIG. 11 , the mobile robot 100 may recognize that the front door is opened ( S1110 ). Recognition of the entrance door open may be recognized based on a signal received from another device such as a sensor attached to the entrance door or a server.

When the entrance door open is recognized, the mobile robot 100 may move to the entrance door (S1120) and recognize the user (S1130).

For example, user recognition may be performed by recognizing an image acquired through the image acquisition unit 120 . Alternatively, the related information may be received from another home appliance or server.

When the user is recognized ( S1130 ), the mobile robot 100 may brief the operation status of the mobile robot 100 by voice in the absence of the user ( S1140 ).

The mobile robot 100 may brief the operation state of the mobile robot 100 by voice when the user is absent ( S1140 ).

Referring to FIG. 12 , a voice guide message 1210 briefing the operation status of the mobile robot 100 is transmitted through the sound output unit 181 in the absence of the user by approaching the user coming in from the entrance of the mobile robot 100 . can be printed out.

When not in the house, when a person enters, the mobile robot 100 may meet and provide voice feedback at the entrance. After the briefing, if there is a lot of fine dust or there is a user's request, the mobile robot 100 may clean the surroundings.

In addition, the mobile robot 100 may brief the operation status and results in the absence of a user of the home appliance included in the home appliance network system connected through Wi-Fi communication.

By detecting the user's return home, the mobile robot 100 may first guide the operation record of the home appliances on the day.

The mobile robot 100 may provide a voice briefing while following the user's movement.

For example, as a companion animal greets a person returning home, the mobile robot 100 may recognize a user entering the house and then follow the user and talk about the cleaning record of the day.

Meanwhile, the mobile robot 100 may operate in a user-following driving mode in response to a user's utterance such as “Roboking, come here!”.

In addition, the mobile robot 100 may provide predetermined data, such as a picture of an uncleaned area that has not been cleaned during house cleaning in the absence of the user, to the user through the application.

According to an embodiment of the present invention, the mobile robot 100 may act similarly to a 'pet' so that the user may feel good and fun.

In addition, the user can know the operation status of the home appliances in the absence of the user through the voice of the mobile robot 100 approaching the user after returning home, without the need for the user to check the status of the home appliances in the home.

In addition, it is possible to increase the satisfaction of the product by giving feedback that the user is actively recognized.

If the user is not recognized (S1130), until a failure criterion set based on time or number of times is satisfied (S1150), user recognition may be performed.

If the user is not recognized until the failure criterion is satisfied (S1150), a preset operation such as standby and home guard after returning to the charging station may be performed (S1160).

13 is a flowchart illustrating a method for controlling a mobile robot according to an embodiment of the present invention, and FIG. 14 is a diagram referenced in the description of a method for controlling a mobile robot according to an embodiment of the present invention.

13 and 14 , the mobile robot 100 may receive predetermined data from other devices included in the home appliance network system (S1810).

For example, the air purifier 200 may convert the dust or air quality index data for each space/area through an indoor air sensor. Also, the air purifier 200 transmits the dust or air quality index data with the mobile robot 100 . can share

The air purifier 200 may detect whether dust is generated, or may detect a space where many people have gathered and then left.

When the air purifier 200 detects the generation of dust, it may start driving and transmit dust generation location data, etc. to the mobile robot 100 .

Meanwhile, the mobile robot 100 may establish and recommend a cleaning plan based on data received from the air purifier 200 and previously learned data ( S1320 ).

For example, the mobile robot 100 may guide by recommending intensive cleaning of a dusty space, and recommending and inducing the direction change of the air purifier 200 .

In addition, the mobile robot 100 may suggest a cleaning plan to the user after a predetermined time in consideration of the degree of dust generation, the number of detected people, and the like. At this time, the mobile robot 100 may utter a voice guidance message 1410 including a current state and a cleaning plan, such as cleaning after one hour.

The mobile robot 100 may perform cleaning according to the user's response.

If there is a user's voice response 1420 including the start of cleaning after two hours, the mobile robot 100 may start cleaning after two hours.

According to an embodiment of the present invention, it is possible to provide a positive synergy effect through the interworking use scenario between home appliances.

In addition, home appliances first detect a situation requiring cleaning and suggest to the user, thereby increasing product reliability.

According to at least one of the embodiments of the present invention, the mobile robot may utter a voice to the user, and may talk and interact with the user through the voice.

In addition, according to at least one of the embodiments of the present invention, the mobile robot actively provides information before making a request and recommends services, functions, etc., thereby increasing user's reliability, preference, and product utilization.

In addition, according to at least one of the embodiments of the present invention, by actively guiding the mobile robot to a dangerous dangerous area and a low-efficiency area that is not easy to clean, the stability and user convenience of the mobile robot are improved, and operation efficiency and cleaning efficiency are improved. etc. can be improved.

In addition, according to at least one of the embodiments of the present invention, the user's reliability, preference, and product utilization may be increased by actively recommending useful functions among functions not frequently used by the user.

In addition, according to at least one of the embodiments of the present invention, by recommending another function related to the user's voice command, the user can easily set and use the related function without additional effort.

In addition, according to at least one of the embodiments of the present invention, effective voice recognition may be performed by performing voice recognition in the mobile robot itself, in the server, or in stages in the mobile robot and the server.

In addition, according to at least one of the embodiments of the present invention, it is possible to provide an evolving user experience (ux) by utilizing artificial intelligence and machine learning for voice recognition, obstacle recognition, product data analysis, and the like.

In the mobile robot according to the present invention, the configuration and method of the described embodiments are not limitedly applicable, but all or part of each embodiment is selectively combined so that various modifications can be made. may be configured.

Meanwhile, the method for controlling a mobile robot according to an embodiment of the present invention may be implemented as a processor-readable code on a processor-readable recording medium. The processor-readable recording medium includes all types of recording devices in which data readable by the processor is stored. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also includes those implemented in the form of carrier waves such as transmission over the Internet . In addition, the processor-readable recording medium is distributed in a computer system connected through a network, so that the processor-readable code can be stored and executed in a distributed manner.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims In addition, various modifications may be made by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Mobile Robot: 100
Server: 70
Storage: 105
Body: 110
Image Acquisition Unit: 120
Input: 125
Controls: 150
Driving part: 160
Sensor unit: 170
Output: 180
Communications Department: 190

Claims (19)

moving and performing cleaning;
identifying a current location and space based on the acquired image;
Determining a special area based on the existing data obtained by performing the previous cleaning;
outputting information on obstacles existing in the special area when arriving at the special area;
outputting a voice guidance message for the special area;
receiving a user's voice feedback on the voice guidance message;
identifying a voice command included in the voice feedback; and,
and performing an operation corresponding to the voice command. According to claim 1,
The special area is
the hazardous area in which the inability to drive occurred; or
A control method of a mobile robot, characterized in that it is a low-efficiency area in which the running speed or cleaning rate is lower than the average value, or the number of rotations is greater than the reference number. According to claim 1,
The existing data obtained by performing the previous cleaning, the control method of the mobile robot, characterized in that it includes driving history data (driving history data) and cleaning history data (cleaning history data). According to claim 1,
The voice feedback is a control method of a mobile robot including a next operation designated according to the position and risk of the obstacle. According to claim 1,
The voice guidance message is
and a message requesting confirmation of whether to clean the special area. According to claim 1,
The voice guidance message is
The control method of the mobile robot, characterized in that it further comprises at least one of a description of the reason for determining the special area, and a guide for examples of commands that can be input by a user's voice. According to claim 1,
If the voice feedback for the voice guidance message is not received within a predetermined time, performing a designated avoidance operation; the control method of the mobile robot further comprising. According to claim 1,
The step of identifying the voice command comprises:
transmitting the received voice feedback data to a voice recognition server including an artificial neural network trained to recognize a voice included in input data; and
and receiving a voice recognition result of the received voice feedback data from the voice recognition server. According to claim 1,
The mobile robot includes an artificial neural network trained to recognize a voice included in input data,
The step of identifying the voice command comprises:
The control method of a mobile robot, characterized in that the artificial neural network recognizes a voice command included in the received voice feedback. According to claim 1,
The step of identifying the voice command comprises:
when the received voice feedback includes a preset keyword, identifying a voice command corresponding to the keyword;
transmitting the received voice feedback data to a voice recognition server including an artificial neural network trained to recognize a voice included in input data when the received voice feedback does not include a preset keyword , and
and receiving a voice recognition result of the received voice feedback data from the voice recognition server. The method of claim 1,
Before performing the above cleaning,
receiving a user's voice command;
outputting a second voice guidance message for recommending at least one of functions used less than a reference number of times based on the usage record;
identifying a feedback voice command included in the received voice feedback when the user's voice feedback for the second voice guidance message is received; and
and setting a predetermined function based on the feedback voice command. 12. The method of claim 11,
The usage record is
The number of times of general cleaning completed, the number of times returned to the charging station, the number of times of cleaning function completed by area, operating time, average cleaning intensity by area, amount of dust by area, frequency of use of cleaning mode by area, ratio of cleaning mode used by area, number of times of use by cleaning function and additional function Control method of a mobile robot, characterized in that it comprises at least one of. 12. The method of claim 11,
Outputting the second voice guidance message comprises:
A method for controlling a mobile robot, characterized in that, according to a preset priority, at least one of the functions used less than a reference number is selected and recommended. 14. The method of claim 13,
The predetermined priority is a control method of a mobile robot, characterized in that the predetermined priority for each function is a high order of importance or a small number of times of use. 12. The method of claim 11,
Recognizing the user's voice command; Control method of the mobile robot further comprising a. delete 16. The method of claim 15,
Outputting the second voice guidance message comprises:
The control method of a mobile robot, characterized in that the selected and recommended functions associated with the identified user's voice command from among the functions used less than the reference number of times.

delete delete

Images