KR102008367B1 - System and method for autonomous mobile robot using a.i. planning and smart indoor work management system using the robot - Google Patents

Abstract

The present invention relates to an autonomous-moving intelligent robot using an artificial intelligence (AI) planning technology, an application thereof, and a smart indoor work management system using the robot and the application to move the robot to a required place in a home and a business site, photograph a required image, recognize the photographed image, and take a required measure based on a recognition result. According to the present invention, the smart indoor work management system comprises: an agent management unit using a simultaneous localization and mapping (SLAM) technology based on data inputted from vision and distance sensors mounted in an intelligent moving robot to make an indoor map for performing a smart indoor business management service during an operation preparation state and using an AI planning technology for executing a command, which is requested to an indoor business management application by a user, to move the intelligent moving robot from a current position to a destination without colliding with an obstacle during an operation state; an object information collection unit directly receiving image data inputted through a camera mounted on the intelligent moving robot in real-time, recognizing an object, in particular an actual object, from an image, and storing a tag for recognized information; and the indoor business management application installed in a user′s smartphone to receive one or more of text, voice, and an execution command through a GUI and transfer the command to the intelligent moving robot to transfer a user′s command to the intelligent moving robot and receiving a result of the command executed by the intelligent moving robot to transfer the result to the user. Accordingly, indoor business management can be efficiently realized with an inexpensive and simple structure.

Description

Self-driving mobile robot using artificial intelligence planning and smart indoor business management system and method using same {SYSTEM AND METHOD FOR AUTONOMOUS MOBILE ROBOT USING A.I. PLANNING AND SMART INDOOR WORK MANAGEMENT SYSTEM USING THE ROBOT}

The present invention is a self-driving intelligent mobile robot and app using artificial intelligence planning technology, using the robot and the app to move the robot to the necessary place in the home and business, take the necessary image, recognize the image, and recognize A smart indoor business management system and method for taking the necessary measures according to the results obtained. In addition, the present invention is an indoor task management system that controls the robot through the robot control technology and artificial intelligence planning technology such as counting the number of revolutions of the robot using the map to create the indoor map by the robot in place of the indoor positioning technology and It is about a method.

In general, a case of installing a plurality of cameras to secure necessary images in a home or a workplace and using them in a fixed or angle changing method is common. However, a more widespread use of a single camera is to mount the camera on a robot and move it to the desired location and desired angle so that the subject is shot under the optimal conditions. To this end, it is necessary to add mobility to the camera. The mobile robot can be mounted on an intelligent mobile robot, and the mobile robot can realize the optimum movement path, position, and angle adjustment to ensure that the camera captures a desired subject with high quality. Can be. The movement of the robot is convenient using positioning technology such as GPS in a given space. However, if the GPS signal is difficult to reach and the 1 ~ 2m error is an important factor, it can be implemented using indoor positioning technology. For example, Patent Document 1 (Publication No. 10-2014-0055537 (published May 09, 2014)) has been proposed as a positioning technology using WiFi, but indoor positioning technology is used when pre-set electronic maps and WiFi are assumed. In order to implement the technology with pre-set equipment, there is a problem in that the installation structure is relatively complicated and expensive.

In addition, for example, as in the case of Patent Document 2 (Patent Registration No. 10-1655760, published date: September 08, 2016), the existing smart home configures and controls the wireless Internet (WiFi) environment in the room. Configure the Internet of Things (IoT) environment on the appliance or device you want to control, control the appliance you want to control through the IoT operations program installed on your smartphone, or use human voices on the IoT hub It is possible to execute an operation such as turning on / off the power of the humidifier by issuing a command such as 'Turn on the humidifier'. In this case, only the products to which the Internet of Things, remote control, remote control, etc. are applied can be controlled in detail, and in the case of other home appliances, a separate power supply is provided in front of the power outlet to support simple power on / off functions. The Internet of Things had to use multi-tap.

However, even in this case, because the WiFi environment and the preset control system system must be used, the installation structure is complicated and the size is large, the system construction cost is excessive, the number of individual parts to be managed is high, the failure rate is high, and the labor and cost for maintenance are high. There is also a problem in that it costs.

Publication No. 10-2014-0055537 (released May 09, 2014) Patent Registration No. 10-1655760 (Announcement date September 08, 2016)

One embodiment of the present invention is to solve the problems of the prior art, in the management of indoor work in the home, building, factory, etc., even if there is no wireless communication-based control system preset in the room to go to the object to be controlled Self-driving mobile robot using artificial intelligence planning to determine the path by itself and recognize the state of the object and transmit the resulting object recognition information to the controller's smartphone app and smart indoor The purpose is to provide a work management system and method.

Another embodiment of the present invention is a system for controlling through a robot control technology and artificial intelligence planning technology, such as creating an indoor map by the robot itself in place of the indoor positioning technology and counting the number of revolutions of the robot using the map and It is an object to provide a method.

In addition, another embodiment of the present invention is a method of using a device and an app, such as a smartphone integrated with a camera and a computer for controlling the camera focus, exposure, time, etc., and recognizes the object, behavior, person, etc. from the photographed subject It is an object of the present invention to provide a system and method for repositioning the position of the camera and the like to obtain a higher recognition rate by feeding back the feedback. The object information thus obtained can be applied to the purpose of managing indoor tasks covering smart house management, buildings, factories, and office-like workplaces.

One aspect of the present invention for achieving the above object is an autonomous driving intelligent mobile robot and app using artificial intelligence planning technology, using the robot and the app to move the robot to the necessary place in the home and business, and necessary images A smart indoor work management system that takes pictures, recognizes these images, and takes necessary measures according to the recognized results, and inputs data from vision sensors and distance sensors mounted on the intelligent mobile robot in operation ready state. Intelligent mobile robot using artificial intelligence planning technology to create indoor map for conducting smart indoor business management service using Localization And Mapping) technology and to carry out commands requested by the user to indoor business management app at all times. An agent manager to move the object from the current position to the destination without collision of the obstacle; An object information collecting unit for directly receiving real-time image data input through a camera mounted on the intelligent mobile robot and recognizing objects in the image, in particular, object objects and storing tagging of the recognition information; And installed in the user's smartphone to receive the user's command to the intelligent mobile robot, receive one or more of text, voice, and execution commands through the GUI, and transmit the command to the intelligent mobile robot, which is performed by the intelligent mobile robot. It is configured to include; indoor business management app that receives the result of a command and delivers it to the user.

Preferably, the agent manager analyzes the command received from the indoor work management app installed in the user's smart phone to distinguish between the mobile action and the object action, and transfers the mobile action to the location control unit, and delivers the object action to the object information collection unit An agent controller for performing a command; Based on the movement action received from the agent controller, the intelligent mobile robot measures the movement amount of the intelligent mobile robot with the rotation amount of the wheel using Odometry so that the intelligent mobile robot can move to a specific position. A position control unit configured to calculate an position of the position and execute an action for moving to the commanded destination of the user; And using the SLAM technique, inputting the position value of the intelligent mobile robot obtained from the position control unit and the distance value calculated by scanning the XY plane with a distance camera, and using the SLAM technology to prepare a map distinguishing the walls from the obstacles. It is configured to include; map management unit for executing an action to update.

Preferably, the position control unit information on the occupied area, free area, unknown area in the map created using the current position of the intelligent mobile robot, obstacle position, SLAM technology A cost map constructing unit configured to execute an action constituting cost map information obtained by calculating a region which can be moved and an region which cannot be moved by using a? And calculate the optimal route to the destination based on the cost map information, and obtains information about the obstacle that changes in real time through the distance sensor to calculate the route that can quickly reach the destination while avoiding obstacles that can collide with the intelligent robot It is configured to include; obstacle avoidance unit for executing an action (Action).

Preferably, the position control unit calculates the position by measuring the distance moved by the intelligent mobile robot to the amount of rotation of the wheel rotation axis through dead-reckoning, and in the inertial measurement unit It consists of an action that corrects the error of the estimated navigation by applying the measured inertial information.

Preferably, the object information collection unit is an interface for receiving image streaming data requiring object information analysis, the image input unit for receiving the image streaming data input from the camera of the intelligent mobile robot; In order to recognize the objects included in the image input through the image input unit, one of an algorithm such as HMM (Hidden Markov Model), GMM (Gaussian Mixture Model), deep neural network, or long short-term memory (LSTM) is used. An image object recognition engine unit configured to form an edge on the objects (by performing an edge operation) to distinguish the objects and to execute an action of recognizing general objects; And the system recognizes and isolates only specific objects (specific objects (appliances, gas valves, boiler thermostats, lighting), people and impressions) among various objects and objects for people included in the general object. And a semantic object recognizer for tagging, storing tagging metadata, and executing an action for distinguishing a meaningful object from the stored object.

Preferably, the semantic object recognition unit includes at least a state of power on / off of a home appliance, a state of on / off of a lighting device, a temperature and a state displayed on a screen of a temperature controller of a boiler appearing from the general object in a building interior. A thing recognizing unit which executes an action for distinguishing the thing object; A person recognizing unit that distinguishes a person object from the general object such as an upper / lower color, an approximate height, wearing glasses / bags / hats, etc. as an impression on the person; And an event analyzer for performing an action for distinguishing a state change including at least one of a power change on and off of an illumination and a temperature change displayed on a screen of a boiler temperature controller in the object object.

Preferably, the indoor work management app provides a GUI that can provide a user and input and output information, and receives one or more of user's input text, voice, and execution commands, and the text and voice commands are included in the ontology conversion unit. An indoor operation management app management unit for transmitting and executing the execution command extracted by the ontology conversion unit and the execution command input to the GUI to the agent management unit of the intelligent mobile robot, and receiving the execution command execution result and displaying the execution command execution result on the GUI of the indoor operation management app; Using the semantic web technology by parsing one or more of the text and voice received from the indoor business management app management unit by applying a collection rule as input, generating and managing OWL data by applying a conversion rule, and converting it into knowledge triple of RDFS level. And an ontology conversion unit for converting a user's natural language into a command that can be processed by a computer.

Preferably, the indoor task management app management unit receives the indoor map, the location of the object located in the indoor map and the intelligent mobile robot from the agent management unit of the intelligent mobile robot and displays the location of the intelligent mobile robot on the GUI of the indoor task management app, inputting the user's command and It consists of receiving the result of command execution and displaying it on the GUI as one or more of text, voice, image, and video.

Preferably, the agent manager registers the movements related to the movement of the intelligent mobile robot and the movements related to the object collection as detailed actions in a domain plan, and inputs an execution command when a user execution command is input. The execution procedure of detailed actions to solve the problem is generated by artificial intelligence planning technology as one of the moving action and the object action, and automatically infers the execution command by executing the ROS detailed actions, and the moving action and the object action are Has sequential execution order information of one or more actions included in the domain plan.

Preferably, the action is included in the domain plan, including information about the precondition for the action to be executed and the result status after the action is executed.

Preferably, the execution instruction is declarerative and includes only the contents of the present state and the target state of the intelligent mobile robot, and how to get to the destination is executed by executing the moving action generated by the agent controller and planning the final destination position. To reach.

Another aspect of the present invention for realizing the above object is an autonomous driving intelligent mobile robot using artificial intelligence planning technology, using the robot to move the robot to the necessary place in the home and business, and take the necessary image, Smart indoor task management method that recognizes and takes necessary actions according to the recognized result.In the state of intelligent mobile robot operation preparation, the user requests execution command or performs autonomous driving through the GUI of the indoor task management app while the mobile robot is ready for operation. When the agent management unit of the intelligent mobile robot to enter the data of the vision sensor and the distance sensor mounted on the mobile robot to create an indoor map for the smart indoor business management service through the SLAM technology; When the intelligent mobile robot is always in operation, the agent control command of the intelligent mobile robot is divided into moving action and object action by the user through the GUI of the indoor work management app. Steps to move the intelligent mobile robot to the destination, the object information collection unit executes the action by inputting the object action to grasp the type and state of the object and deliver the result to the indoor work management app ; And the intelligent mobile robot detects obstacles during autonomous driving in the idle state, and the agent control unit of the intelligent mobile robot transmits the object action to collect the object information to the object information collecting unit, and the object information collecting unit executes the action to perform the object information. Or executing an action for distinguishing the state of the object through the object action delivered to the object information collecting unit by the agent control unit while the indoor map is created.

Preferably, when the intelligent mobile robot is ready for operation, the agent management unit of the intelligent mobile robot inputs the data of the vision sensor and the distance sensor mounted on the mobile robot by using one or more of command and autonomous driving through the indoor task management app. In the step of creating the indoor map for the smart indoor business management service through the SLAM technology, when the position controller of the intelligent mobile robot performs autonomous driving, it controls the wheels of the intelligent mobile robot using odometry. Executing an action for determining the position of the intelligent mobile robot through dead-reckoning, which measures the distance of movement of the wheel and measures the distance moved; The map management unit of the intelligent mobile robot generates an indoor map using SLAM technology by inputting the calculated position value of the intelligent mobile robot calculated by the position controller and the XY plane by using a distance camera. Executing an action; The cost control system calculates the areas of movement and areas that cannot be moved based on information about occupied areas, free areas, and unknown areas on a map created by the location controller in real time. And executing an action of proceeding autonomous driving while constructing Cost Map information and avoiding obstacles.

Preferably, while the intelligent mobile robot is always in operation, the agent control unit of the intelligent mobile robot distinguishes the execution command input through the GUI of the indoor work management app into the mobile action and the object action and the mobile action is the AI in the position controller. By executing the planning technology, the intelligent mobile robot is moved to the destination, and the object information collecting unit performs the object action to identify the type and state of the object and delivers the result to the indoor task management app. Enter one or more of the text, voice, and execution commands on the indoor map provided by the task management app GUI to move to specific objects such as home appliances (fridges, washing machines, air conditioners, air cleaners), windows, boiler thermostats, and door locks. 'Please check the status of the gas valve lock, the opening and closing of the window, the on and off status of the lighting equipment. Issuing a specific command among instructions that also include; The indoor task management app management unit converts the user's command into an execution command and transmits the command to the agent control unit of the intelligent mobile robot, and the mobile controller distinguishes the optimal path to the destination based on the cost map information. And executing an action for moving the intelligent mobile robot while avoiding obstacles using an algorithm (Dijkstra Algorithm), an A * algorithm, an electromagnetic field algorithm, and the like.

Preferably, the intelligent mobile robot detects obstacles during autonomous driving in the operation ready state, and the agent control unit of the intelligent mobile robot delivers the object action to collect the object information to the object information collection unit to collect the object information, or in the continuous operation state. In the step of executing the action that distinguishes the state of the object through the object action transmitted to the object information collecting unit by the agent manager, the intelligent mobile robot is based on the obstacle information (the position of the obstacle and the distance between the obstacles) delivered by the agent manager. Controlling an camera installed in the camera to photograph the subject at a desired angle, and executing an action of distinguishing the type and state of the object through the object information collecting unit; The object information collecting unit recognizes the state value of the object object in the impression of the person object and executes an action of distinguishing the state of the general object by the object action input distinguished by the agent control unit from the execution command of the user; And displaying the result of executing the execution command by the agent manager on the GUI of the app of the user's smartphone as one or more of the object photographed image and the state value of the object as a result.

According to the above configuration, the present invention can obtain the following effects.

The technical field of the present invention does not use expensive indoor positioning technology in a building where WIFI wireless Internet is installed, and as an alternative, an electronic device having a computer function having a central processing unit in an intelligent mobile robot, and a camera capable of taking a depth image Maps can be used to replace indoor positioning techniques. If a general camera is attached to a mobile robot, the technology for installing a fixed camera and setting it as an IoT system can be avoided. That is, in the case of the present invention, a camera having mobility is secured, and the user can obtain various advantages through the mobile camera. For example, through the object recognition in the image, it is possible to remotely grasp the state of the subject object such as the gas valve open state, to recognize the behavior such as the fall of the elderly in the home, or to recognize unwanted visitors through the face recognition. You will be able to understand the rich content and meaning that it contains. That is, in the present invention, the problem solved by installing a plurality of cameras in a given space when the camera is fixed, by attaching a wheeled robot that can give mobility to the camera has a benefit that can be secured through multiple cameras with one camera You can get it.

In the present invention, the intelligent mobile robot moves indoors and analyzes the input data of the vision sensor and the distance sensor of the depth camera mounted on the mobile robot, estimates the current position of the intelligent mobile robot using the SLAM technology, Prepare video maps of walls, objects, etc. in the surrounding space and prepare them for regular operation. Initially, the intelligent mobile robot moves the various places in the room by the user's request or autonomous driving, and proceeds to create the indoor map by itself, and the location information of the obstacles identified during the indoor map and the vision sensor and distance sensor installed in the intelligent mobile robot It receives data in real time and avoids collision with obstacles located in the room. The intelligent mobile robot receives a real-time image of the obstacle from a mounted camera and analyzes the input image to distinguish object information of the obstacle.

When the indoor map is ready, there is a need to find a way to recognize the surrounding obstacles faced by the intelligent mobile robot and to avoid them by switching to the normal operation state. Advanced technology to meet this need is AI planning technology. With the help of this planning technology, the user can perform the robot's own unit movements such as right turn, left turn and straight movement that can occur in the process of moving from the robot's origin to the destination. You can move the robot by specifying it. Therefore, it is possible to easily obtain object information or object state information of the entire room with only a single camera without installing an IoT-based control system in a state where a plurality of cameras are set in advance, so that appropriate object control can be performed accordingly. Can provide a foundation.

The user uses the 'in-house task management app' installed on the smartphone to input the intelligent mobile robot through one or more of voice, text, and graphical user interface (GUI), 'go to the user's location', ' Move to a certain object such as home appliances (refrigerators, washing machines, air conditioners, air cleaners), windows, boiler thermostats, door locks, '' state of gas valve lock, opening / closing window, on / off state of lighting equipment When the command is executed, the intelligent mobile robot analyzes the input command, calculates the optimal route to the destination using artificial intelligence planning technology, and performs the unit movement behavior by itself.

In order to convey the status of a specific object to the user, the camera of the smartphone is set to the focus, distance, etc. using the specific object as a subject, and the image is captured according to the purpose, and the image and the specific object of the specific object are used as the status information of the object. It includes a system and method for delivering one or more of the status information of the 'indoor work management app'. In this case, if the robot arm is installed in the intelligent mobile robot, the robot may move to the light switch to turn on / off the power. Therefore, it is possible to grasp the status of the target object in real time regardless of the user's location for indoor tasks such as housework or office work with a small amount of resources without the need for a large number of preset cameras or control systems. This can effectively control the indoor work at low cost.

In the present invention, the robot utilizes a planning tool called an artificial intelligence planner to generate an optimal route from the starting point to the destination on the indoor map. This path finding involves finding the optimal path while avoiding this obstacle if there is an obstacle in the middle. Planner is a technique that uses the method of declarative programming using the logical knowledge representation of artificial intelligence. Declarative programming is a program that does not tell you how to get from the initial state to the destination state in the middle of the process. This declarative program differs greatly from the way we use the procedural language we take in common computer programming languages such as C and Python. In the case of using procedural language, it suggests a specific way to reach the target state from the initial state (for example, turn right, turn left, go straight), so that the robot can reach the target state simply by performing this procedure. Therefore, when using such a procedural language, the robot can only perform a procedure for the programmer. However, in a planner with declarative programming, this procedure is found by the planner itself, which makes it very different for the programmer to express all of the initial state, destination state, and variables or actions that describe this environment in logical language. In particular, when a planner focused on declarative programming uses a language called PDDL, there are a problem file describing a problem itself and a domain file describing a domain. This domain file describes all of the Freddie kits and actions. Each action in the domain file has the necessary preconditions to perform and the effects are derived. The domain file simply describes all the paths including the number of cases that the robot goes from the initial state to the destination state. It is the same as describing the entire domain searching all branches of the tree in the search tree.

The problem file describes the objects, their initial state, and their target state. In other words, the heuristic finds an optimal path that can reach the target state in all cases that satisfy the prerequisites for performing the actions described in the domain file. Therefore, in the case of the procedural language method of the prior art, the programmer should assume all the possibilities that can occur, and if there is no impaired detail action, the AI robot will not operate even if there is a disorder, but the declarative program of the present invention. The method is simple and can actively cope with sudden failures, thereby realizing a simpler structure of the indoor work system compared with the conventionally set indoor work system.

1 is a block diagram of a self-driving mobile robot using artificial intelligence planning technology and a smart indoor business management system using the same according to an embodiment of the present invention.
Figure 2 is an internal configuration of an intelligent mobile robot according to an embodiment of the present invention.
3 is an internal configuration diagram of an agent management unit according to an embodiment of the present invention.
4 is an internal configuration diagram of a position control unit according to an embodiment of the present invention.
5 is an internal configuration diagram of an object information collecting unit according to an embodiment of the present invention.
6 is an internal configuration diagram of a semantic object recognition unit according to an embodiment of the present invention.
Figure 7 is an internal configuration of the indoor work management app according to an embodiment of the present invention.
8 is a flowchart illustrating a method of executing an autonomous mobile robot using artificial intelligence planning technology according to an embodiment of the present invention.
9 is a flow chart illustrating a method for creating an indoor map in an intelligent mobile robot ready to operate according to an embodiment of the present invention.
10 is a flowchart illustrating a method of executing an instruction of a user in an always-operated state by an intelligent mobile robot according to an embodiment of the present invention.
11 is a flowchart illustrating a method of collecting object information according to an embodiment of the present invention.

An embodiment of the present invention will be described.

Techniques used in the present invention may be broadly divided into three categories.

First, when the operation of the robot for the first time in the new space, that is, the operation ready state, such as home or business place, it is necessary to create a map to proceed the service. In the room, the intelligent mobile robot analyzes the images taken by autonomous driving or with the help of humans, and creates a map using SLAM technology. It measures and estimates the current position of the robot on the map by itself and recognizes the position of obstacles that hinder the movement of the intelligent mobile robot on the map created by SLAM technology using the vision sensor and distance sensor mounted on the intelligent mobile robot.

Second, the mobile robot enters a permanent operation state of the mobile robot based on the map created by the intelligent mobile robot and the location information of the intelligent mobile robot. In the normal operation state, the user can specify a location through an app on a device such as a smartphone and send a command such as “come here” to the robot so that the user can move to a specific location on the map. By calculating the optimal route to go to the commanded destination, the robot can infer and drive the robot without directing unit movements such as turning left, turning right and going straight. Artificial intelligence planning technology is used to output a series of unit movements that can move from the origin to the destination by itself. Therefore, the planning technology referred to in the present invention refers to a technology that allows a robot to perform a function of moving to a destination determined by a user by preparing a map and calculating an optimal route without receiving an instruction of a unit movement action.

Third, it is an application field such as image capturing and analysis using a camera mounted on the intelligent robot. In the room where the intelligent mobile robot is operated, the intelligent mobile robot moves to the desired point through user's request or autonomous driving, and then the subject is photographed by the camera mounted on the intelligent mobile robot at that point, and the captured image is input and received. It analyzes the data in real time or non real time and informs the object information necessary for the user's command execution.

In another embodiment of the present invention, a user uses a housekeeping app installed on a smartphone, and inputs one or more of voice, text, and graphical user interface (GUI) to display a specific object (home appliance, window, boiler thermostat). 'Go to where you are', 'check the status of gas valves, windows open and closed, lights on and off', intelligent mobile robot analyzes the input command and moves to the location For example, the present invention may include a system and a method of transmitting an image of photographing a specific object or delivering a result of execution of a command requested by the user to the 'household management app' in order to convey a state of the specific object to the user.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to Figure 1 will be described in detail the configuration and operation of the intelligent mobile robot image capture system using the artificial intelligence planning technology and the smart indoor task management system using the indoor task management app.

According to the present invention, an intelligent mobile robot 1000 equipped with wheels in any indoor space is equipped with a cognitive sensor 200 including at least one of a vision sensor, a distance sensor, and a camera, and a user 5000 of the smartphone 2000 is provided. When the command requested to the indoor work management app 2000a is transmitted to the intelligent mobile robot through the wireless communication 3000, the intelligent mobile robot executes the command of the user 5000 and the result is the GUI of the indoor work management app 2000a. Mark on.

At this time, LRF, ultrasonic sensor, infrared sensor, etc. are used for vision sensor, and stereo camera, mono camera, Kinect, Xtion, etc. are used for distance sensor.

As shown in FIG. 2, the intelligent mobile robot uses the GUI of the indoor work management app 2000a installed in the user's smartphone 2000 to execute an execution command requested by the user 5000 through the communication control unit 1300. 1100 is received and executed.

The agent manager 1100 of the intelligent mobile robot performs autonomous driving through a user's command or a scheduled command execution and operates a vision sensor 200-1, a distance sensor ( 200-2) receives data input from one or more of the input and output control unit 1400 to create an indoor map using the SLAM technology and to store in the storage space using the data storage unit 1500.

When the intelligent mobile robot 1000 detects an obstacle while running in the autonomous driving mode, the agent manager 1100 issues an object action to collect object information to the object information collecting unit 1200 and recognizes the object information collecting unit 1200. An action for distinguishing an object after receiving real time image data on an obstacle through the input / output controller 1400 by controlling one or more of the vision sensor 200-1 and the camera 200-3 among the sensors 200. Proceed with the process.

When the user 5000 enters one or more of text, voice, and execution commands through the GUI of the indoor work management app 2100a of the smart phone 2000, the indoor work management app 2100a is a ROS (installed on an intelligent mobile robot). The robot operating system converts the execution command into a command that can be processed and delivers it to the agent manager 1100.

The agent manager 1100 moves to a place where a specific object (home appliance, window, boiler temperature controller) is requested by the user 5000, 'state of gas valve, state of opening / closing a window, on / off of lighting device'. Analyze the command such as 'check the status,' and divided into the action and the object action, and transfers the object action to the object information collection unit 1200, and the object to execute the object action using the intelligent planning technology The object action performing action that executes the action of moving to the object and checks the object state information (gas valve state, window open / close state, lighting on / off state) analyzed by the object information collection unit 1200 ( Action) delivers the result to the indoor work management app 2100a through the communication control unit 1300.

At this time, the intelligent mobile robot 1000 is a compact having a central processing unit, a memory, an input / output device, and the like, capable of processing compression, analysis, storage, and recognition of images and data obtained through the mounted recognition sensor 200. It is used as a term that includes a computer, and has a feature that is more precisely and intelligently controlled by mounting a robot operating system (ROS) and artificial intelligence planning technology on a mechanical device that can be moved by mounting the recognition sensor 200.

The recognition sensor 200 mounted on the intelligent mobile robot 1000 may be installed on the gimbal 100 attached to the intelligent mobile robot to photograph the subject by adjusting a desired angle and height, and to the input / output controller 1400. It has a feature that is precisely controlled by it.

At this time, the artificial intelligence planning technology registers the motion related to the movement of the intelligent mobile robot and the motion related to the object collection as detailed actions in the domain plan, and executes the user in the agent manager 1100. When the command is entered, the detailed action to solve the execution command is created through artificial intelligence planning technology as one of the mobile action and the object action, so that the user can not execute the detailed action of the ROS detailed action even if the user does not care about the execution method of the intelligent mobile robot. It is a method of automatically inferring how to solve an execution command by executing (Actions).

At this time, the movement action and the object action have sequential execution order information of one or more actions included in the domain plan. The agent controller 1100 monitors the execution result of each action and detects an unexpected obstacle while the movement action is executed, and thus, if the movement action cannot be performed, the agent controller 1100 moves a movement action from the current position to the destination. Updates newly and executes newly created move action. The agent controller 1100 repeatedly updates and executes the mobile action to move the intelligent mobile robot to the final destination. In the case of an object action, if an exception occurs, the agent controller 1100 updates a new object action and finds a method of processing a user's execution command.

Each action is registered in the domain plan, including information on preconditions for executing the action and the result status after the action is executed, and the agent control unit 1110 of the agent management unit 1100 is configured to resolve the execution command. Based on the precondition result status of the actions registered in the domain plan, one or more of the movement action and the object action are generated.

Execution instructions are declarerative and contain only the current and intended states of intelligent mobile robots. For example, the execution instruction for the movement of the intelligent mobile robot includes information on the current position and the destination position of the intelligent mobile robot, and how to get to the destination by executing the movement action generated by the agent controller 1110 Reach the final destination location.

The agent control unit 1110 of the agent management unit 1000 distinguishes detailed actions for solving the user's problem plan into moving actions and object actions, which are sets of the detailed amounts of, and processes the moving actions. Actions are executed in order to execute user commands.

Referring to the problem that can be handled through actions included in the domain plan, as shown in FIG. 3, the agent manager 1100 is a user input through the GUI of the indoor work management app 2100a. Receives an execution command of 5000 and delivers it to the agent controller 1100, and the agent controller 1110 is based on the current location of the intelligent mobile robot managed by the map manager 1130 and the location information of the object targeted for the command. As a result, the command of the user 5000 is analyzed as a movement action and an object action, and the movement action of the intelligent mobile robot 1000 is transmitted to the position controller 1120, and the object action is transmitted to the object information collection unit 1200.

The position control unit 1120 controls the wheels of the intelligent mobile robot 1000 by using an odometry based on artificial intelligence planning technology to perform a moving action, and measures the amount of rotation of the wheel rotation axis to move the intelligent movement. An action for transmitting the current position of the robot 1000 to the map manager 1130 is executed. In this case, since an error occurs in the dead-reckoning for measuring the distance traveled by the amount of rotation of the wheel axis, the position controller 1120 obtains an inertia correction with an IMU sensor and reduces the error through position compensation.

The map manager 1130 stores the data using the SLAM technique as the input of the calculated position value of the intelligent mobile robot 1000 transmitted by the position controller 1120 and the XY plane by scanning the XY plane with a distance camera. When the map data stored in the unit 1500 is updated and an obstacle is detected, the corresponding information is transmitted to the agent controller 1110 and the action of updating the position of the obstacle to the coordinates on the map is executed. When the location information on the obstacle is updated, the agent controller 1110 may transfer the object action to collect the object information about the obstacle to the object information collecting unit 1200 to collect the object information about the obstacle.

In this case, SLAM (Simultaneous Localization And Mapping) refers to an intelligent mobile robot 1000 moving around a certain space while using one or more of the vision sensor 200-1 and the distance sensor 200-2 to recognize an obstacle around it. To estimate your current location and create a map at the same time.

As shown in FIG. 4, the cost map component unit 1130 receives the current location of the intelligent mobile robot and the position information of the obstacle on the map managed by the map manager 1130 to perform the unit mobile behavior. Cost map information is calculated based on information about occupied area, free area, and unknown area on the map for Configure. The obstacle avoidance unit 1122 may move the optimal path to the destination based on the configured cost map from the current position of the intelligent mobile robot 1000 using Dijkstra Algorithm, A * algorithm, and electromagnetic field algorithm. Iterate over the shortest possible distance nodes to find the best path. At this time, the map generated by the map manager 1130 through the SLAM can not provide information about the obstacles that are changed in real time to reach the destination one of the vision sensor (200-1), the distance sensor (200-2) Based on the information obtained above, an action of moving the intelligent mobile robot 1000 to the destination is performed while avoiding a real-time obstacle using a dynamic Windows Approach (DWA).

As illustrated in FIG. 5, the image input unit 1210 receives image data in real time through the input / output controller 1400 and streams the image data to the image object recognition engine unit 1220.

The image object recognition engine unit 1220 uses an algorithm such as HMM (Hidden Markov Model), GMM (Gaussian Mixture Model), deep neural network, or long short-term memory (LSTM) to recognize the objects included in the image data. By using the edges on the objects, the general objects are distinguished, and in the separated general objects, the object objects and the human objects are distinguished, and each state value is assigned.

The semantic object recognition unit 1230 is a target object (in particular, home appliances, gas valves, etc.) for the intelligent intelligent robot 1000 to execute an indoor work management command among various object objects and human objects included in the general objects. Recognizing, separating, tagging, and storing the tagging metadata in the data storage unit 1500 may be performed by recognizing, separating, and controlling the temperature of the boiler, lighting, and the user's impression.

As illustrated in FIG. 6, the object recognizing unit 1231 distinguishes an object from a general object and recognizes a state value (color, shape, brightness, displayed text) of the object. Among the object objects, home appliances, gas valves, boilers, temperature controllers, lighting, etc., which are used for housekeeping, are classified into target objects, and power on / off states of home appliances are turned on through the state values of the target objects. Perform an action to distinguish between the off state, the temperature displayed on the boiler's thermostat and the condition.

The person recognizing unit 1232 distinguishes a person object from a general object and performs an action of distinguishing a person object such as an upper / lower color, an approximate height, and wearing glasses / bags / hats as an impression on the person. . Among the objects, the user 5000 having the smart phone 2000 in which the indoor work management app 2000a is installed is distinguished as a target object. In this case, the method of classifying the user 5000 as a target object may use sensors capable of recognizing the location of the user in addition to the smart phone.

The event analysis unit 1233 distinguishes a state change such as power on / off of a light identified as a target object among temperature objects, a temperature change displayed on a screen of a boiler temperature controller, and a change in text or an image displayed on a screen of a home appliance. Then, it performs an action (Action) for monitoring a state change, such as moving the position of the user 5000.

As illustrated in FIG. 7, the indoor work management app managing unit 2100 may include a map in a building on a smart phone 2000 of a user 5000, a location of an intelligent mobile robot on a map, and a lower limit or more of a target object. Provides a graphical user interface (GUI) for displaying a user, and when the user receives one or more of a button click, a gesture, a text input, and a voice input of the GUI through the input / output control unit 2300, the indoor work management app managing unit 2100 The button clicks and gestures are converted into execution commands, and the text input and the voice input are transmitted to the ontology converter 2200. The ontology conversion unit converts the natural language input through the natural language processing algorithm into an execution command that can be processed by the ROS, and delivers it to the indoor business management app management unit 2100, and the indoor business management app management unit 2100 uses the communication control unit 2400. Transfer to the agent manager 1100 of the intelligent mobile robot (1000). At this time, the map displayed on the GUI displays walls, obstacles, and specific objects on the indoor map. In addition, the user 5000 provides functions necessary for the control of the intelligent mobile robot 1000, such as the addition and deletion of the target object on the GUI, the map creation schedule function through autonomous driving.

Hereinafter, a self-driving mobile robot using artificial intelligence planning technology according to the present invention and a smart indoor business management method using the same will be described in detail with reference to FIGS. 8 to 11.

As shown in FIG. 8, when the intelligent mobile robot 1000 requests the execution command through the GUI of the indoor work management app 2000a or proceeds to the autonomous driving mode, the agent management unit of the intelligent mobile robot 1000 operates. The 1100 inputs data of the vision sensor 200-1 and the distance sensor 200-2 mounted on the mobile robot, and prepares an indoor map for proceeding with the smart indoor business management service using the SLAM technology ( S100).

Referring to FIG. 9, when the position controller 1120 of the intelligent mobile robot 1000 performs autonomous driving, the wheel of the intelligent mobile robot 1000 is controlled using an odometry, and the An action of determining the position of the intelligent mobile robot is performed through dead-reckoning by measuring the amount of rotation (S110).

Subsequently, the map manager 1130 of the intelligent mobile robot 1000 scans the position estimate of the intelligent mobile robot 1000 calculated by the position controller 1120 and the XY plane with a distance camera to calculate the distance value. As an input, an action of creating an indoor map using the SLAM technique is executed (S120).

The location controller 1120 calculates the areas that can be moved and areas that cannot be moved based on information about occupied areas, free areas, and unknown areas on a map created in real time. An action of autonomous driving is performed while constructing cost map information and avoiding obstacles (S130).

As described above, the agent control unit 1110 of the intelligent mobile robot 1000 executes an execution command input by the user through the GUI of the indoor work management app 2000a in the intelligent mobile robot 1000 at all times. The object is classified into object actions and the movement action is executed by the AI planning technique in the position controller 1120 to move the intelligent mobile robot 1000 to the destination, and the object information collecting unit 1200 performs the object action. And grasping the state and delivers the result to the indoor work management app 2000a (S200).

In more detail with reference to Figure 10, the user inputs one or more of the text, voice, execution command on the indoor map provided by the indoor task management app (2000a) GUI, for example, 'a home appliance that is a specific object ( Refrigerators, washing machines, air conditioners, air cleaners), windows, boiler thermostats, move to door locks, "check the status of gas valve locks, opening and closing of windows, lighting on and off. Give a command (S210).

The indoor task management app manager 2100 converts an execution command and transmits the command to the agent controller 1110 of the intelligent mobile robot 1000 and based on the movement action distinguished by the agent controller 1110, the position controller 1120. To move the intelligent mobile robot 1000 while avoiding obstacles using Dijkstra Algorithm, A * algorithm, electromagnetic field algorithm, etc., based on the transmitted cost map information. To execute (S220).

As described above, the intelligent mobile robot 1000 detects an obstacle during autonomous driving in the non-operational state, so that the agent control unit 1110 of the intelligent mobile robot 1000 collects the object information. Object information is collected by transferring the information to the object or the agent controller 1110 executes an action for distinguishing the state of the object through the object action delivered to the object information collection unit 1200 in the constant operating state (S300).

In more detail with reference to FIG. 11, the subject is controlled by controlling the camera 200-3 installed in the intelligent mobile robot 1000 based on the obstacle information (the location of the obstacle and the distance between the obstacles) transmitted by the agent manager 1100. Shoot at a desired angle and performs an action (Action) to distinguish the type and state of the object through the object information collecting unit 1200 (S310).

The object information collecting unit 1200 recognizes the state value of the object object and the state of the general object through the object action distinguished by the agent controller 1110 in the execution command of the user. To execute (S320).

The agent manager 1100 displays the result of executing the execution command in the GUI of the indoor work management app 2000a of the user's smartphone as one or more of the object photographed image and the state value of the object (S330).

Although preferred embodiments of the present invention have been described as described above, the scope of the present invention is not limited thereto, and it is obvious that various changes and modifications can be made by those skilled in the art without departing from the following claims. All such modifications can be said to fall within the scope of the present invention.

100: gimbal 200: recognition sensor
1000: intelligent mobile robot 1100: agent management unit
1110: agent control unit 1120: location control unit
1121: cost map component 1122: obstacle avoidance
1130: map management unit 1200: object information collection unit
1210: Image input unit 1220: Image object recognition engine unit
1230: semantic object recognition unit 1231: object recognition unit
1232: person recognition unit 1233: event analysis unit
1300: communication control unit 1400: input and output control unit
1500: data storage 2000: user smartphone
2100: indoor business management app management unit 2200: ontology conversion unit
2300: input and output control unit 2400: communication control unit
2500: data storage unit 3000: wireless communication
4000: user interface 5000: user

Claims (15)

Self-driving intelligent mobile robots and apps using artificial intelligence planning technology, using the robots and apps to move robots to where they are needed in homes and businesses, take necessary images, recognize these images, As a smart indoor business management system to take necessary measures
In preparation for operation, the indoor map for the smart indoor business management service is created by using the SLAM (Simultaneous Localization And Mapping) technology by inputting the data of the vision sensor and the distance sensor mounted on the intelligent mobile robot, An agent manager to move the intelligent mobile robot from the current location to the destination without collision by using artificial intelligence planning technology in order to perform a command requested by the user to the indoor business management app;
An object information collection unit for directly receiving real-time image data input through a camera mounted on the intelligent mobile robot and recognizing objects and objects in the image and storing tagging of the recognition information; And
In order to deliver the user's command to the intelligent mobile robot, it is installed in the user's smartphone and receives one or more of text, voice, and execution commands through the GUI, and delivers the command to the intelligent mobile robot, which is performed by the intelligent mobile robot. It includes an indoor business management app that receives the result of the command and delivers it to the user,
The agent manager,
An agent controller for analyzing a command received from an indoor work management app installed in a user's smart phone, distinguishing it into a mobile action and an object action, transferring the mobile action to a location controller, and transferring the object action to an object information collection unit to perform a command;
Based on the movement action received from the agent controller, the intelligent mobile robot measures the movement amount of the intelligent mobile robot with the rotation amount of the wheel using Odometry so that the intelligent mobile robot can move to a specific position. A position control unit for calculating an position of the position and executing an action for moving to a destination commanded by the user; And
Create and update a map that distinguishes walls from obstacles using SLAM technology by inputting the position value of the intelligent mobile robot obtained from the position control unit and the distance value calculated by scanning the XY plane with a distance camera. A map manager that executes an action;
The position control unit
The area that can be moved by using information about occupied area, free area and unknown area on the map created using the current position, obstacle location, and SLAM technology of intelligent mobile robot. A cost map constructing unit configured to execute an action constituting cost map information for calculating an area that cannot be moved; And
Calculate the optimal route to the destination based on the cost map information and calculate the route to get to the destination quickly while avoiding obstacles that could collide with the intelligent robot by obtaining information about obstacles that change in real time through the distance sensor. An obstacle avoidance unit that executes the action,
The position of the intelligent mobile robot is measured by measuring the distance moved by dead-reckoning as the amount of rotation of the wheel axis and estimated by applying the inertial information measured by the inertial measurement unit. Smart indoor affairs management system, characterized in that it comprises an action (correction) to correct the error of navigation. delete delete delete The method of claim 1,
The object information collecting unit
An interface for receiving image streaming data requiring object information analysis, the interface comprising: an image input unit configured to receive image streaming data input from a camera of an intelligent mobile robot;
In order to recognize the objects included in the image input through the image input unit, one of an algorithm such as HMM (Hidden Markov Model), GMM (Gaussian Mixture Model), deep neural network, or long short-term memory (LSTM) is used. An image object recognition engine unit configured to form an edge on the objects (by performing an edge operation) to distinguish the objects and to execute an action of recognizing general objects; And
Among the objects for various objects and people included in the general object, the system recognizes only specific objects (specific objects (household appliances, gas valves, boiler thermostats, lighting), people and impressions) and separates them. And a semantic object recognition unit configured to perform tagging, storing tagging metadata, and executing an action for distinguishing meaningful objects from the stored objects. The method of claim 5,
The semantic object recognition unit
An action for distinguishing an object from the general object that includes at least a state of power on / off of a home appliance appearing in a building, an on / off state of a lighting device, a temperature and a state displayed on a screen of a temperature controller of a boiler ( A thing recognizing unit executing an action);
A person recognizing unit that distinguishes a person object from the general object such as an upper / lower color, an approximate height, wearing glasses / bags / hats, etc. as an impression on the person; And
Smart indoor work comprising an event analysis unit for performing an action (Action) for distinguishing the state change including at least the temperature change displayed on the screen of the boiler temperature controller, the power on / off of the light object in the object object Management system. The method of claim 1,
The indoor business management app
It provides a GUI that can provide the user and input and output information, receives one or more of user's input text, voice, and execution commands. The text and voice commands are transmitted to the ontology converter, and the executable commands extracted by the ontology converter are selected. An indoor task management app manager to transmit the execution command input to the GUI to the agent manager of the intelligent mobile robot, and receive the execution command execution result and display the GUI on the indoor task management app;
Using the semantic web technology by parsing one or more of the text and voice received from the indoor business management app management unit by applying a collection rule as input, generating and managing OWL data by applying a conversion rule, and converting it into knowledge triple of RDFS level. Smart indoor work management system, characterized in that it comprises an on-to-rotary conversion unit for converting the user's natural language into a command that can be processed by a computer. The method of claim 7, wherein
The indoor business management app management unit
Receives the location of the indoor object, the object located on the indoor map, and the location of the intelligent mobile robot from the agent management unit of the intelligent mobile robot, and displays it on the GUI of the indoor business management app. Smart indoor work management system, characterized in that configured to display in the GUI as one or more of the image, video. The method of claim 1,
The agent manager registers the motion related to the movement of the intelligent mobile robot and the motion related to the object collection as detailed actions in the domain plan, and when the user executes the command, The execution procedure of detailed actions is generated by one of moving action and object action through artificial intelligence planning technology, and automatically infers execution command by executing ROS detailed actions.
The mobile action and the object action has a smart indoor business management system, characterized in that the sequential execution order information of one or more actions (action) included in the domain plan. The method of claim 9,
Action is a smart indoor work management system, characterized in that included in the domain plan, including information on the preconditions for executing the action, the result status after the action is executed. The method of claim 9,
Execution command is declarerative and contains only the current state and destination state of intelligent mobile robot.How to get to the destination is executed by the agent controller to execute the mobile action planned and created to reach the final destination location. Smart indoor work management system characterized by. Self-driving intelligent mobile robot using artificial intelligence planning technology, using this robot to move the robot to the necessary place in the home and business, take the necessary image, recognize the image, and take necessary action according to the recognized result As a smart indoor business management method to take,
When the intelligent mobile robot is ready for operation, when the user requests execution command through the indoor task management app's GUI or proceeds to autonomous driving mode, the agent management unit of the intelligent mobile robot performs the vision sensor and the distance sensor of the mobile robot. A first step of preparing an indoor map for performing a smart indoor business management service through a SLAM technology with data input;
When the intelligent mobile robot is always in operation, the agent control command of the intelligent mobile robot is divided into moving action and object action by the user through the GUI of the indoor work management app. To move the intelligent mobile robot to the destination, and the object information collection unit executes the action by inputting the object action to grasp the type and state of the object and delivers the result to the indoor work management app. Step 2; And
When the intelligent mobile robot detects obstacles during autonomous driving, the agent control unit of the intelligent mobile robot sends the object action to the object information collection unit to collect the object information, and the object information collection unit executes the action to execute the object information. A third step of performing an action of discriminating a state of an object through an object action transmitted from the agent control unit to the object information collecting unit while the indoor map is created;
In the first step,
When the position controller of the intelligent mobile robot performs autonomous driving, it uses dead-reckoning to control the wheel of the intelligent mobile robot using odometry and measures the distance moved by measuring the amount of rotation of the wheel. Executing an action for determining the location of the intelligent mobile robot;
The map management unit of the intelligent mobile robot generates an indoor map using SLAM technology by inputting the calculated position value of the intelligent mobile robot calculated by the position controller and the XY plane by using a distance camera. Executing an action;
The cost control system calculates the areas of movement and areas that cannot be moved based on information about occupied areas, free areas, and unknown areas on a map created by the location controller in real time. Smart indoor work management method comprising the steps of constructing the cost map) information and executing the action (Action) to proceed autonomous driving while avoiding obstacles. delete The method of claim 12,
When the intelligent mobile robot is always in operation, the agent control command of the intelligent mobile robot is divided into movement action and object action by the user through the GUI of the indoor work management app, and the movement action is executed by the AI planning technology in the position control. In the step of moving the intelligent mobile robot to the destination, the object information collection unit performs the object action to identify the type and state of the object and deliver the result to the indoor work management app,
The user inputs one or more of the text, voice, and execution commands on the indoor map provided by the indoor task management app GUI to display a specific object such as a household appliance (refrigerator, washing machine, air conditioner, air purifier), window, boiler thermostat, door lock, etc. Giving a specific command among at least one command including at least one of moving to a place ',' checking a state of a gas valve locking device, a window opening and closing state, and an on / off state of a lighting device ';
The indoor task management app management unit converts the user's command into an execution command and transmits the command to the agent control unit of the intelligent mobile robot, and the mobile controller distinguishes the optimal path to the destination based on the cost map information. Executing an action of moving the intelligent mobile robot while avoiding obstacles using an algorithm (Dijkstra Algorithm), an A * algorithm, an electromagnetic field algorithm, and the like;
Smart indoor work management method, characterized in that configured to include. The method of claim 12,
When the intelligent mobile robot detects obstacles during autonomous driving in the operation ready state, the agent control unit of the intelligent mobile robot sends the object action to collect the object information to the object information collection unit to collect the object information, or the agent management unit in the constant operation state In the step of executing an action for distinguishing the state of the object through the object action passed to the information collector,
Action that captures the subject at a desired angle by controlling the camera installed in the intelligent mobile robot based on the obstacle information (the location of the obstacle and the distance between the obstacles) delivered by the agent manager and distinguishes the type and state of the object through the object information collection unit. Executing);
The object information collecting unit recognizes the state value of the object object in the impression of the person object and executes an action of distinguishing the state of the general object by the object action input distinguished by the agent control unit from the execution command of the user;
Displaying the result of executing the execution command by the agent manager as one or more of the object photographed image and the state value of the object in the GUI of the app of the user's smartphone;
Smart indoor work management method, characterized in that configured to include.

Images