KR100917377B1 - A central information processing system and method for service robot, with layered information structure according to recognition and reasoning level - Google Patents

Abstract

Centrally divides and stores information about indoor space and objects according to the level of recognition / inference and manages them centrally to provide space / object information to mobile service robots in the room according to the level of recognition / inference of each robot. An information processing system and method, comprising: at least one interface unit corresponding to the robot, wherein the interface unit interprets the information request of the corresponding robot, converts the requested information into a form that the robot can interpret, and transmits the information. Characterized in that the broker means; A spatial information manager configured to store and store the spatial information of the indoor space according to a recognition / inference level, and to retrieve and return the spatial information requested by the spatial information request of the broker means; An object information management unit configured to store and store objects existing in the room in layers according to recognition / inference levels, and retrieve and return object information requested by the object information request of the broker means; And a spatial object inference unit for searching and inferring lower layer information about an object or a space by using upper layer recognition / inference information.

By using the central information processing system and method of the service robot as described above, a vast amount of information about the space and objects in the room are hierarchically integrated and managed according to the recognition / inference level, and the service robots can meet the recognition / inference level. By providing the information, it is possible to provide highly compatible and efficient indoor information for various recognition / inference level service robots, and to support high level recognition / inference service even for low recognition / inference level service robots.

Indoor, Robot, Cognition, Reasoning, Ontology

Description

A central information processing system and method for service robot, with layered information structure according to recognition and reasoning level}

The present invention hierarchically separates and integrates information about space and objects in a room according to the level of recognition / inference to provide space / object information that is compatible with the level of recognition / inference of each robot to service robots that are movable indoors. Belongs to the technical field of the central information processing system and method.

In general, a mobile service robot performs a variety of tasks such as recognition of an object or a situation, navigation to a target point, holding a designated object, and storing a DB. do. In addition to the basic functions such as recognition, driving, and manipulation, mobile service robots have advanced technologies such as self-localization and map building (SLAM), self-modeling, and category recognition (Categorization). You should also be able to perform these necessary tasks. Currently, research on each function of such a mobile service robot is actively conducted.

However, if we simply combine the individual development results for each function to create one mobile service robot, since the robots have different data formats or storage for each function, the compatibility between modules for each function is reduced. As a result, there will be considerable overlap in terms of information operability, with each having similar information. For example, when a mobile service robot is ordered to fetch green tea from a refrigerator located in the kitchen, the robot performs a series of tasks such as driving, recognition and manipulation. If the green tea information for recognition and the green tea information for manipulation exist separately, and the refrigerator for driving and the refrigerator information for recognition are scattered separately, redundancy occurs because the information is not integratedly managed. If this waste persists, inefficiencies in terms of storage space and compatibility are increasing. In addition, if one robot already has sensing or perception information on the refrigerator, green tea, and kitchen, but another service robot acquires the same information through the same sensing / perception process, the space / It causes waste of time duplication.

In addition, if the service robot is developed and configured independently according to each function, the service that the service robot can perform will of course be limited and low level. For example, even if a mobile service robot is equipped with advanced reasoning techniques such as self-location recognition and map generation, and has excellent driving ability, there is no high level of inference or context information about space or objects. The cognitive ability to acquire knowledge and understand relationships between and objects will be low. The robot can then use tasks based on an understanding of space and objects within the indoor environment, such as bringing green tea beverages from the refrigerator in the home environment, and juice from the kitchen table to the sofa in the living room. You will not be able to carry out the work that you bring to the sitting owner. In order to perform these high-level tasks and services, a knowledge system and structure of object / space is required separately from the ability to drive accurate maps. This means understanding the robot's space or objects.

Moreover, in an indoor environment in which a plurality of service robots are operated instead of one service robot, information sharing on spaces or objects is more urgent. That is, information acquired by one service robot should be shared by other robots in the same environment, and its contents should be reflected in the service of another robot. In particular, when robots with different levels of recognition or reasoning exist, information needs to be processed, transformed, and shared for each level of recognition / inference.

Therefore, research on a system for integrating, managing, and sharing information required by various robots is needed, and such research is being conducted in part.

A representative method of effectively representing and managing geographic and spatial information is the Geographic Information System (GIS). GIS is one of the technologies that help users make intelligent decisions. GIS is a system that maps data having a property on Geographical Information to a multi-layer, and usually expresses only information that is suitable for a purpose in detail. General GIS data is limited to vast outdoor and ground areas, making it unsuitable for service robots for precise manipulation in home environments. In addition, indoor environment modeling using computer aided design (CAD) is made as many as there are many commercial products, and standardization is being attempted through IFC (Industrial Foundation Classes). However, CAD also provides only geometric building structure information, which does not meet the requirements of real home robots.

Attempts to integrate and manage information about individuals who are serviced by home service robots have been actively made in the web service field. The main purpose is to observe human behavior patterns, to construct the observed data with ontology, and to infer the constructed ontology to generate new user information. This approach is expected to help robots more easily determine people's needs, rather than helping them perform a given mission.

The Information System Approach on Physical Data or Raw Data is often used to collect data that is limited to the characteristics of the Sensor used, or to obtain raw data processed at a lower level. It is used in the field. Most of the information thus obtained is problematic in that reusability is insufficient to be used as information for a service robot or cannot be used in connection with abstract knowledge.

Among the researches related to human speech from the perspective of human-robot interaction (HRI), the Aurora Project is an attempt to link source data with high-dimensional abstract knowledge. The Aurora Project puts the vocabulary of the human voice on a central server, sharing the server for analysis and error handling of preprocessed data. However, the original intent of Distributed Sound Recognition is to reduce bottlenecks caused by the lack of Terminal's Computational Resources.

Orca is a software component sharing system that does not share data about robots. Orca aims to simplify the interface commonly used between each component of the robot and make it easier to use, aiming for effective and continuous reuse of the component. Orca is implemented not only through the design of software components, but also through the effective management of component repositories. Orca, however, is a middleware approach that combines the connection and repository of components into a framework.

In addition, while a study discloses a method of expressing and storing spatial information and object information recognized in quadruped robot soccer, information exchange with other objects is limited to task sharing through tokens. There is a disadvantage [Refer 1].

In addition, other studies have disclosed task-oriented information generation and robot learning. Although robots that need to operate in a normal home or office environment for each task consider a wide variety of environments and movements, it is practically difficult for the robots to fully build up prior knowledge of these environments. Therefore, this study proposes a new teaching framework called Task Model-Based Interactive Teaching [Ref. 2].

In addition, there are various attempts to organize data / information / knowledge into robots / automation devices, but information systems that meet the requirements of service robots must be newly developed.

[1] 'Assignment of dynamically perceived tasks by token passing in multirobot systems, Proceedings of the IEEE, Special issue on Multi-Robot Systems, 2006' by Farinelli, A. Iocchi, L. Nardi, D and Ziparo, V.A.

[2] Jun Miura, Koji Iwase and Yoshiaki Shirari's' Interactive Teaching of a Mobile Robot, In Proceeding of 2005 Int. Conf.on Robotic and Automation, pp. 3389-3394, 2005 '

An object of the present invention is to solve the problems described above, by providing the indoor information necessary for the mobile service robot indoors and by receiving the changed information from the service robot, a plurality of indoor service robots to share the indoor information It is to provide a central information processing system and method of a service robot that integrates and manages information about space and objects in the room.

Another object of the present invention is to hierarchically separate and integrate information about space and objects in a room according to the level of recognition / inference, thereby providing space / object information to mobile service robots in the room according to the level of recognition / inference of each robot. It is to provide a central information processing system and method for providing.

It is still another object of the present invention to provide a central information processing system and method for retrieving information and inferring information using lower layer recognition / inference information from lower layer information on an object or space that a service robot recognizes / infers. will be.

In order to achieve the above object, the present invention relates to a central information processing system for communicating with at least one mobile service robot indoors, including at least one interface unit corresponding to the robot, wherein the interface unit requests information of a corresponding robot. Broker means, characterized in that for converting and transmitting the requested information in a form that the robot can interpret; A spatial information manager configured to store and store the spatial information of the indoor space according to a recognition / inference level, and to retrieve and return the spatial information requested by the spatial information request of the broker means; An object information management unit configured to store and store objects existing in the room in layers according to recognition / inference levels, and retrieve and return object information requested by the object information request of the broker means; And a spatial object inference unit for searching for and inferring information using upper layer recognition / inference information from lower layer recognition / inference information on an object or space.

In addition, the central information processing system of the service robot according to the present invention, the spatial information or object information, the measurement information layer for storing the measured information, the photometric information layer for storing the photometric feature (space or object) geometric A geometric information layer that is segmented and stored as a geometric structure, a structural information layer that distinguishes, symbolizes and stratifies and stores components of a space or an object, and topological information that stores information representing an object's position or obstacle in space A hierarchy, space or object, or a universal information layer that defines and stores a universal model of the space or its components, or a space-time information layer that stores changes and relationships over time or space in an object or space From the hierarchical information the space or object, or the state of the space or object It characterized in that it consists of a reasoning information layer for storing the information used to infer theory.

In addition, the central information processing system of the service robot according to the present invention, the measured information layer of the space includes the basic map information for storing the geographic information of the indoor space, the photometric information layer of the space is characterized by the photometer characteristics of the indoor space map The geometric information layer of the space includes geometrically segmented information of the interior space, and the topological information layer of the space includes information about an object that is recognized as an object's global coordinates and attitude, or an obstacle. The universal information layer of the space includes semantic information about the space or a part of the space, and the space-time information layer of the space includes light brightness, direction of light, temperature, and humidity depending on the temporal and spatial situation of the room. Characterized in that it comprises a.

In addition, in the central information processing system of the service robot according to the present invention, the measured information layer of the object includes 2D / 3D original image information for storing two-dimensional or three-dimensional representation data obtained by actual measurement of the object, and the object The topological layer of information includes a plurality of three-dimensional contact points and information in an accessible direction represented by three-dimensional direction vectors, and the space-time information layer of the object is a time-dependent relationship of object-space. And the existence and space-time dependence of objects, space-time dependence between objects and objects, space-time dependence between objects and functions, and space-time dependence of object-physical characteristics. It is characterized by including an evidence structure that stores a graph that builds the relationship with probability.

In addition, in the central information processing system of the service robot according to the present invention, the photometric characteristics include information about a scale invariant feature transform (SIFT), a Harris corner, a color, and a line. It features.

In addition, the central information processing system of the service robot according to the present invention is characterized in that the obstacle information is represented and stored in a mesh (mesh) or volume.

In addition, in the central information processing system of the service robot according to the present invention, the two-dimensional or three-dimensional representation data is a 2D image (Raw Image), a laser scanner, a stereo camera, a structured light camera lighting calculated through at least two camera angles. And at least one of depth data obtained from any one of the structured light cameras and mesh data generated from the depth data.

In the central information processing system of the service robot according to the present invention, the spatial information management unit or the object information management unit configures and stores the spatial information or object information as an ontology, and the spatial object inference unit infers by ontology. It is done.

In addition, in the central information processing system of the service robot according to the present invention, when the interface unit requests a map from the corresponding robot, the interface unit may display basic map information of the actual information layer of the space and object / obstacle position information of the topological information layer. The combined map is prepared and provided to the robot.

In addition, the central information processing system of the service robot according to the present invention creates and stores a probability-based behavior structure that determines the unit behavior to be executed by the robot from the mission given to the robot, and returns the behavior structure at the request of the broker means. The method may further include a mission analysis manager that analyzes the mission given to the robot and extracts and returns the unit action to be executed by the robot and information necessary for the action.

In addition, in the central information processing system of the service robot according to the present invention, the mission analysis management unit is a Bayesian having a probabilistic priority relation of perception data, evidence structure, and behavior. The behavioral structure is composed of a network (Bayesian network), the optimal path is determined by the probability of the precedence relation, and the behavior on the path is extracted as a unit action to be executed. It is done.

In addition, the present invention relates to an information processing method of a service robot using a central information processing system that communicates with at least one mobile service robot in the room, (a) the spatial and object information for the room according to the recognition / reasoning level Organizing and storing each layer; (b) for each of the robots, constructing an interface for interpreting the information request of the robot and converting the requested information into a form that the robot can interpret; (c) receiving an information search request for a space or an object of the robot; (e) determining if inference is needed to retrieve the information, and if necessary, inferring the information using higher layer recognition / inference information from the given information about the object or space; (f) retrieving information about the requested space or object; (g) constructing an integrated map if the requested information requires space and object information at the same time; (h) transmitting the retrieved space or object information to the requested robot.

In the central information processing method of the service robot according to the present invention, the spatial information or the object information may include a measurement information layer storing measured information, a photometric information layer storing photometric features, a space or an object. A geometric information layer that is segmented and stored as a geometric structure, a structural information layer that distinguishes, symbolizes and stratifies and stores components of a space or an object, and topological information that stores information representing an object's position or obstacle in space A universal information layer that defines and stores a hierarchy, space or object, or a universal model of the space or its components, or a space-time information layer that stores changes and relationships over time or space in an object or space, or a sublayer for an object or space Infer the space or object, or the state of the space or object from the information It is characterized in that it comprises a reasoning information layer for storing the reasoning information for.

In addition, the central information processing method of the service robot according to the present invention, the measurement information layer of the space includes the basic map information for storing the geographic information of the indoor space, the photometric information layer of the space is characterized by the photometer characteristics of the indoor space map The geometrical information layer of the space includes geometrically segmented information of the interior space, and the topological information layer of the space includes information about an object recognized as an object's global coordinates and attitude, or an obstacle. The universal information layer of the space includes semantic information about the space or a part of the space, and the space-time information layer of the space includes light brightness, direction of light, temperature, and humidity depending on the temporal and spatial situation of the room. Characterized in that it comprises a.

In addition, the central information processing method of the service robot according to the present invention, the measured information layer of the object includes 2D / 3D original image information for storing two-dimensional or three-dimensional representation data obtained by the actual measurement of the object, the object The topological layer of information includes a plurality of three-dimensional contact points and information in an accessible direction represented by three-dimensional direction vectors, and the space-time information layer of the object is a time-dependent relationship of object-space. And the existence and space-time dependence of objects, space-time dependence between objects and objects, space-time dependence between objects and functions, and space-time dependence of object-physical characteristics. Evidence structure for storing a graph that creates the relationship with probability.

In addition, the central information processing method of the service robot according to the present invention is that the photometer features include information about the Scale Invariant Feature Transform (SIFT), Harris Corner (Color), Color (Line) It features.

In addition, the central information processing method of the service robot according to the present invention is characterized in that the obstacle information is represented and stored in a mesh (mesh) or volume.

In the central information processing method of the service robot according to the present invention, the two-dimensional or three-dimensional representation data is a 2D image (Raw Image), a laser scanner, a stereo camera, a structured light camera calculated through at least two camera angles. And at least one of depth data obtained from one of the structured light cameras, and mesh data generated from the depth data.

In the central information processing method of the service robot according to the present invention, the step (a) is configured to store the spatial information or object information as an ontology, and the step (e) is inferred by the ontology. .

In the central information processing method of the service robot according to the present invention, the step (a) is configured to store the spatial information or object information as an ontology, and the step (e) is inferred by the ontology. .

In the central information processing method of the service robot according to the present invention, if the interface unit requests a map from the corresponding robot, the base map information of the actual information layer of the space and the object / obstacle position information of the topological information layer are provided. It is characterized in that the integrated map is created and provided to the robot.

In addition, the central information processing method of the service robot according to the present invention comprises the steps of: (a0) creating and storing a probability-based behavior structure that determines the unit behavior to be executed by the robot from the mission given to the robot before step (a); (d) after step (c), returning the action structure by the search request of the robot, or analyzing the mission given to the robot, and extracting and returning unit action to be executed by the robot and information necessary for the action. Characterized in that.

In the central information processing method of the service robot according to the present invention, in step (a0), a bay having a probabilistic priority relation of perception data, evidence structure, and behavior may be used. The behavioral structure is composed of a Bayesian network, and step (d) determines an optimal path based on the probability of the precedence relation to execute an action on the path. It is characterized in that the extraction as a unit action.

The present invention also relates to a computer-readable recording medium recording the information processing method of the Vis robot.

As described above, according to the central information processing system and method of the service robot according to the present invention, by comprehensively managing and providing the information necessary for the mobile service robot to comprehensively share, providing a high efficiency of data management and room with a large number of robots In this situation, the effect of reducing the space cost for knowledge storage and the time cost for learning space / object is obtained.

In addition, according to the central information processing system and method of the service robot according to the present invention, by providing information corresponding to the recognition / inference level of the mobile service robot, information sharing and integration even in an environment where heterogeneous recognition / inference level robots exist A manageable effect is obtained.

In addition, according to the central information processing system and method of the service robot according to the present invention, by providing the information inferred at a high level even to the service robot of low recognition / inference level, it is possible to The effect of receiving the service corresponding to the level is obtained.

DETAILED DESCRIPTION Hereinafter, specific contents for carrying out the present invention will be described with reference to the drawings.

In addition, in describing this invention, the same code | symbol is attached | subjected and the repeated description is abbreviate | omitted.

1 is a view showing the configuration of an entire system for implementing a central information processing system and method of a service robot according to the present invention.

As shown in FIG. 1, the central information processing system 30 of the service robot according to the present invention includes a database 20, and communicates with a plurality of mobile service robots 10 to transmit and receive information.

Communication between the information processing system 30 and the mobile service robot 10 exchanges information with each other by wireless communication. The wireless communication may be any wireless communication such as wireless LAN, RF (Radio Frequency), Bluetooth (bluetooth). But at least it should be a wireless communication standard that can cover the indoor space range. For example, IrDA may not be suitable because its cover range is small. However, if the building is large or the coverage of the wireless communication is small, it may be possible to enable wireless communication anywhere in the room through a repeater. In addition, applications such as broadband wireless communication such as WiBro, HSDPA, Wi-Fi, and WiMAX will be possible. Since the wireless communication technologies are well known in the art, detailed description thereof will be omitted.

The database 20 may be a general relational database, but according to the present invention, when the object / spatial information is configured as an ontology, it may mean an ontology repository. That is, the database 20 is a known database appropriate for the configuration form of the data of the present invention. It is also equipped with a known search engine for managing and searching the database 20. For example, if it consists of an ontology, it means that it is an ontology repository in which middleware is installed, such as an engine capable of ontology inference.

The mobile service robot 10 refers to an intelligent service robot that can move indoors and perform commands. The robot includes sensors such as an ultrasonic sensor, an infrared sensor, a laser sensor, a general camera, a laser scanner, a stereo camera, and a microphone for recognizing an object or a space. In addition, the robot 10 may be provided with a function of recognizing or inferring such as recognizing an object or recognizing a magnetic position by analyzing measured values or images read from such a sensor. In addition, the robot 10 performs recognition, navigation, manipulation, and autonomous object / environment modeling. In addition, having a space or an object or semantic information therebetween may also have a function to perform semantic reasoning. That is, the mobile service robots 10 may have different recognition or inference levels.

The mobile service robot 10 requests information necessary for the central information processing system 30 through communication, and receives and uses the requested information. In this case, the data stored in the central information processing system 30 may be simply requested, or the data processed internally by the central information processing system 30 may be received. In addition, the robot 10 requests the central information processing system 30 for the data desired by the robot 10 itself, and the central information processing system 30 analyzes and retrieves the data and provides appropriate data. The information updated by the administrator of the central information processing system or the robot 10 is again provided to other robots through the central information processing system 30.

2 is a block diagram of a configuration of a central information processing system of a service robot according to an embodiment of the present invention. As shown in FIG. 2, the central information processing system 30 of the service robot includes a database 20, and the broker means 31, the spatial information management unit 33, the object information management unit 34, and spatial object inference. It consists of a part 35. In addition, the mission analysis management unit 36 may be further configured.

The broker means 31 includes at least one interface unit 32 corresponding to the service robot 10, the interface unit 32 interprets the information request of the corresponding robot, and the robot requests the requested information. Transmit it into an interpretable form.

That is, the broker means 31 is composed of interface units 32 one-to-one corresponding to the service robots 10 connected by communication. If the same type of service robots may be configured to connect to one interface unit 32 to process. However, in this case, the embodiment should be configured such that one interface unit 32 has a function of multiprocessing capable of simultaneously performing a plurality of tasks.

The interface unit 32 analyzes the information requests of the corresponding service robots 10 and converts the information requests into requests that can be processed by the central information processing system 30, and conversely, the central information processing system 30. Performs the task of converting the request result generated in response to the information request into a form that the service robot can interpret. That is, the interface unit 32 converts the information so that the service robot 10 may be compatible with each other even though the service robot 10 has a different information processing system from the central information processing system 30. In the future, when robots emerge as a universal means for people's daily life, intelligent service robots of different specifications can be made by different companies, and users will purchase different service robots and use them in the same room at the same time. Therefore, by simply making the interface unit 32 for the service robot 10 of such different specifications, it is possible to exchange information in conjunction with the central information processing system 30.

The spatial information manager 33 and the object information manager 34 configure and store information about the space or an object in each layer according to a recognition / inference level. Referring to FIG. 3, the configuration of the information layer according to the recognition / inference level will be described. 3 is a block diagram of an information layer of a central information processing system according to an embodiment of the present invention.

As shown in FIG. 3, the information layer 40 of the central information processing system includes a measurement information layer 41, a photometric information layer 42, a geometric information layer 43, and position / configuration information from a lower layer to an upper layer. The layer 44, the functional information layer 45, the spatiotemporal information layer 46, and the inference information layer 47. The information layer 40 is a layered information according to the recognition / reasoning level of the robot. That is, the lower information layer is information obtained by primitive recognition or inference of a space or an object through a sensor or the like, and the upper information layer is information capable of high-order inference as semantic information of the space or object.

The measured information layer 41 is an information layer that stores measured information. That is, information on the size or shape of the measured object. The measured information about the space serves as a basic map such as architectural blueprints, and includes measurements of the three-dimensional solid shape and the measured size of the space. The actual measurement information about the object corresponds to a 2D / 3D image of the object and an intrinsic / extrinsic camera parameter. The camera parameter is used for error correction of an image as data such as a camera position, a lens size, and the like.

The photometer information layer 42 is an information layer that stores photometric features. Photometric information is a mathematical analysis of photometric data such as brightness, hue & saturation, and texture of 2D / 3D points in an image as a set or geometry of specific values. It is expressed. Since the object recognition of the robot is mainly through the camera for the object exposed by the light, this photometric characteristic plays an important role in the object recognition. Examples of photometric information include color, Scale Invariant Feature Transform (SIFT), Harris Corner, and Line.

The geometric information layer 43 is an information layer that stores a space or an object by dividing it into a geometric structure. That is, the information may mean information that geometrically expresses the shape of the object in a narrow sense, but also includes information that is stored in a structural form in a broader sense. For example, if a cup with a handle is recognized as a camera, the handle is also part of the shape of the whole cup, but is a part that can be segmented into a structural form. In the case of a refrigerator, the parts that can be segmented are an upper door, a lower door, a freezer, a refrigerator, a drawer, and a handle. In space, windows, doors, ceilings, columns, corners, corners, etc. are objects that can be structurally segmented.

The structural information layer 44 is an information layer that distinguishes elements of a space or an object and stores them symbolically and hierarchically. The structural information is symbolic and hierarchical information about elements belonging to the object. One building consists of several floors, each of which consists of a corridor or a room. Each object consists of the components that make up the object. In contrast, one room is located on a certain floor and one floor is located on a building. That is, the structural information is information describing a spatial relationship between objects in terms of the object. The structural information should also be managed in connection with the geometric segments of the lower layers. However, the geometrical segment of the lower layer is segmented by its features, whereas the composition of the object is distinguished by its functional aspects. However, since the object is classified based on the mechanical segment when the object is divided into components, there is an advantage that the robot can recognize the object step by step. That is, the robot can recognize the "door", which is a component of space, as a "square plate" that is geometrically segmented in space. The opposite is also possible. For example, the components of a specific apartment are divided into functional characteristics such as living rooms, bedrooms, kitchens, and toilets in association with sub-layer information below the structural information layer 44, and the symbolic representations thereof are symbolic. This is the case. Also in the object, the refrigerator is classified hierarchically into the upper door, the lower door, the freezer compartment, the refrigerating compartment, the drawer, the handle, and the like, and is symbolically represented.

The topological information layer 45 is an information layer that stores information representing a position or an obstacle of an object in a specific space. Topological information is information representing the global position and posture of an object or the position and volume of an obstacle in a specific space. The topological information is information representing accessibility for gripping an object. For example, if an object in a particular kitchen includes a refrigerator, a microwave, a dishwasher, a cup, a plate, a cutlery, a spoon, and the like, the topological information of the objects is information about a global position, a posture, and the like in the kitchen. The obstacle information is information for representing an unknown object, and is information about the obstacle represented by a mesh or a volume.

The universal information layer 46 is an information layer that defines and stores a universal function of a space or an object or its components. The universal model is a model that expresses and stores the semantic definitions of functions, sub-parts, and geometries of spaces or objects and their relationships. For example, the function of the "door", which is a component of the space, is information about a function such as "a passage that can enter or exit the space" or "can open and close". In addition, "cup" is information such as "tool used for drinking a drink such as water", and "handle" of the cup is information such as "component used when lifting a cup" and "component holding a cup". The segment is to divide the refrigerator into an upper door, a lower door, a freezer, a refrigerator, a drawer, a handle, and the like in the object as in the geometric information layer 43, and in the space, it is divided into a window, a door, a ceiling, a pillar, a corner, a corner, and the like. . The difference between the segment referred to in the universal information layer 46 and the segment referred to in the geometric information layer 43 is the information obtained by segmenting an instance based on the data measured by the geometric information layer 43, while the universal information layer 46 is used. Universally defines the segmentable nature of an object or space. Geometry refers to points, lines, faces, ellipses, spheres, boxes, and so on, and generally defines the geometry characteristics that an object or space should have. In addition, the universal component of the apartment in the space is divided into a living room, bedroom, kitchen, toilet, and the like is also applicable to this. The lower information layer of the universal information layer 46 is information about an external structure or configuration of a specific object, while the upper layers from the universal information layer 46 are mainly about semantic information. That is, the classification of the configuration of the lower layer is the external classification by the function, while the universal information layer 46 is the semantic information about the configuration.

The space-time information layer 47 is an information layer that stores changes and relationships according to space-time of an object or space. Spatio-temporal information is semantic information about the relationship between objects. In other respects, it is also semantic information about the change of the object caused by the change of the relative. Related to the object can be time, environment, other objects, and the like. That is, the relationship with time (or the change of the object according to the change of time), the relationship with the environment (or the change of the object due to the change of the environment), the relationship with the other object (or the change of the object due to the change of another object) ) And the like. Information about the brightness, direction of light, temperature, humidity, etc. of indoor space or objects depending on temporal and spatial conditions. In addition, information on the change in position of an object over time or the influence of the existence of another object when a specific object exists in a specific space-time.

The inference information layer 48 is an information layer that stores inference information for inferring a space or an object or a state of the space or an object from a situation that is directly measured. Inference information is information about an inference structure constructed using information of lower layers. For example, an evidence structure in which a graph based on a probability for recognizing an object is prepared. Inference can also be made in the lower information layer of the inference information layer 48. For example, since the universal information layer and the space-time information layer are semantic information, semantic inference is naturally possible. In addition, the information about the external structure of the lower layer may be inferred through methods of searching for an object to which a specific attribute (or characteristic) matches. The inference information layer 48 is different from the inference of the lower layer in that it is information for inferring more professionally based on probability.

In total, upper information layers are constructed based on lower information layers. In addition, although each information layer can be implemented separately, all or part thereof can be implemented as a single data structure. For example, all hierarchical information may be organized by ontology. For example, a cup or a class called instance may be created to define all attributes from the measured information layer to the topological information layer, and the information on the universal information layer may also be defined as attributes of the class.

The characteristic part of the information on the information layer for the spatial information management unit 33 and the object information management unit 34 will be described in more detail.

The spatial information manager 33 configures and stores the spatial information of the room according to a recognition / inference level, and searches for and returns the spatial information requested by the spatial information request of the broker means. As shown in FIG. 4, the spatial information includes information on the eight information layers described above.

The actual information layer of the space is a part that stores the geographic information of the indoor space like the GIS and is the most basic information, which is a base map that serves as an architectural design. The base map includes a CAD form (ifcXML and dxf) according to the IFC 2xx protocol, which is a two-dimensional grid map that schematically shows the structural information of the indoor environment, basic building design rules, and international building design interchange document standard. A three-dimensional map created in a similar format). The base map created with respect to the actual map of FIG. 5A is shown in FIG. 5B. As shown in FIGS. 5A and 5B, the reference coordinates of the base map containing basic environmental information use the rectangular coordinates of XY as the lowest left origin of the horizontal plane, and the vertical component is represented by the Z axis. . In addition, the implementation method using the DFX format of CAD for sharing shape information is shown.

The photometric information layer of the space stores a photometric feature map of the indoor space. A photometric feature map represents an image processing result that can be obtained using only brightness and color values from the entire image area such as SIFT, Harris Corner, and Color values. 6A is a diagram illustrating a photometer feature (SIFT) extracted in an indoor environment according to an embodiment of the present invention, and FIG. 6B is a diagram of a photometer feature (SIFT) extracted in an indoor environment according to an embodiment of the present invention. It is the figure expressed on the dimension integrated map.

Information stored in the geometrical information layer of the space is a geometric environment structure (Geometric Environment Structure), and the indoor environment is geometrically segmented and stored. Examples include environmental objects such as doors, windows, walls, columns, and corners, sides, and faces that are global geometric features. It also expresses their structural relationships. These information can be used for manipulating environmental objects such as robot 10 to recognize its own position or to open and close the door while driving.

As described above, the structural information layer of a space is information that distinguishes elements of the space and stores them symbolically and hierarchically. Structural information about a building consists of several floors, each floor consisting of a corridor or a room, and each component symbolically displays. The information is stored in association with a lower layer containing geometric / photometric feature / actual information. For example, the structural information of a specific apartment may be divided into geometric and photometric features and measured space components of the lower part of the subpart into functional characteristics such as living rooms, bedrooms, kitchens, toilets, and the like. The information is indicated by.

The topological information layer in space is an information layer about the positional information of objects and obstacles. The position and attitude of the movable objects used for manipulation are represented. This layer does not actually store the object model, but represents only the global coordinates and poses of the object. An unknown object that does not have any information about the object in the object DB is regarded as an obstacle, and the obstacle is represented and stored as a mesh or a volume (eg, Octree). FIG. 7A is an exemplary diagram illustrating an obstacle according to an embodiment of the present invention in Octree, which is a volumetric representation method, and FIG. 7B is a diagram illustrating an indoor space and an object according to an embodiment of the present invention. Object Modeling Reference Frame is the object reference coordinate axis where the horizontal component is the X axis, the vertical component is the Y axis, and the depth component to the speaker is the Z axis. Frame). Coordinates of Objects stored / managed by the space manager 33 move reference coordinates for object modeling to the lowest point in the height direction. This provides a more intuitive placement and coordinate movement of the object. Orientation information for objects stored / managed by the space manager 33 is a unit vector for X, Y, and Z of world coordinates for each of U, V, and W, which are reference axes of the object. (Unit Vector) or rotation angle.

On the other hand, when the interface unit 32 requests a map from a corresponding robot, the interface unit 32 generates a united map by combining the information of the basic map information layer of the spatial information and the object / obstacle position information layer and the robot ( 10) to provide. 9 is a diagram illustrating an ER table of an integrated map according to an embodiment of the present invention.

In the universal information layer of space, a universal space model is stored. The universal space model expresses semantic spaces such as living room, bedroom, bathroom, utility room and toilet. In addition, semantic information about the doors, windows, etc., which are components of the space, are also expressed. This is used by the robot 10 to locate the robot 10 and to semantically separate the areas.

The space-time information layer of space stores environmental information. Environmental information refers to the brightness of the light, the direction of the light (including natural light), temperature, humidity, etc., depending on the temporal or spatial situation in the indoor space. These informations act as factors that additionally influence the robot 10 to perform services such as recognition / modeling.

The inference information layer of space is information about the evidence structure in the inference information layer 48 described above. 8 is a diagram illustrating an evidence structure in the inference information layer of a space according to an embodiment of the present invention. As shown in FIG. 8, the function of the kitchen is generally a dish and a conversation or a wine bar. In the kitchen, there is a gas stove, a sink, a refrigerator and a table. If all of these conditions are met, then the probability of any random space being a kitchen is close to 1, and if it is more than a certain amount, we assume that any random space is a kitchen. This information is stored in spatial evidence structures.

The object information manager 34 configures and stores objects existing in the room according to a recognition / inference level and searches for and returns object information requested by the object information request of the broker. As shown in Figure 4, the object information is composed of a total of eight information layers and stored.

The measured information layer of the object is information about a raw image, and stores actual measured 2D / 3D raw data about an object obtained from a 2D / 3D camera. That is, 2D data calculated through various camera angles, and includes depth data obtained from a raw image, a laser scanner, a stereo camera, or a structured light camera.

The photometric information layer of the object is information about the photometric feature of the object, and the color, scale invariant feature transform (SIFT), Harris corner, line, etc., which are obtained from each original image Save the information. This acts as evidence in the recognition process.

The geometric information layer of an object is information about a geometric feature, and stores geometric information of the object in a structural form. These information are obtained from the original image. This is not a method of generating a model for the recognition of the rigid body center, but an object feature representation for understanding and connecting higher level information and a modeling method for an object having a lower part or a segment It includes a method. For example, in an object, a refrigerator is divided into an upper door, a lower door, a freezer, a refrigerator, a drawer, a handle, and the like, and a space is divided into a window, a door, a ceiling, a pillar, a corner, a corner, and the like. 11 shows an embodiment of a refrigerator of the present invention.

The data structure used in the geometric information layer of an object can be easily borrowed from a CAD format such as x3d, dfx, 3ds, wrl, or obj, or configured as a separate storage method. However, it is not possible to express all the structural information in the CAD format, so it is necessary to develop a separate format. In the experiment of the present invention, the data format of the CAD model is used for convenience. FIG. 10A is a diagram illustrating a result of modeling a pencil, which is a three-dimensional object, according to an embodiment of the present invention as an x3d file, which is one of CAD forms, and FIG. 10B is an x3d file of FIG. 10A, according to an embodiment of the present invention. The code modeled below is shown.

The structural information layer of an object is information about the composition of the object. That is, information describing the elements constituting the object. And this information is managed in connection with the lower layer which contains geometric / photometric feature information. For example, the cup has components of the body and the handle. The component corresponds to a segment of the cup. Therefore, it needs to be linked with the segmentation information of the lower layer which contains geometric / photometric characteristic information. 11 is a diagram illustrating object configuration information in a structural information layer according to an embodiment of the present invention. As shown in FIG. 11, the refrigerator may include a body, an upper / lower door, a freezer, a refrigerator, a shelf, and the like. The upper left is the body of the refrigerator and the lower left is the refrigerator door. And the figure which combined these is the figure of the right side in FIG.

The topological information layer of an object contains information that expresses accessibility for the gripping of the object. The object accessibility is a kind of object property, which is information related in which direction the object should be approached and held. 12A is a diagram illustrating an approach direction of an object in a topological layer of an object according to an embodiment of the present invention, and FIG. 12B is a view of an object in a topological layer of an object according to an embodiment of the present invention. A diagram illustrating a contact point. As shown in FIG. 12, this information consists of a number of three-dimensional contact points that can pick up objects and an accessible direction value represented by a three-dimensional direction vector.

The universal information layer of an object is an information layer that stores a generic object model. For a specific model (Specific Model or Instance) in a specific object class, it collects only common geometric features at the common sense level. Information is expressed in an abstract structure. For example, the general characteristics of the cup can be summarized in terms of functions such as preserving the drink, drinking the drink, and holding the cup, as well as a concave space of a suitable volume to hold water, a handle to hold the cup, and the shape of the cup. It can be seen that it can be structured as a cylinder-shaped body, an ellipse representing the upper end of the body of the cup, a convex surface and a concave surface expressing the interior and exterior characteristics of the cup. This information is used when modeling and recognizing an object when it does not have a specific model of the object, that is, its own instance model, that is, a category is known but does not have an accurate model, or This method is useful for suggesting the best segmentation method for recognizing and inferring the function of each segment by recognizing the segment of the object, and for finding the effective recognition process according to the object. 13 is a diagram illustrating a universal model of a cup according to an embodiment of the present invention. As shown in FIG. 13, the universal model of the cup represents the function of the cup at level 1, the cup is segmented at level 2, and the level is represented by the most similar geometry to estimate each segment. This is possible.

The spatiotemporal information layer of an object relates to context spatiotemporal information. The context includes the time-dependency of the object-space, which represents how the object's position in space changes over time, the space-time dependency between the object-object, and how the object is affected by its existence in space-time. Space-time dependence of the existence of an object that expresses its effect on the existence of another object when it exists in a specific space-time, space-time dependency of an object-function expressing how an object's function can change from space-time, Or the spatiotemporal dependence of object-physical properties to express what physical changes (shape, color, texture, transparency, etc.) vary depending on the environmental information mentioned in the previous section.

The reasoning information layer of an object is an evidence structure, which expresses the relationship between the lower information layers and basic common sense for object recognition and the probability of each evidence in the form of a graph. This is as shown in FIG.

The spatial object inference unit 35 searches for and infers information using lower layer recognition / inference information from lower layer recognition / inference information about an object or space. The spatial object inference unit 35 infers using inference information of the inference information layer, makes semantic inference through semantic information of the universal information layer or the spatiotemporal information layer, or the information of the information layer below it. Inference is performed through methods of searching for an object that matches a specific property (or feature). In particular, when the information of each space or object is composed of an ontology, the spatial object inference unit 35 performs inference using an ontology inference engine.

The mission analysis manager 36 creates and stores a probability-based behavior structure that determines the unit behavior to be executed by the robot from the mission given to the robot, returns the behavior structure at the request of the broker means, or sends the mission given to the robot. The robot extracts and returns the unit action to be executed by the robot and the information required for the action.

The mission analysis manager 36 configures the behavior structure as a Bayesian network having perceptual priority relations of perception data, evidence structure, and behavior. The optimal path is determined based on the probability of the precedence relation, and the action on the path is extracted as a unit action to be executed.

15 is a diagram illustrating a behavior structure according to an embodiment of the present invention. As shown in FIG. 15, the robot may be given a mission to "take a cup", "find a cup" in the room, or "find a cup". When a robot receives such a mission, it can be subdivided back into missions such as "identify the object", "move to the object", "pick up the object", "and come back in place". have. In addition, the "find" command "gets the shape and photometric characteristics of an object called a cup", "looks around", "finds a match in the image with the shape of the object", and "when it is dark, Turn it on. "," Look for a light switch on the ceiling. " It can be divided into detailed missions, such as "go to the switch", "get information on the switch", "turn on the switch according to the information", that is, the unit action for performing the mission. In this case, the mission is interpreted as a unit command, but it can be interpreted according to rules. However, when the surroundings are dark, it can be divided into various detailed missions such as turning on a light and walking a curtain. Therefore, we construct these situations into a probabilistic based Bayesian network.

As described above, the mission is formed step by step to find the optimal path by probability and perform unit action on the path. If the best unit action fails, then the best unit action is found and executed.

On the other hand, information for the unit action is needed. For example, information such as "cup", "switch", "how to grab", "how to turn on" and "photometer characteristics of the cup" is also extracted and returned together.

Next, the interface unit 32 of the central information processing system processes the information request of the indoor mobile service robot 10 in more detail.

The interface unit 32 processes a simple information request to the mobile service robot 10. That is, when the mobile service robot 10 requests the object / obstacle position information of the spatial information, the interface unit 32 takes the stored information as it is and sends it.

In addition, when the interface unit 32 requests a map from a corresponding robot, as described above, the interface unit 32 combines the information of the basic map information layer of the spatial information and the object / obstacle position information layer to form an integrated map. Create and provide to the robot. The United map is mainly used when the robot is driving.

In addition, the interface unit 32 may send the result of deduction. For example, when the service robot 10 requests "spatial information up to the cup" in order to perform a mission for bringing the cup, the interface unit 32 infers a specific "cup" for the "cup". To do this, the spatial object inference unit 35 requests an inference about this. Therefore, the interface unit 32 detects the position of the cup inferred by the spatial object inference unit 35 and transmits spatial information from the position of the current robot 10 to the cup. In this case, instead of giving information about all objects in the spatial information up to the cup, it may transmit a result of recognizing the object as an obstacle. This is because other objects in space do not have meaning in the mission of bringing a cup, so they may be treated as obstacles. That is, the interface unit 32 may provide only information corresponding to the recognition / inference level of the service robot 10. For example, when the floor cleaning command is given to an intelligent mobile service robot dedicated to indoor cleaning only, all objects are recognized as obstacles and the united map or obstacle map for the room can be transmitted. This is all determined by the level of recognition / inference of the service robot 10 and implements the interface appropriate for that level.

Meanwhile, the interface unit 32 may receive the information recognized by the service robot 10 and update the information therein. That is, the central information processing system 30 basically assumes not only one direction of information but also sharing through updating. Therefore, this means that the central information processing system 30 will act as a memory, not just a database. For example, if a person ate a bag that had been placed on a tea table in the morning for lunch, the drug no longer exists on the tea table. If another robot 10 finds the medicine bag, this information must be shared together. That is, when an object is lost or moved, or a new object enters the room, the service robot 10 recognizes a situation different from the spatial information received from the central information processing system 30 and sends it to the central information processing system 30. send.

In this step, the information updated by the robot 10 is preferably limited to information below the topological information layer. This is because the information below the topological information layer is the external features of the object, and thus are the data to be measured by the sensing of the robot 10. However, since the information of the upper information layer is semantic information, the updated information can be trusted only when the intelligence of the robot 10 has reasoning ability close to artificial intelligence such as learning. That is, due to the capability or limitation of the service robot, it is unreasonable to change information about the object to be serviced or semantic information of the upper information layer to the judgment of the robot. In the future, if the service robot has a reliable spatial perception ability or object modeling capability, the information of the upper information layer may be updated by the service robot.

As described above, each robot 10 is connected to the central information processing system 30 to acquire / modify information. Since the central information processing system 30 does not play a role of inferring new data or making a decision to control the robot 10, the improvement of performance is due to the fact that the capability of the user of the robot 10 stage is actual. It may be more important to improve performance. However, there is an advantage that input / output becomes clear and interworking with other modules is possible through effective information management and information layer policy through framework.

Next, using the framework of the central information processing system 30 of the service robot according to an embodiment of the present invention, the performance is improved or newly enabled experiments will be described according to Figures 16 to 25.

16 is a view showing a robot used in an experiment according to an embodiment of the present invention, Figure 17 is a view showing the input and output relationship of the map DB of the spatial information management unit in the experiment according to an embodiment of the present invention, FIG. 18 is a diagram illustrating a time diagram used in an experiment according to an embodiment of the present invention, and FIG. 19 is a 3D provided through a manual according to an embodiment of the present invention in an experiment according to an embodiment of the present invention. FIG. 20 is a diagram illustrating a map (left) and an integrated SLAM map (right), and FIG. 20 is a diagram illustrating a driving simulation using a path planner of a robot in an experiment according to an exemplary embodiment of the present invention. 21 is a diagram illustrating a similar object model stored in a geometric information layer of object information in an experiment according to an embodiment of the present invention, and FIG. 22 is an initial particle using ontology in an experiment according to an embodiment of the present invention. FIG. 23 is a diagram illustrating a step of recognizing a space or an object in an experiment according to an embodiment of the present invention.

Experiment 1 Vision-based Driving and SLAM Map Update

This experiment focuses on reducing the redundancy of data among the requirements of the information system of the service robot 10 and ensuring the concurrency and validation of the data. As shown in FIG. 16, the robot 10 assumed as an experimental environment has two map up-date modules. In addition, as shown in FIGS. 17 and 18, in the experiment, the central information processing system 30 organizes the information coming from the SLAM and the information coming from the manual (Manual) according to a policy system that can consistently and prevent collisions. Make it angry. The information by the manual is information input by a system administrator, and the information by SLAM is information collected by receiving and recognizing information sensed and recognized by various robots 10 in the room. The path planner of the robot is connected to the central information processing system 30 to receive the latest information. That is, the central information processing system 30 receives the spatial information by the SLAM from the robot, manages the entire map based on the manual, and transmits the map to the robot requesting the map.

Experimental results of applying this to the robot 10 are as follows. The left map of FIG. 19 is a three-dimensional map provided in advance to the robot 10 through a manual, and FIG. 20 illustrates occupancy grid after representing the data coming through the vision as an octree. SLAM process and the process of simulating the robot 10 is driving the path generated by the path planner (Path Planner) using the same.

<Experiment 2> 3D object recognition

This experiment was attempted to show that the efficiency of work can be improved by using data of various information layers. The mission of this experiment is to approach the refrigerator first to catch the target object in the refrigerator, find the juice of the object in the refrigerator, and then pinpoint the location of the object.

First, since the approximate location of the refrigerator is shown on the map of the central information processing system 30 of the service robot, the refrigerator is located and approached as in the case of <Experiment 1>. In the process of visiting the refrigerator, when a similar object is encountered as shown in FIG. 21, constraint knowledge for matching, which is organized in the evidence structure of the inference information layer as well as the geometric information layer and the photometric information layer information of the object information. By using to determine the exact relative spatial position between the robot 10 and the refrigerator. For example, the main information that distinguishes the refrigerator from the air conditioner is the location and number of horizontal lines detected. Also, to distinguish it from the book shelf, which is a similar object, it is necessary to check the ratio of the horizontal line and the vertical line. If it is difficult to distinguish between these simple lines, the result inferred from the more abstract domain should be used for matching. It uses Ontological Inference that the functional nature of the refrigerator requires a handle and the air conditioner does not. That is, based on the inference of the handle or not, the SIFT of the handle is found in the photometric information layer of the object and used to determine whether the refrigerator is through the handle SIFT matching. FIG. 22 illustrates a relationship between ontology inference information and a line model generated from information of geometric features of an object.

When the robot 10 arrives at the refrigerator as described above, the robot 10 starts looking for the target object. With the knowledge that a more accurate object position can be estimated by SIFT matching when the search area is small enough and the information of the spatiotemporal information layer that the target object is generally in the refrigerator, the robot 10 makes a sensing plan about the target object. According to this detection plan, the robot 10 determines the exact position of the object. FIG. 23 illustrates a process in which the robot 10 arriving at the refrigerator finds a juice that is an object using a photometric information layer or a measured information layer.

The use of this hierarchical information through the central information processing system 30 in accordance with the present invention is a method of reducing the unnecessary search area in the robot 10 performing missions and increasing the authenticity of the targets found during the search. Reduce the working time. This means that the performance of the service robot can be improved only by supporting the integrated information processing according to the present invention.

<Experiment 3> Go to N floor of Building A and bring a cup

Another experiment according to an embodiment of the present invention will be described with reference to FIGS. 24 and 25. 24 and 25 illustrate the process in which the robot performs a user's command "Go to Building N and bring a cup" using the central information processing system 30 of the service robot in order. will be.

First, in order for a robot to use a building indoor map and an object model to perform a task, these information must be input to the central information processing system 30 in advance by a system administrator. To this end, the manager models the information of each information layer about space and objects as defined in FIG. At this time, indoor map, which is measured information in space, is CAD model type (ifcXML or dfx, object according to 2D grid map and basic building design rules and IFC 2xx protocol, which is the international document design interchange standard). Stores x3d, dfx, 3ds, wrl, obj, etc.) The information is registered in the database 20 and stored. If the object / map model needs to be edited, the information is corrected from the DB. Then update them on the DB server. In order to make such modifications and edits, the central information processing system 30 should have a modeling means (not shown). Since the modeling means corresponds to commercial graphic modeling means such as general 3D Architect, Auto CAD, 3D Studio Max, Maya, and the like, detailed description thereof will be omitted. And when a new object that was not previously known by the robot is found and stored in the DB, the administrator does not know the name and type (category) of the object yet, so the administrator needs to record the information directly in the DB.

FIG. 24 is a view of an indoor central information processing system for a mobile service robot occurring in this process when moving to Building N, Room C, to perform the command of "Go to Building N and bring a cup". It shows the communication process.

The first step in this process is that the robot 10 does not know what the cup is, so the term cup should be informed as a specific model for recognition so that the robot can understand it. Therefore, for this purpose, first, the broker means 31 (or the corresponding interface unit 32) is requested for the position of an object called a cup, and the broker means 31 uses the spatial object inference unit 35 to determine the category of the word cup. Inference, and inference confirms that a cup is an object that can contain liquids that can be searched and drunk, and has a separate category from the whole object category. And now, the topological information layer is searched for where objects belonging to this cup category are located, and the fact that they are mainly on the N floor C kitchen or C desk is confirmed. The broker 31 requests the N-layer C map to the spatial information management unit 33 using this information, and checks the position of the object / obstacle of the structural information layer stored in the spatial information DB 22 to locate the object called a cup. Make sure that is recorded and pass this information to the robot. At this time, the robot 10 stores the location of the retrieved cup locally, and then brokers the map of the floor where the robot is currently located and the integrated map of the N floor, which is the destination, to move to the place where the object is located (N floor C). Ask the means 31, download the result and store it in the local map DB. If the integrated map of the floor where the robot is already located is stored in the robot's local map DB, it does not download separately.

In the next step, the robot must find an elevator to move to N floor C. Using the map of the current floor to check the approximate location of the elevator, while moving the robot requests the object information management unit 34 through the broker means 31 to recognize the elevator to download the elevator model. The elevator model is composed of the evidence structure expressing the judgment criteria of the object recognition of the information layer in the object information, the universal structure of the universal information layer expressing the universal features, and the CAD model expressing the geometric features of the geometric information layer. . The rest of the information layer is filled with null values because the elevator model has no texture and is highly reflective. After the robot recognizes the elevator, it must be able to press the elevator button, ride the elevator and manipulate the elevator button again. Therefore, the interior model of the elevator is also required. Since the elevator internal model corresponds to the spatial information unlike the exterior of the elevator, it is requested by the spatial information management unit 33 to create and bring up an integrated map.

FIG. 25 is a view illustrating the construction of the A building, the N floor, to perform the command of "go to the N floor of Building A and bring a cup" by using the information provided by the central information processing system 30 of the mobile service robot according to the present invention. It shows the process from the situation before entering the room C to finding the object after the entry. In this process, if an unknown object is found, it includes adding a new object to the object information, and if there is no information about the found object and no information about the universal model is present, it is considered as an obstacle. To update the spatial information.

In this step, when the robot reaches the N floor, it moves to the room C using the integrated map previously downloaded. When the robot reaches the room C, the space information (or map information) inside the room C and its Information on the fixed objects (refrigerators, washing machines, desks, etc.) and moving objects (cups, watches, chairs, etc.) existing therein are required. Therefore, as in the above process, the robot 10 requests the broker means 31 for these information, and the broker means 31 requests the information from each space or object management unit and receives the result and retransmits it to the robot. . The robot 10 receives these information and stores it in the local integrated map DB. At this time, since the storage inside the robot is limited in storage capacity, if the local storage is insufficient, it is downloaded and stored in order from a fixed object to a moving object up to a predetermined allowable limit.

After the robot enters in, if the robot finds an unknown object, it should check the approximate characteristics of the object and make sure that an object of this category is recorded in the central information processing system 30. Therefore, the robot 10 requests the universal model information from the broker means 30, and the broker means 31 transmits the searched result to the robot 10 to the object information management unit. The robot uses the acquired universal model information to compare the identified features and infer whether there is a category to which the found object belongs. When it is confirmed that a similar category exists, the broker means 31 requests model data belonging to this category. And using the model received from the broker means 31 attempts to recognize the unknown object. If the recognition is successful, ask the broker means 31 to update the position and attitude of the object, and the broker means 31 requests the spatial information manager 33 to update the object / obstacle position of the topological information layer. Register the object's position and pose on the unified map.

If a category to which an object found in this process belongs does not exist or a category exists but there is no specific model data, a process corresponding to steps 3 and 4 of FIG. 25 is performed. In other words, an unknown object that does not exist in the database should be considered as an obstacle and reflect this information. The robot can then use this information to safely manipulate and drive the obstacles. Finally, the robot must have the modeled robot itself until the manager checks the object and inputs the information later. Therefore, the robot models and stores shapes and features of an object through modeling means held by the robot itself. A request is made to the broker means 31 so that the spatial information management unit 33 updates each information layer (mainly a topological information layer or less) related thereto.

Next, an information processing method of a service robot according to an embodiment of the present invention will be described with reference to FIG. 26. 26 is a flowchart illustrating an information processing method of a service robot according to an embodiment of the present invention.

As shown in Figure 26, the information processing method of the service robot comprises the steps of (a) configuring and storing the space and object information for the room for each layer according to the recognition / reasoning level; (b) for each of the robots, constructing an interface for interpreting the information request of the robot and converting the requested information into a form that the robot can interpret; (c) receiving an information search request for a space or an object of the robot; Determining if inference is needed to retrieve the information, and if necessary, inferring the information from the given information about the object or space using higher layer recognition / inference information; (f) retrieving information about the requested space or object; (g) constructing an integrated map if the requested information requires space and object information at the same time; (h) transmitting the retrieved space or object information to the requested robot.

The spatial information or object information may include: a measurement information layer storing measured information, a photometric information layer storing photometric features, a geometric information layer storing segmented spaces or objects in a geometric structure; A structural information layer for classifying, symbolizing, layering, and storing components of a space or an object, and a topological information layer for storing information representing a position or an obstacle of an object in a space; Universal information layer that defines and stores a space or object, or a universal model of the space or its components, space-time information layer that stores changes and relationships over time and space in an object or space, sublayer information about an object or space And an inference information layer that stores inference information for inferring a space or an object or a state of the space or an object from the object.

The measured information layer of the space includes basic map information for storing geographical information of the indoor space, the photometric information layer of the space includes a map as a photometric characteristic of the indoor space, and the geometric information layer of the space represents the indoor space. Geometrically segmented information, wherein the topological information layer of the space includes information about the global coordinates and attitudes of objects or objects recognized as obstacles, and the universal information layer of the space is space or space And the semantic information of the components of the space and the space-time information layer of the space includes the brightness of the light, the direction of the light, the temperature, the humidity that varies depending on the temporal and spatial situation of the room.

The measured information layer of the object includes 2D / 3D original image information that stores two-dimensional or three-dimensional representation data obtained by actual measurement of the object, and the space-time information layer of the object includes a time-dependent relationship of object-space, object Existence and space-time dependence, space-time dependency between object and object, space-time dependency between object and function, and space-time dependency of object-physical characteristics. Include evidence structures to store graphs created by probability.

The photometer features include information about Scale Invariant Feature Transform (SIFT), Harris Corner, Color, and Line.

The obstacle information is expressed and stored in a mesh or volume.

The two-dimensional or three-dimensional representation data is depth data obtained from any one of a 2D image (Raw Image), a laser scanner, a stereo camera, a structured light camera calculated through at least two camera angles ( Depth Data), and one or more of mesh data generated from the depth data.

In step (a), the spatial information or object information is configured and stored as an ontology, and step (e) is inferred by the ontology.

When the interface unit requests a map from the corresponding robot, the interface unit combines the information of the basic map information layer of the spatial information and the object / obstacle position information layer to create an integrated map and provide the integrated map to the robot.

The information processing method of the service robot may include: (a0) creating and storing a probability-based action structure for determining a unit action to be executed by the robot from a mission given to the robot before step (a); (d) after step (c), returning the action structure by the search request of the robot, or analyzing the mission given to the robot, and extracting and returning unit action to be executed by the robot and information necessary for the action. do.

In step (a0), perception data, evidence structure, and behavior are composed of the behavioral structure as a Bayesian network having a probabilistic priority relation. In step (d), an optimal path is determined based on the probability of the priority relation, and the action on the path is extracted as a unit action to be executed.

The present invention also relates to a computer-readable recording medium recording the information processing method of the service robot.

For the insufficient description of the central information processing method of the service robot, refer to the description of the central information processing system of the service robot described above.

As mentioned above, although the invention made by this inventor was demonstrated concretely according to the said Example, this invention is not limited to the said Example and can be variously changed in the range which does not deviate from the summary.

The present invention can be applied to the field of sharing and providing information required for the mobile service robot in the room, in particular, even when a number of robots are used in the room.

1 is a view showing the configuration of an entire system for implementing a central information processing system and method of a service robot according to the present invention.

2 is a block diagram of a configuration of a central information processing system of a service robot according to an embodiment of the present invention.

3 is a block diagram of an information layer of a central information processing system according to an embodiment of the present invention.

4 is a diagram illustrating a configuration of an information layer of a space or an object according to an embodiment of the present invention.

5 is a diagram illustrating a real map and a base map according to an embodiment of the present invention.

FIG. 6A is a diagram illustrating a photometric feature (SIFT) extracted in space according to an embodiment of the present invention.

FIG. 6B is a diagram illustrating a photometric characteristic (SIFT) of spatial information according to an embodiment of the present invention on a three-dimensional map.

FIG. 7A illustrates an obstacle represented by Octree, which is a volumetric expression method, according to an embodiment of the present invention.

7B is a view showing the interior space and the object according to an embodiment of the present invention.

9 is a diagram illustrating an ER table of an integrated map according to an embodiment of the present invention.

8 is a diagram illustrating an evidence structure in the inference information layer of a space according to an embodiment of the present invention.

FIG. 10A is a diagram illustrating a result of modeling a pencil, which is a 3D object, according to an embodiment of the present invention as an x3d file, which is one of CAD forms.

FIG. 10B illustrates code modeled with the x3d file of FIG. 10A in accordance with an embodiment of the present invention.

11 is a diagram illustrating object configuration information in a structural information layer according to an embodiment of the present invention.

12A illustrates an approach direction of an object in a topology information layer of an object according to an embodiment of the present invention.

12B is a diagram illustrating a contact point of an object in the topology information layer of the object according to an embodiment of the present invention.

13 is a diagram illustrating a universal model of a cup according to an embodiment of the present invention.

14 is a diagram illustrating an evidence structure of an object according to an embodiment of the present invention.

15 is a diagram illustrating an action structure represented by a Bayesian network according to an embodiment of the present invention.

16 is a diagram illustrating a robot used in an experiment according to an embodiment of the present invention.

FIG. 17 is a diagram illustrating input / output relations of a map DB of a spatial information management unit in an experiment according to an embodiment of the present invention.

18 is a diagram illustrating a time diagram used in an experiment according to an embodiment of the present invention.

FIG. 19 illustrates a 3D map (left) and an integrated SLAM map (right) provided through a manual according to an embodiment of the present invention in an experiment according to an embodiment of the present invention.

20 is a diagram illustrating a driving simulation using a path planner of a robot in an experiment according to an exemplary embodiment of the present invention.

21 is a diagram illustrating a similar object model stored in a structural information layer of object information in another experiment according to an embodiment of the present invention.

FIG. 22 illustrates generation of initial particles using ontologies in another experiment according to an embodiment of the present invention.

23 is a diagram illustrating a step of recognizing a space or an object in another experiment according to an embodiment of the present invention.

24 is a flowchart illustrating a communication process occurring in a process of moving to perform a command in another experiment according to an embodiment of the present invention.

25 is a flowchart illustrating a process of finding an object in a step of entering a room to perform a command in another experiment according to an embodiment of the present invention.

26 is a flowchart illustrating an information processing method of a service robot according to an embodiment of the present invention.

Explanation of symbols on the main parts of the drawings

10: service robot 11: robot control unit

20: Database 22: Spatial Information DB

23: object information DB 24: behavior structure DB

30: central information processing device of the robot 31: broker means

32: interface unit 33: spatial information management unit

34: object information management unit 35: spatial object reasoning unit

36: Mission Analysis Management Department

Claims (23)

A central information processing system for communicating with at least one mobile service robot indoors, A broker means comprising at least one interface unit corresponding to the robot, wherein the interface unit interprets an information request of a corresponding robot, converts the requested information into a form that the robot can interpret, and transmits the requested information; A spatial information manager configured to store and store the spatial information of the indoor space according to a recognition / inference level, and to retrieve and return the spatial information requested by the spatial information request of the broker means; An object information management unit configured to store and store objects existing in the room in layers according to recognition / inference levels, and retrieve and return object information requested by the object information request of the broker means; A spatial object reasoning unit for searching for and inferring information by using upper layer recognition / inference information from lower layer recognition / inference information on an object or space, The spatial information or object information, A measurement information layer including measured information, A photometric information layer comprising a photometric feature, A geometric information layer comprising information segmented into spaces or objects into geometric structures, Structural information layer including information that is classified, layered, and layered on components of a space or an object, A topological information layer including information representing the position or obstacle of an object in space or accessibility information for holding of the object; A universal information layer containing information about a space or object, or a universal model of components of the space or object, Spatiotemporal information layer including information about changes and relationships of objects or spaces in time and space, And an inference information layer including inference information for inferring the space or the object or the state of the space or the object from the lower layer information about the object or the space. delete The method of claim 1, The measured information layer of the space includes basic map information including geographic information of the indoor space, The photometric information layer of the space includes a photometric feature map of the indoor space, The geometric information layer of the space includes geometrically segmented information of the interior space, The topological information layer of the space includes information about an object recognized as an obstacle or global coordinates and attitude of the object, The universal information layer of space includes semantic information about the space or its components, The space-time information layer of the space is a central information processing system of a service robot, characterized in that it comprises the brightness of the light, the direction of the illumination, temperature, humidity that varies depending on the temporal and spatial situation of the room. The method of claim 1, The measured information layer of the object includes 2D / 3D original image information for storing two-dimensional or three-dimensional representation data obtained by actual measurement of the object, The topological information layer of the object includes a plurality of three-dimensional contact point (access point) and information of the accessible (accessible direction) represented by a three-dimensional direction vector, The space-time information layer of the object is information about the time-dependent relationship between object-space, the existence and space-time dependency between objects, the space-time dependency between object and objects, the space-time dependency between object and functions, and the space-time dependency between object and physical properties. Including, The inference information layer of the object includes the information on the evidence structure consisting of a graph to create a relationship between the information of the lower information layer as a probability, the central information processing system of the service robot. The photometric characteristic of claim 3 or 4, wherein A central information processing system for a service robot, comprising information on a scale invariant feature transform (SIFT), a Harris corner, a color, and a line. The method of claim 3, wherein the obstacle information, Central information processing system of a service robot, characterized in that it comprises information expressed in a mesh (mesh) or volume. The method of claim 4, wherein the two-dimensional or three-dimensional representation data, 2D images generated by at least two camera angles, Depth data obtained from one of a laser scanner, a stereo camera, or a structured light camera, Mesh data generated from the depth data Central information processing system of a service robot, characterized in that it comprises any one or more of. The method of claim 1, The spatial information manager or object information manager configures and stores the spatial information or object information as an ontology, The spatial object inference unit of the service robot central information processing system, characterized in that the inference by ontology. The method of claim 3, wherein the interface unit, If there is a request for a map from a corresponding robot, a service robot comprising a combination of basic map information of the actual information layer of the space and object / obstacle position information of the topological information layer is prepared and provided to the robot. Central information processing system. The method of claim 1, Create and store a probability-based behavior structure that determines the unit behavior to be executed by the robot from the mission given to the robot, return the behavior structure at the request of the broker means, or interpret the mission given to the robot and And a mission analysis management unit for extracting and returning information necessary for the action. The method of claim 10, wherein the mission analysis management unit Perception data, evidence structure, and behavior are composed of the behavioral structure as a Bayesian network having a probabilistic precedence relation, And determining an optimum path based on the probability of the precedence relation, and extracting the action on the path as a unit action to be executed. In the information processing method of a service robot using a central information processing system for communicating with at least one mobile service robot indoors, (a) constructing and storing space and object information about the room for each layer according to a recognition / inference level; (b) for each of the robots, constructing an interface for interpreting the information request of the robot and converting the requested information into a form that the robot can interpret; (c) receiving an information search request for a space or an object of the robot; (e) determining if inference is needed to retrieve the information, and if necessary, inferring the information using the higher layer recognition / inference information from the given information about the object or space; (f) retrieving information about the requested space or object; (g) sending the retrieved space or object information to the requested robot; Information processing method of the service robot comprising a. The method of claim 12, wherein the spatial information or object information, A measurement information layer including measured information, A photometric information layer comprising a photometric feature, A geometric information layer comprising information segmented into spaces or objects into geometric structures, Structural information layer including information that is classified, layered, and layered on components of a space or an object, A topological information layer including information representing an object's position or obstacle in space; A universal information layer containing information about a space or object, or a universal model of components of the space or object, Spatiotemporal information layer including information about changes and relationships of objects or spaces in time and space, And an inference information layer including inference information for inferring the space or the object or the state of the space or the object from the lower layer information on the object or the space. The method of claim 13, The measured information layer of the space includes basic map information including geographic information of the indoor space, The photometric information layer of the space includes a photometric feature map of the indoor space, The geometric information layer of the space includes geometrically segmented information of the interior space, The topological information layer of the space includes information about an object recognized as an obstacle or global coordinates and attitude of the object, The universal information layer of space includes semantic information about the space or its components, The space-time information layer of the space is information processing method of a service robot, characterized in that it comprises the brightness of the light, the direction of the light, temperature, humidity that varies depending on the time and space of the room. The method of claim 13, The measured information layer of the object includes 2D / 3D original image information for storing two-dimensional or three-dimensional representation data obtained by actual measurement of the object, The topological information layer of the object includes a plurality of three-dimensional contact point (access point) and information of the accessible (accessible direction) represented by a three-dimensional direction vector, The space-time information layer of the object is information about the time-dependent relationship between object-space, the existence and space-time dependency between objects, the space-time dependency between object and objects, the space-time dependency between object and functions, and the space-time dependency between object and physical properties. Including, The inference information layer of the object includes information about the evidence structure composed of a graph that creates a relationship between the information of the lower information layer as a probability. The method of claim 13, wherein the photometer features, An information processing method of a service robot, comprising information on a scale invariant feature transform (SIFT), a Harris corner, a color, and a line. The method of claim 13, wherein the obstacle information, An information processing method of a service robot, characterized by including information represented by a mesh or a volume. The method of claim 15, wherein the two-dimensional or three-dimensional representation data, 2D images generated by at least two camera angles, Depth data obtained from one of a laser scanner, a stereo camera, or a structured light camera, Mesh data generated from the depth data Information processing method of a service robot, characterized in that it comprises any one or more of. The method of claim 12, In step (a), the spatial information or the object information is configured and stored as an ontology, In the step (e), the information processing method of the service robot, characterized in that the reasoning by the ontology. The method of claim 13, wherein step (g) And if the requested information includes the basic map information of the actual information layer of the space and the object / obstacle position information of the topological information layer, the information is combined to form and send an integrated map. The method of claim 12, (a0) creating and storing a probability-based behavior structure that determines the unit behavior to be executed by the robot from the mission given to the robot before step (a); (d) after step (c), returning the action structure by the search request of the robot, or analyzing the mission given to the robot, and extracting and returning unit action to be executed by the robot and information necessary for the action. Information processing method of a service robot, characterized in that. The method of claim 21, In step (a0), perception data, evidence structure, and behavior are composed of the behavioral structure as a Bayesian network having a probabilistic priority relation. The step (d) determines the optimal path based on the probability of the precedence relation, and extracts the action on the path as a unit action to be executed. Treatment method. A computer-readable recording medium recording the information processing method of the service robot of claim 12 or 22.

Images