CN113010631B

CN113010631B - Knowledge engine-based robot and environment interaction method

Info

Publication number: CN113010631B
Application number: CN202110423443.1A
Authority: CN
Inventors: 高岳
Original assignee: Shanghai Jiaotong University
Current assignee: Shanghai Jiaotong University
Priority date: 2021-04-20
Filing date: 2021-04-20
Publication date: 2022-11-11
Anticipated expiration: 2041-04-20
Also published as: CN113010631A

Abstract

The invention discloses a robot and environment interaction method based on a knowledge engine, which solves the problem that a robot can not be designed and planned specifically according to a specific task, and has the technical scheme that a knowledge acquisition mechanism is established, a knowledge graph consisting of points and edges is established, a knowledge interpretation mechanism is established, knowledge storage is carried out, a knowledge inference mechanism is established, a robot inquiry library is established, and a feasible path from a task starting point to a task ending point is obtained through an inquiry language.

Description

Knowledge engine-based robot and environment interaction method

Technical Field

The invention relates to a robot and environment interaction technology, in particular to a robot and environment interaction method based on a knowledge engine.

Background

With the establishment and application of a large-scale knowledge system of many robots, such as google knowledge graph, IBMWatson, wikipedia, etc., robots are more and more commonly applied, and task solving manners are more and more, such as data mining, natural language processing, image processing, voice processing, etc., but most of these knowledge systems are only used for solving daily problems, are knowledge bases designed for solving the problem of how to use the robots by human beings, and do not completely realize that the robots perform specific design and planning according to specific tasks, which is more valuable for human beings.

When a robot is dealing with a specific task, such as transferring a specific item to a specific place, it needs to know various detailed information to perform natural language understanding, perception, planning and control. The robot now needs to acquire language symbols based on knowledge of the physical object, knowledge of the object's placement in a particular location, and also grasp the appropriate plan for guessing and manipulating the object. It is still an open question to effectively deal with this joint knowledge in different tasks and scenarios.

Disclosure of Invention

The invention aims to provide a robot and environment interaction method based on a knowledge engine, which can greatly improve the acquisition breadth and depth of the robot knowledge, reduce the processing difficulty of different data, quickly inquire and execute the current task and is more practical.

The technical purpose of the invention is realized by the following technical scheme:

a robot and environment interaction method based on a knowledge engine comprises the following steps:

s1, establishing a knowledge acquisition mechanism;

s2, constructing a knowledge graph composed of points and edges, and connecting knowledge acquired by the robot according to logic, time or space sequence to form a knowledge network graph;

s3, establishing a knowledge interpretation mechanism, and uniformly managing different forms of data of each knowledge source;

s4, knowledge storage is carried out, and data in different forms are stored in a data storage cluster mode;

s5, establishing a knowledge inference mechanism, and modifying and reconstructing a knowledge graph;

and S6, establishing a robot query library, and obtaining a feasible path from the task starting point to the task ending point through a query language.

Preferably, the knowledge acquisition in step S1 specifically includes:

establishing an image video data set related to human behaviors, and performing behavior analysis prediction on human actions based on a deep learning semantic segmentation principle;

performing behavior prediction on the wearable perspective display through an augmented reality technology based on computer vision principles and a behavior prediction algorithm, and providing a plurality of options for determining behavior intentions to form knowledge graph nodes;

and carrying out authorization request and connection with a plurality of network knowledge sources to acquire internet knowledge.

Preferably, the building of the knowledge graph in the step S2 specifically includes:

representing any object by using nodes, representing a task relation between two nodes by using edges, and forming a knowledge graph by using the nodes and the edges;

the object represented by the node comprises a person, an object, audio, video and a data set;

the edges comprise a unidirectional edge and a bidirectional edge; the one-way edge represents tasks with requirements on time sequence and space sequence; the bidirectional edges represent tasks that require data interaction.

Preferably, the knowledge interpretation in step S3 is specifically:

connecting the knowledge acquired by the robot according to logic, space or time sequence to form a knowledge network diagram;

managing knowledge in a metadata manner to store different forms of data obtained from different sources;

adding an inverse pointer points to the knowledge base in the various sources.

Preferably, the step S4 of knowledge storage specifically includes: storing data by establishing a data storage cluster; the stored data comprises various knowledge, crowdsourcing feedback of users and other machine learning algorithm parameters; and placing large-scale media information including pictures, videos and 3D point clouds in a distributed storage system.

And establishing a graphic database taking the data storage cluster as a unique trust source based on the data storage cluster.

Preferably, the knowledge inference mechanism in step S5 is specifically:

managing the latest updated knowledge points by adopting a distributed queue system;

managed knowledge points can be consumed for populating the graph database;

according to the difference of the confidence degrees of the knowledge acquired from different sources, the new knowledge with high confidence degree is inserted into the existing graph, and the knowledge graph can be updated by adding, deleting or splitting.

Preferably, the query in step S6 is specifically:

adopting a graph retrieval function to find a path for completing the task by traversing all paths from a task starting point to a task ending point in the knowledge graph;

realizing a specified task according to all knowledge points and edges contained in the path;

the graph retrieval function supports sorting all paths according to specified conditions.

In conclusion, the invention has the following beneficial effects:

by the method, the robot can be allowed to learn and share relevant knowledge expression based on various knowledge sources, the robot can interact with relevant knowledge, natural language, visual data and the like in the Internet and the existing knowledge base while sensing, planning and controlling are executed, the self knowledge graph is established, the knowledge graph is inquired, reasoned, improved and perfected in continuous learning, the width and the depth of acquisition of the robot knowledge are greatly improved, the difficulty in processing different data is reduced, the difficulty of the types of tasks which can be executed by the robot is greatly improved due to various off-line and on-line knowledge bases and the real-time sensing capability, and the established knowledge inquiry mechanism enables the robot to quickly inquire and use the relevant knowledge to complete the current task and has strong practicability.

Drawings

FIG. 1 is a flow diagram of knowledge engine build and query;

FIG. 2 is a schematic diagram of a knowledge engine architecture;

fig. 3 is a schematic diagram of a knowledge graph of an example of a robot execution end teacup.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings.

According to one or more embodiments, a knowledge engine based robot and environment interaction method is disclosed, as shown in fig. 1, comprising the following steps:

s1, establishing a knowledge acquisition mechanism.

And S2, constructing a knowledge graph composed of points and edges, and connecting the knowledge acquired by the robot according to logic, time or space sequence to form a knowledge network graph.

And S3, establishing a knowledge interpretation mechanism, and uniformly managing different forms of data of each knowledge source.

and S5, establishing a knowledge inference mechanism, and modifying and reconstructing the knowledge graph.

For clarity, the solution of the invention is now set out and described in detail, with reference to the task of commanding the robot to move the cup from the upper end of the table to the human hand, as a specific example, as shown in figures 2 and 3:

first, a knowledge acquisition mechanism is established. Firstly, an image video data set about human behaviors can be established, and the human actions are analyzed and predicted based on a deep learning semantic segmentation principle and by combining technologies such as human joint freedom degree division; furthermore, objects in the perception range of the visual sensor can be collected and identified through Augmented Reality (AR) technology and a behavior prediction algorithm based on the computer vision principle, behavior prediction is carried out on the wearable perspective display through analysis of human behaviors, and a plurality of options are provided to determine behavior intentions so as to form nodes of a knowledge graph; meanwhile, authorization requests and connection can be carried out with a plurality of network knowledge sources, and the acquisition of internet knowledge is realized. A variety of knowledge acquisition mechanisms can ensure the diversity of knowledge sources.

And secondly, building a knowledge graph. The knowledge graph is composed of nodes and edges, and knowledge acquired by the robot is connected according to logic, time or space sequence to form a knowledge network graph. Representing any object by using nodes, representing the task relationship between two nodes by using edges, or representing the operation to be executed from one point to another point, and forming a knowledge graph by using the nodes and the edges; the object represented by the node comprises a person, an object, audio, video and a data set; the edges comprise a unidirectional edge and a bidirectional edge; the unidirectional edge is mainly used for representing tasks with requirements on time sequence and space sequence; the bidirectional edges are primarily concerned with representing tasks that require data interaction.

In the example, a person, a teacup and a node of a grabbing action need to be added into a knowledge graph in advance, and the edges represent the relationship among the nodes, namely, the robot can grab the teacup, and the person can receive the things grabbed by the robot.

And thirdly, establishing a knowledge interpretation mechanism. Since knowledge structures obtained from different sources may be different, knowledge is uniformly managed in a metadata manner in order to store different forms of data. The knowledge acquired by the robot can be connected according to logic, space or time sequence to form a knowledge network diagram. Meanwhile, in order to avoid great influence on the knowledge graph when the knowledge base from each source changes, an inverse pointer is added to point to the knowledge base in each source so as to trace and update the source.

In this embodiment, the robot includes a sensing module and a planning module, and includes data such as object recognition and robot motion planning parameters. To avoid having a large impact on the knowledge graph when the knowledge base from the various sources changes, inverse pointers are added to point to the underlying knowledge in the sources involved.

And fourthly, storing knowledge. A data storage cluster is established for storing data in different forms, including various kinds of knowledge, crowd-sourced feedback of users and parameters of other machine learning algorithms. And placing large-scale media information including pictures, videos and 3D point clouds in a distributed storage system. Based on the data storage cluster, a graphic database which takes the data storage cluster as a unique trust source is established, and the graphic database takes the data storage cluster as the unique trust source, so that the graphic database can be conveniently and rapidly rebuilt under the condition that knowledge is wrong or malicious knowledge is encountered.

In the example, the picture information of the person and the teacup is placed in the distributed storage system and added into the graphic database, and the database takes the data storage cluster as the only trust source, so that the graphic database can be conveniently and quickly reconstructed under the condition that knowledge such as the shape of the teacup, the grippability of the teacup and the like is wrong or malicious knowledge is encountered.

And fifthly, establishing a knowledge reasoning mechanism and modifying the knowledge graph through knowledge reasoning. The confidence degrees of the knowledge acquired from different sources may be different, and according to the difference of the confidence degrees of the knowledge acquired from different sources, the new knowledge with high confidence degree is inserted into the existing map, and the update of the knowledge map can be specifically performed through addition, deletion, splitting or the like, so as to realize the update of the knowledge map. And managing the knowledge points which are updated recently by adopting a distributed queue system. Managed knowledge points can be consumed by inference algorithms, graph generators, and machine learning plug-ins to populate a graph database. These plug-ins, as well as other learning algorithms applied to the overall graphical operation, form a framework for learning reasoning.

As in this example, the more optimal parameters for robot path planning are updated to the original knowledge graph, the path planning parameters for the robot to reach the person in the original knowledge graph may be deleted, and the new parameters added to the original location.

And sixthly, establishing a robot query library and retrieving. The query library supports a graph retrieval function, and a path for completing a task is found by traversing all paths from a task starting point to a task ending point in a knowledge graph by adopting the graph retrieval function; realizing a specified task according to all knowledge points and edges contained in the path; and the graph retrieval function supports sorting all paths according to specified conditions.

In this example, all feasible paths from the teacup on the table to the person, that is, all paths for completing the target task in this embodiment, can be obtained through the graph retrieval function, and according to all knowledge points and edges included in the paths, the task of bringing the teacup from the upper end of the table to the person is realized. The specific query statement may be expressed as:

Paths＝fetch({name：’Human’})→[r*]→({name：’Cup’})

where, () represents a node, [ ] represents an edge, and fetch is the name of the function that queries the path.

If necessary, all feasible paths can be ranked according to the confidence of the knowledge points to select the best path. The concrete query statement may be expressed as:

SortBy(λP→BeliefP)paths

where SortBy is the name of the sorting function, and the function sorts all feasible paths from high to low according to confidence and returns.

Establishing a robot knowledge engine which can be learned and shared, and establishing a perfect knowledge acquisition mechanism, including a local acquisition mechanism and a network acquisition mechanism; establishing a knowledge graph based on the acquired knowledge; the data of different modes such as symbols, natural language, touch sense, robot motion trail, visual characteristics and the like are processed, so that the uniformity of processing the data of different modes is realized; when the robot executes a task, the robot senses in real time, knowledge from the Internet and various existing robot knowledge bases are utilized simultaneously, and the task is planned based on a knowledge graph; an interaction mechanism between the robot and the knowledge engine is constructed so as to realize task understanding, environment perception and task planning of the robot.

The present embodiment is only for explaining the present invention, and it is not limited to the present invention, and those skilled in the art can make modifications without inventive contribution to the present embodiment as required after reading the present specification, but all of them are protected by patent law within the scope of the present invention.

Claims

1. A robot and environment interaction method based on a knowledge engine is characterized by comprising the following steps:

s1, establishing a knowledge acquisition mechanism; the method specifically comprises the following steps:

establishing an image video data set about human behaviors, and performing behavior analysis prediction on human actions based on a deep learning semantic segmentation principle;

authorization requests and connection are carried out with a plurality of network knowledge sources to obtain internet knowledge;

s2, constructing a knowledge graph composed of points and edges, and connecting the knowledge acquired by the robot according to logic, time or space sequence to form a knowledge network graph; the method specifically comprises the following steps:

the objects represented by the nodes comprise people, objects, audio, video and data sets;

the edges comprise a unidirectional edge and a bidirectional edge; the one-way edge represents tasks with requirements on time sequence and space sequence; the bidirectional edge represents a task needing data interaction;

s3, establishing a knowledge interpretation mechanism, and uniformly managing different forms of data of each knowledge source; the knowledge interpretation is specifically as follows:

adding an inverse pointer to point to a knowledge base in various sources;

s4, knowledge storage is carried out, and different forms of data are stored in a data storage cluster mode;

s5, establishing a knowledge reasoning mechanism, and modifying and reconstructing a knowledge graph;

s6, establishing a robot query library, and obtaining a feasible path from a task starting point to a task ending point through a query language; the method specifically comprises the following steps:

2. The knowledge engine-based robot and environment interaction method of claim 1, wherein the knowledge storage in step S4 is specifically: storing data by establishing a data storage cluster; the stored data comprises various knowledge, crowdsourcing feedback of the user and other machine learning algorithm parameters; and placing large-scale media information including pictures, videos and 3D point clouds in a distributed storage system.

And establishing a graphic database with the data storage cluster as a unique trust source based on the data storage cluster.

3. The robot and environment interaction method based on knowledge engine as claimed in claim 2, wherein the knowledge inference mechanism in step S5 is specifically:

managed knowledge points can be consumed for populating the graph database;