WO2022103081A1

WO2022103081A1 - Neural symbolic-based rule generation method and device, for inferring intention of user

Info

Publication number: WO2022103081A1
Application number: PCT/KR2021/015998
Authority: WO
Inventors: 박영택; 박현규; 신원철; 노재승
Original assignee: 숭실대학교산학협력단
Priority date: 2020-11-12
Filing date: 2021-11-05
Publication date: 2022-05-19
Also published as: KR102365838B1

Abstract

Disclosed are a neural symbolic-based rule generation method and device, for inferring an intention of a user. Provided, according to the present invention, is a neural symbolic-based rule generation device comprising: a processor; and a memory connected to the processor. The memory comprises program instructions performable by the processor so as to: collect, from a living environment of a user, perception information including an action and a pose; convert the collected perception information into triple data by using an ontology defined with respect to a relationship between an intention of the user and the perception information; embed, in a multidimensional space, a vector associated with a relation and entities of the converted triple data; when a target triple, expressing an intention of the user, is inputted, update parameters of a preset rule template through the modules AND, OR, Unify and Aggregation of a Neural Theorem Prover (NTP); and, through the parameter update, generate a rule having, as a condition thereof, one or more perception information indicating characteristics of the target triple.

Description

Method and Apparatus for Generating Neural Symbol Based Rules for Inference of User Intent

The present invention relates to a method and apparatus for generating a neural symbol-based rule for user intention inference.

By understanding the intentions of users, especially the elderly, service according to the situation is expected to contribute to improving the quality of life, such as supplementing the elderly's abilities, and is being studied at many companies and universities.

Among them, a study that infers the intentions of the elderly using the Event Calculus-based reasoning engine from the behavior and posture information that changes depending on the time of occurrence and end of the behavior and posture of the elderly and performing life pattern analysis through this is representative.

The technology analyzes the relationship between the observed behavior and posture information of the elderly and defines the Event Calculus-based rules to infer the intentions of the elderly. In order to infer the intention according to the perceptual information generated at this time, a correct rule that reflects the perceptual information is essential, and it is written by an expert.

Through this, the rule analyzed perceptual information showed good performance in intention inference, but there is a problem that a lot of cost is consumed to define a rule to reflect this as the living data of the elderly are accumulated.

This is because it takes a lot of time and money for an expert to analyze the perception information of the elderly observed for a long time and generate a rule.

In order to solve the problems of the prior art, the present invention intends to propose a method and apparatus for generating a neural symbol-based rule for inference of a user's intention, which can generate a rule for inference of a user's intention in a small amount of time and cost.

In order to achieve the above object, according to an embodiment of the present invention, there is provided an apparatus for generating a neural symbol based rule for inferring user intention, comprising: a processor; and a memory connected to the processor, wherein the memory collects perceptual information including behavior and posture from the user's living environment, and uses the collected perceptual information to define a relationship between the user's intention and perceptual information. When an ontology is used to convert triple data, a vector about entities and relations of the transformed triple data is embedded in a multidimensional space, and a target triple expressing a user's intention is input, AND of Neural Theorem Prover (NTP) , OR, Unify and Aggregation module to update a parameter of a preset rule template, and update the parameter to generate a rule conditional on one or more perceptual information indicating characteristics of the target triple, executed by the processor An apparatus for generating a neural symbol based rule including possible program instructions is provided.

The rule template includes a rule head corresponding to a conclusion and a rule body corresponding to a condition, wherein the rule head and rule body are composed of triple data of a predicate corresponding to a relation, a subject corresponding to an entity, and an object; A triple for the perceptual information may be arranged in the rule body.

The rule head may include a first parameter for an object corresponding to a user's intention, and the rule body may include second and third parameters for an object corresponding to each of the user's actions and postures.

The program instructions may update an embedding vector value of a parameter corresponding to an object of the rule template.

The program instructions calculate a similarity between the object of the target triple and the first parameter, calculate a similarity between the collected perceptual information and the second and third parameters, and connect the target triple to an action and posture triple You can create a path to a tree that becomes

The subject of the rule template may be bound to a specific user.

The program instructions may use a weight matrix based on a frequency of perceptual information related to a user's intention to update the parameter.

According to another aspect of the present invention, there is provided a method for generating a rule based on a neural symbol for inference of user intention in a device including a processor and a memory, the method comprising: collecting perceptual information including behavior and posture from a user's living environment; converting the collected perceptual information into triple data using an ontology defined for a relationship between user intention and perceptual information; embedding a vector regarding entities and relations of the transformed triple data into a multidimensional space; updating parameters of a preset rule template through AND, OR, Unify, and Aggregation modules of Neural Theorem Prover (NTP) when a target triple expressing the user's intention is input; and generating a rule having one or more perceptual information indicating characteristics of the target triple as a condition through updating the parameter.

According to another aspect of the present invention, there is provided a computer readable program for performing the above method.

According to another aspect of the present invention, there is provided a system for generating a neural symbol based rule for inferring user intention, comprising: a perceptual information collecting unit including actions and postures from a user's living environment; an ontology storage unit for storing an ontology defining a relationship between user intention and perception information; a preprocessor for converting the collected perceptual information into triple data using the ontology; an embedding unit for embedding a vector related to an entity and a relation of the transformed triple data into a multidimensional space; When a target triple expressing the user's intention is input, the parameters of the preset rule template are updated through AND, OR, and Unify modules Neural Theorem Prover (NTP); and a rule generator configured to generate a rule having one or more perceptual information indicating the characteristics of the target triple as a condition through the update of the parameter.

According to the present invention, it is possible to understand the intentions of the elderly and provide services according to changes in circumstances, reducing worries about the elderly in a family living with the elderly or an elderly home living away from their family, thereby reducing accidents for the elderly at home and promptly responding to emergency situations. There are advantages.

1 is a diagram illustrating an example of similarity calculation for triple data.

2 is an example showing a process of finding a rule that satisfies a target triple through the NTP method.

3 is a diagram illustrating an example of an ontology expressing the relationship between perceptual information and the intention of the elderly.

4 is a diagram illustrating a Neural Symbolic-based rule generation system for inferring the intention of an elderly person.

5 is a diagram illustrating an example of a rule creation process for an elderly person's eating (MEAL) intention.

6 is a table summarizing behaviors appearing with many frequencies for each intention.

7 shows a perceptual information intention matrix according to intention.

8 is a diagram illustrating a configuration of an apparatus for generating a neural symbol based rule according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

The present invention relates to a technique for generating a rule for inferring a user's intention based on perceptual information including actions and postures obtainable in a user's living environment using an embedding method.

In particular, the present invention is suitable for inferring the intention of an elderly person, and a method for generating a rule for inferring an elderly person's intention will be described in detail below.

Prior to the description, a Neural Theorem Prover (NTP) method, which is one of the methods for extracting rules from the knowledge base, will be described below.

The NTP method is a method of completing knowledge through a similarity between entities and relations from a knowledge base, and learning rules through a given rule template.

In the NTP method, a triple that satisfies the rule template for a given triple is learned through the unification process by using the backward chaining method to obtain the similarity to the entity of the knowledge base. In this method, not only triples satisfying the rule but also triples having similar meanings can be obtained.

In two triple data, the similarity of subject, object, and predicate in each triple is calculated respectively.

Using the similarity learning of triple data, the NTP method completes the rules to derive a given target triple by modularizing the three methods of AND, OR, and Unify of the Backword Chaining method.

2, when the target triple is grandFatherOf(ABE, BART), and the rule template is #1(X, Y) :- #2(X, Z), #3(Z, Y). A satisfactory rule template is found through the triple existing in the knowledge base.

In more detail, the similarity between the grandFatherOf(ABE, BART) triple and #1(X, Y) is calculated, and since X and Y are variables in #1(X, Y), the ABE and BART of the target triple are respectively It is bound as X=ABE, Y=BART, and the similarity between #1 and grandFatherOf is obtained.

The following compares with the triple existing in the knowledge base for #2(ABE, Z) according to the value of the bound variable for #2(X, Z) of the rule template.

Therefore, the similarity between #2(X, Z) and the first triple of the knowledge base, fatherOf(ABE, HOMER) is calculated.

Find the similarity between #2 and fatherOf, and find the similarity between the ABE bound to X and the ABE of the first triple.

Finally, since Z is a variable, the value of HOMER in the first triple is bound as Z=HOMER. In this way, it is a method to learn to have a high degree of similarity to a given target triple through comparison with all triples in the knowledge base.

The existing NTP method has a disadvantage in that it takes a lot of time as the knowledge base grows.

In addition, a study using the event calculation method to predict the intention of the elderly from perceptual information including behaviors and postures collected from the elderly showed good performance, but it is expensive to utilize the data collected over a long period of time. There is this.

Here, the intention of the elderly person may include eating (Meal), cleaning (Cleaning), reading (Reading), communication and watching TV (Watching TV).

In the present invention, in order to infer the intention of the elderly person from the recognized perception information, a rule is generated by learning the relationship with the perception information, and the intention of the elderly person is inferred through the generated rule.

3 shows, as an example, an ontology related to the relationship with 5 intentions (eating, reading, cleaning, communication, leisure) with 55 types of behaviors and 10 types of postures for the perception information of the elderly.

The defined ontology expresses the relationship between behavior and posture, which are perceptual information, in a triple form.

As such, FIG. 3 shows an example of an ontology schema and instance for the relationship between the behavior and posture bases and the five types of intentions.

Perceptual information collected from the living environment of the elderly is converted into triple-type data through a defined ontology, and the transformed triple data becomes an input value of the NTP-based rule generation model for rule generation that well represents the intention of the elderly.

As shown in FIG. 4, the system according to this embodiment includes a perceptual information collection unit 400, an ontology storage unit 402, a preprocessor 404, an embedding unit 406, and a Neural Theorem Prover (NTP, 408). and a rule generator 410 .

The perceptual information collecting unit 400 collects perceptual information including behavior and posture in the living environment of the elderly.

The ontology storage unit 402 stores an ontology defining the relationship between the intention and perception information of the elderly.

The preprocessor 404 converts the collected perceptual information into triple data using the ontology.

The triple data in which the perceptual information is converted is stored in the knowledge base (KB).

The embedding unit 406 embeds the transformed triple data in a multidimensional space for learning a rule based on a neural symbol.

Through the embedding process, entities and relations of triple data are expressed as vectors.

The NTP 408 performs a learning process to create a rule based on a neural symbol, and learns a rule in which an intention is derived from embedded perceptual information through AND, OR, Unify, and Aggregation modules using the Backword Chaining method.

When the target triple expressing the intention of the elderly person is input, the NTP 408 updates the parameter so that the triple representing the perceptual information representing the characteristics of the target triple is learned.

The rule generator 410 generates a rule having one or more perceptual information indicating the characteristics of the target triple as a condition through the parameter update.

Here, the rule template includes a rule head corresponding to a conclusion and a rule body corresponding to a condition, wherein the rule head and rule body are composed of triple data of a predicate corresponding to a relation, a subject corresponding to an entity, and an object, A triple for the perceptual information may be arranged in the rule body.

It is also defined as the process of updating the embedding vector value of the parameter corresponding to the object of the rule template.

As described above, in this embodiment, in the predefined rule template, triples for behavior and posture, which are perceptual information, are configured so that the condition (rule body) of the corresponding rule can come.

That is, as a method for inferring the intention of the elderly while solving the problem of processing a large amount of data, which was a disadvantage of the existing NTP method, each element of perceptual information is configured to come to the condition of the rule.

In addition, in order to derive the perception information related to the intention of the elderly, the intention according to the perception information of each elderly person is learned through similarity learning between entities for actions and postures.

In the present invention, unlike the conventional method, the relation (predicate) of the target triple expresses the intention, the relation of the remaining rule conditions consists of actions and postures, the subject is an entity for the user (elderly person), and the object is a specific action and posture Consists of a perceptual information entity comprising

According to the entity for the elderly who can come to the subject, the perceptual information entities constituting the object are searched through the unify process according to the target triple to collect the entity candidates for similarity comparison.

The collected entity candidate group consists of perceptual information entities collected in the living environment of the elderly.

Also, according to the present embodiment, a weight matrix for the perception information that occurs with high frequency in each intention according to the elderly is used so that the similarity of the perception information according to each intention is learned to be high.

The NTP-based rule generation model according to the present embodiment receives a target triple of hasIntention(KIM, MEAL) as a process for learning relevant perceptual information in order to derive the eating (MEAL) intention of the elderly KIM.

The NTP-based rule creation model is an entity that satisfies the variable X by performing the target triple and rule template hasIntention(X, #1_0_0) :- hasAction(X, #2_0_0), hasPose(X, #3_0_0). to collect intent and perceptual information entities similar to #1, #2, and #3.

Here, #1, #2, and #3 are a first parameter for an object corresponding to the user's intention, respectively, and second and third parameters for an object corresponding to each of the user's actions and postures.

The object corresponding to the user's behavior is defined as behavior perception information, and the object corresponding to the posture is defined as attitude perception information.

Therefore, KIM is bound to variable X by comparing hasIntention(KIM,MEAL) and hasIntention(X, #1_0_0), which are the target triples, and the similarity between #1_0_0 and the intention entity (MEAL entity) is obtained.

Next, the condition of the rule template, hasAction(X, #2_0_0), is compared with all behavior perception information collected from the elderly.

In FIG. 5, since it is assumed that there are two types of behavior perception information, hasAction(X, #2_0_0), hasAction(KIM, Eating Food), and hasAction(KIM, Using a gas stove) are compared, and the value of X as an entity (variable) Because it is bound to this KIM, we compare the behavioral perception information where the subject's entity is the KIM.

Therefore, the similarity between #2_0_0 and Eating Food, which is one of the behavior perception information, is obtained, and the similarity between #2_0_0 and Using a gas stove, one of the behavior perception information, is obtained.

Since a plurality of actions cannot be taken at the same time for each action, a path of the tree is generated as shown in FIG. 5 . Finally, hasPose(X, #3_0_0) of the rule template is compared with the posture perception information collected from the elderly.

At this time, similar to the behavioral perception information of the elderly, matching posture perception information is searched through the entity bound to the variable X.

In FIG. 5, hasPose(X, #3_0_0) and hasPose(KIM, Sitting on the floor), hasPose(KIM, Sitting on a chair), and hasPose(KIM, Standing) are compared, and the similarity between #3_0_0 and the collected posture perception information will save

Since the obtained similarity information needs to be highly learned for the perceptual information related to the intention, which is the target triple, the similarity must be obtained while the path for each perceptual information is maintained as shown in FIG. 5 .

In addition, in order to learn perceptual information related to the corresponding intention, learning is performed by applying a frequency-based weighting matrix of the perceptual information related to the intention according to each elderly person.

Referring to FIG. 6 , behaviors that appear frequently are behaviors that can be characterized in each intention, and must be learned so that the value is high when calculating the degree of similarity to the intention.

In order to similarly learn the perceptual information related to the intention, each intention was configured in a column, and the behavior and posture information related to each intention was configured in a matrix consisting of rows.

7 shows a perceptual information intention matrix according to intention.

As shown in FIG. 7 , the first row of the intention matrix means eating (MEAL), and in each column of the first row, a high weight value is applied to the part of perceptual information on the behavior and posture related to the intention to eat.

As shown in FIG. 8 , the rule generating apparatus according to the present embodiment is an apparatus for performing the processes of FIGS. 4 to 5 , and may include a processor 800 and a memory 802 .

The processor 800 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 802 may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 802 may also include volatile memory, such as various random access memories.

The memory 802 stores program instructions executable by the processor 800 .

The program instructions according to this embodiment collect perceptual information including behavior and posture from the user's living environment, and convert the collected perceptual information into triple data using an ontology defined for the relationship between the user's intention and perceptual information. Transformation, embedding a vector about entities and relations of the transformed triple data into a multidimensional space, and when a target triple expressing the user's intention is input, through the AND, OR, and Unify modules of Neural Theorem Prover (NTP) A parameter of a preset rule template is updated, and a rule having one or more perceptual information indicating characteristics of the target triple as a condition is generated through the updating of the parameter.

As described above, the rule template includes a rule head corresponding to a conclusion and a rule body corresponding to a condition, wherein the rule head and rule body are triple data of a predicate corresponding to a relation, a subject corresponding to an entity, and an object. configured, and a triple for the perceptual information may be arranged in the rule body.

In addition, the rule head may include a first parameter for an object corresponding to the user's intention, and the rule body may include second and third parameters for an object corresponding to each of the user's actions and postures, The embedding vector values of the first to third parameters are updated through the learning process.

The above-described embodiments of the present invention have been disclosed for purposes of illustration, and various modifications, changes, and additions will be possible within the spirit and scope of the present invention by those skilled in the art having ordinary knowledge of the present invention, and such modifications, changes and additions should be considered as belonging to the following claims.

Claims

As a neural symbol-based rule generator for user intention inference,

processor; and

a memory coupled to the processor;

The memory is

Collecting perceptual information including behavior and posture from the user's living environment,

The collected perceptual information is converted into triple data using an ontology defined for the relationship between the user's intention and perceptual information,

Embedding the vector about the entity and relation of the transformed triple data in a multidimensional space,

When a target triple expressing the user's intention is input, the parameters of the preset rule template are updated through the AND, OR, Unify, and Aggregation modules of Neural Theorem Prover (NTP),

to generate a rule conditional on one or more perceptual information indicative of the target triple through updating of the parameter;

A neural symbol-based rule generating apparatus including program instructions executable by the processor.
According to claim 1,

The rule template includes a rule head corresponding to a conclusion and a rule body corresponding to a condition, wherein the rule head and rule body are composed of triple data of a predicate corresponding to a relation, a subject corresponding to an entity, and an object; A neural symbol-based rule generating apparatus in which a triple for the perceptual information is disposed in a rule body.
3. The method of claim 2,

The rule head includes a first parameter for an object corresponding to the user's intention,

The rule body is a neural symbol-based rule generating apparatus including second and third parameters for an object corresponding to each of the user's actions and postures.
3. The method of claim 2,

The program instructions are a neural symbol-based rule generating apparatus for updating an embedding vector value of a parameter corresponding to an object of the rule template.
4. The method of claim 3,

The program instructions are

calculating the similarity between the object of the target triple and the first parameter,

calculating a degree of similarity between the collected perceptual information and the second and third parameters;

A neural symbol-based rule generator for generating a tree path from the target triple to the action and posture triple.
3. The method of claim 2,

The subject of the rule template is a neural symbol-based rule generating device that is bound to a specific user.
According to claim 1,

The program instructions are

An apparatus for generating a neural symbol based rule using a weight matrix based on a frequency of perceptual information related to a user's intention to update the parameter.
A method for generating a rule based on neural symbols for inference of user intent in a device comprising a processor and a memory, the method comprising:

collecting perceptual information including behavior and posture from the user's living environment;

converting the collected perceptual information into triple data using an ontology defined for a relationship between user intention and perceptual information;

embedding a vector regarding entities and relations of the transformed triple data into a multidimensional space;

updating parameters of a preset rule template through AND, OR, Unify, and Aggregation modules of Neural Theorem Prover (NTP) when a target triple expressing the user's intention is input; and

and generating a rule having one or more perceptual information indicating characteristics of the target triple as a condition through updating the parameter.
A computer readable program for performing the method according to claim 8 .
A system for generating a neural symbol based rule for inferring user intent, comprising:

Perceptual information collecting unit including behavior and posture from the user's living environment;

an ontology storage unit for storing an ontology defining a relationship between user intention and perceptual information;

a preprocessor for converting the collected perceptual information into triple data using the ontology;

an embedding unit for embedding a vector related to an entity and a relation of the transformed triple data into a multidimensional space;

When a target triple expressing the user's intention is input, the parameters of the preset rule template are updated through AND, OR, and Unify modules Neural Theorem Prover (NTP); and

and a rule generator configured to generate a rule having one or more perceptual information indicating the characteristics of the target triple as a condition through the update of the parameter.