WO2023017930A1

WO2023017930A1 - Method and device for inferring behavioral intention on basis of user feedback

Info

Publication number: WO2023017930A1
Application number: PCT/KR2022/000299
Authority: WO
Inventors: 박영택; 김제민; 박현규; 신원철
Original assignee: 숭실대학교산학협력단; 명지대학교 산학협력단
Priority date: 2021-08-13
Filing date: 2022-01-07
Publication date: 2023-02-16
Also published as: KR20230025141A

Abstract

Disclosed in the present invention are a method and a device for inferring a behavioral intention on the basis of user feedback. According to the present invention, provided is a device for inferring a behavioral intention comprising: a processor; and a memory coupled to the processor, wherein the memory includes program instructions executable by the processor that are configured to: constitute a plurality of event operation rules consisting of a precondition corresponding to perceptual information and a conclusion corresponding to a behavioral intention, wherein at least some of the plurality of event operation rules include an event operation rule consisting of a plurality of conclusions for the same precondition, each of the plurality of conclusions at an initial time point having the same probability value; sequentially collect perceptual information including at least one of behavior, posture, object, and sound from a user's living environment over time; when there are a plurality of behavioral intentions inferred through the perceptual information collected at a first time point, select one of the plurality of behavioral intentions according to a probability distribution of each of the plurality of behavioral intentions; and adjust the probability value of each of the plurality of behavioral intentions on the basis of feedback received from the user with respect to the selected behavioral intention.

Description

Method and apparatus for inferring action intention based on user feedback

The present invention relates to a method and apparatus for inferring action intention based on user feedback.

Services according to the situation by identifying the behavioral intentions of users, especially the elderly, are expected to contribute to improving the quality of life, such as supplementing the abilities of the elderly, and are being studied by many companies and universities.

Among them, the behavior and posture information that changes according to the time of occurrence and end of the user's action and pose are used to determine the user's behavior intention by using an inference engine based on event calculation rules. A representative example is a study that infers .

In order to infer the user's action intention, an event operation rule is constructed based on the action intent ontology that specifies the action intention, behavior, posture, and relationship with surrounding objects and places.

In addition, in order to clearly infer the end point of action intention, the user's current location is inferred based on surrounding objects and heuristics are applied.

Based on this technology, when percept information defined by behavior, posture, and surrounding objects is input, the intention of the current user's action is inferred through event operation rules.

This action recognition method has the advantage of not requiring learning data to build a model used for inference compared to the method applying machine learning.

However, in the prior art, it is very difficult to reflect each individual's habits reflected in the intention because the intention of the action is inferred using a fixed event operation rule generated by specified knowledge. In addition, when an action intention is misrecognized, the same recognition error may continue because it is not optimized so that it can be correctly inferred later.

Because of this, the problem of increasing the error of recognizing the intention of the action occurs, which causes the defect rate to increase when applied to the product.

In order to solve the problems of the prior art, the present invention proposes a user feedback-based action intention inference method and apparatus that can reflect each individual's habit.

In order to achieve the above object, according to an embodiment of the present invention, an action intention inference device based on user feedback includes a processor; and a memory connected to the processor, wherein the memory constitutes a plurality of event operation rules composed of a precondition corresponding to perception information and a conclusion corresponding to an action intention, and at least some of the plurality of event operation rules Include an event operation rule consisting of a plurality of conclusions for the same precondition, each of the plurality of conclusions at an initial time point has the same probability value, and includes at least one of behavior, posture, object, and sound from the user's living environment Perceptual information is sequentially collected according to time, and when there are multiple action intentions inferred through the perception information collected at a first point in time, one of the plurality of action intentions is selected according to a probability distribution of each of the plurality of action intentions; An action intention inference apparatus including program instructions executable by the processor is provided to adjust a probability value of each of the plurality of action intentions based on a feedback received from a user for the selected action intention.

The program instructions may adjust a probability value of each of a plurality of action intentions through the following equation.

[mathematical expression]

Here, P(x) is an existing probability value applied to a predetermined action intention, P(x)' is a new probability value adjusted,

is the probability control value

The probability adjustment value may be determined according to the number of action intentions having the same precondition and the number of feedbacks.

If there is a first action intention and a second action intention having the same precondition in the initially set event operation rule, the first action intention is inferred through the perception information collected at the first point in time, and positive feedback is obtained from the user. If delivered, the probability of the first action intent

At this time, the probability of the second action intention having the same precondition is

can be reduced as much as

The program instructions identify a plurality of places corresponding to the sequentially collected perception information using an object place mapper that stores information corresponding to a plurality of objects and places where each of the plurality of objects can be located, and , each of the plurality of action intentions using at least one of the action intention collision information defining a relationship between action intentions that cannot be simultaneously generated, the average maintenance time for each of the inferred plurality of action intentions, and the identified plurality of places end point can be determined.

The program instructions count places corresponding to each of a plurality of objects included in perception information collected at a first time point based on the object place mapper, and determine a place having the highest association score according to the counting at the first time point. can be identified by its place in

The program instructions may include an action intention inferred in a first period using at least one of the action intent collision information, the average maintenance time, and the identified plurality of places, and an action intent inferred in a second period subsequent to the first period. An action intent can be determined as a composite action intent that can occur simultaneously.

According to another aspect of the present invention, a method of inferring an action intention in a device including a processor and a memory, comprising the steps of constructing a plurality of event operation rules composed of preconditions corresponding to perceptual information and conclusions corresponding to the action intention. - at least some of the plurality of event calculation rules include event calculation rules composed of a plurality of conclusions for the same precondition, and each of the plurality of conclusions has the same probability value at an initial time point -; sequentially collecting perceptual information including at least one of behavior, posture, object, and sound from a user's living environment according to time; selecting one of the plurality of action intentions according to a probability distribution of each of the plurality of action intentions when there are multiple action intentions inferred through the perception information collected at the first point in time; and adjusting a probability value of each of the plurality of action intentions based on a feedback received from a user for the selected action intention.

According to another aspect of the present invention, a computer readable program for performing the method described above is provided.

According to the present invention, since the case operation rule-based inference method is used, learning data for learning the model required for reasoning is not required, probability is applied to the case operation rule for action intention inference, and inference probability is calculated based on feedback. Because it is adjustable, it has the advantage of being able to reflect the habits of each individual to some extent.

1 is a diagram showing the configuration of an action intention inference device according to a preferred embodiment of the present invention.

2 is a diagram illustrating an example of an ontology expressing the relationship between perception information and action intention.

3 is a diagram illustrating action intention conflict information in the form of a graph.

4 is a diagram illustrating behavior patterns of the elderly.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The present embodiment relates to a method of gradually reinforcing an event caculus for inferring an action intention through feedback from a user in order to reflect an individual's habit in inferring an action intention based on the event caculus.

Here, the event operation rule is a form of reasoning for action and change, and is a rule for inferring change in intention fluent according to events that occur over time.

According to the present embodiment, the event operation rule infers a change in action intention by using information of actions, postures, surrounding objects, and sound occurring in chronological order as events.

Here, the fluent is defined as a first-order logical term or predicate whose value changes with time.

As shown in FIG. 1 , the device according to the present embodiment may include a processor 100 and a memory 102 .

The processor 100 may include a central processing unit (CPU) capable of executing a computer program or other virtual machines.

Memory 102 may include a non-volatile storage device such as a non-removable hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, and the like. Memory 102 may also include volatile memory, such as various random access memories.

Program instructions executable by the processor 100 are stored in such a memory 102 .

The action intention inference apparatus according to the present embodiment sequentially collects perception information including actions, postures, objects, and sounds from the user's living environment through program commands, and applies the collected perception information to predefined event operation rules. to infer the intent of the action.

To this end, the device according to the present embodiment stores in advance a plurality of event operation rules composed of preconditions corresponding to perception information and conclusions corresponding to action intentions.

In general, an event operation rule is composed of one or more preconditions and conclusions, and the conclusion when one or more preconditions are satisfied is defined as one. However, individual habits cannot be reflected through these event operation rules.

According to this embodiment, at least some of the plurality of event calculation rules include event calculation rules composed of a plurality of conclusions for the same precondition, and each of the plurality of conclusions is set to have the same probability value at an initial point in time.

For example, "cleaning" and "preparing a meal" may be included as action intentions that are conclusions of preconditions such as picking up food with a spoon/fork and washing dishes, and the same probability value for these user's action intentions is granted

When there are a plurality of action intentions inferred through the perception information collected at the first point in time, the action intention inference device according to the present embodiment selects one of a plurality of action intentions according to a probability distribution of each of the plurality of action intentions and informs the user of the action intention. output, and receives feedback from the user to adjust a probability value of each of a plurality of action intentions.

The action intention inference device outputs a plurality of action intentions inferred through the event operation rule to the user at an initial point in time, receives feedback from the user, and adjusts the probability value of each of the plurality of conclusions for the same precondition as follows.

Here, P(x) is the original probability value applied to the conclusion, P(x)' is the adjusted new probability value,

is a probability adjustment value.

The larger , the higher the optimization speed, but the lower the probability of finding the optimal probability.

For example, when a first action intention and a second action intention having the same precondition exist in an initially set event operation rule, the probabilities of the first and second action intentions have the same value (eg, 50). The first action intention is inferred through the perception information collected at a specific point in time and the probability of the first action intention is determined when positive feedback is received from the user.

reduce as much

Conversely, when receiving negative feedback, the probability of the first action intention

and the probability for the second action intention is

increase as much

According to this embodiment,

may have a fixed value determined experimentally and may be preferably set to a value of 5 (0.05 as a percentage).

However, depending on the number of feedback received and the number of conclusions with the same precondition, without being limited thereto

may be dynamically determined.

Perceptual information is information collected from the user's living environment, and actions collected from the user's living environment refer to various movements that the user can perform in the living environment, such as lifting a spoon/chopsticks, opening and closing the refrigerator, and cutting a knife. This means using, drinking water, sitting/getting up from a chair (sofa), washing the floor, making a phone call, etc.

The posture refers to a pose that the user takes, such as standing, lying down, sitting, squatting, and the like.

Objects refer to various living tools, home appliances and furniture provided in the user's living environment, such as TVs, refrigerators, gas ranges, remote controls, mobile phones, vacuum cleaners, sofas, dining tables, wardrobes, beds, chairs, sticks, chopsticks, books, It may contain objects such as newspapers.

Sound is a variety of sounds generated in a living environment, and may include TV sound, vacuum cleaner sound, dishwashing sound, and the like.

Perceptual information according to the present embodiment may include not only information on behavior, posture, object, and sound, but also information on time and on/off of the object when the object is a home appliance.

Next, the action intention inference device infers the user's action intention corresponding to the sequentially collected perception information using an ontology defining the relationship between the user's action intention and the perception information.

As shown in FIG. 2, the ontology defines a data relationship between an action intention and actions, postures, objects, and sounds in a form that can be shared.

Inference of an action intention using an ontology may mean that an event generated from perceptual information such as actions, postures, objects, and sounds is a prerequisite, and action intention is a result of a rule.

The action intention inference apparatus according to the present embodiment uses a plurality of objects corresponding to perceptual information sequentially collected by using an object place mapper that stores information in which a plurality of objects and places where each of the plurality of objects can be located are stored. identify a place

For example, the place identification may be identifying a place as a kitchen when a refrigerator is included in the perception information, a living room when a sofa is included, and a multipurpose room when a washing machine is included.

According to the present embodiment, the identification of the place can be performed through the following correlation score calculation formula, and based on the object place mapper, the place corresponding to each of the plurality of objects included in the perception information collected at the current time is counted and each The association score (counting score) for the place is calculated, and the place with the highest association score is determined as the place at the current time.

After inferring a plurality of action intentions over time and identifying a plurality of places, the device for inferring action intentions determines each of the plurality of action intentions by using the action intention collision information defining the relationship between action intentions that cannot occur simultaneously. determine the end point.

Action intention collision information is information stored by matching one action intention with one or more other action intentions that are difficult to occur simultaneously.

For example, one or more action intentions such as cleaning, washing dishes, and taking a bath may be stored as being in conflict with the action intention of “eating”.

In addition, the action intention conflict information may have a graph form, and FIG. 3 is a diagram illustrating the action intention conflict information in a graph form.

Referring to FIG. 3 , various action intentions may be expressed in a vector space, and whether or not action intentions collide may be determined based on a Euclidean distance in the vector space.

The action intention inference apparatus according to the present embodiment may use the average maintenance time of each action intention to determine the end of each of a plurality of action intentions.

Here, the average maintenance time may be the maintenance time of each action determined using perception information collected from a specific user.

For example, the average maintenance time of an act of eating may be determined to be 60 minutes, watching TV to 50 minutes, and reading to be 30 minutes.

The average maintenance time may be a time obtained from a user's action as a current inference target, but is not limited thereto and may be a time obtained from a specific age or older, that is, a general elderly person.

In addition, the action intention inference apparatus according to the present embodiment may determine whether or not the previous action intention has ended by using whether or not the place moves over time.

For example, a user who was sitting at the table and holding chopsticks for a time from t ₁ to t ₂ (first period) moves to a room at t ₂ and sits down until t ₃ (t ₂ to t ₃ : second period) and turns on the TV When sitting, the action intention inference device infers the action intention of eating for the first period and sleeping (or resting) for the second period, the consecutive action intentions conflict with each other, and the first period is eating. It is within the average holding time of the action of doing, and the intention of the action to eat at time t ₂ is completed by using the fact that the places identified as objects included in the perception information in the first period and the second period are identified as a kitchen and a room. can be determined as

In addition, a first action intention inferred in a first period using at least one of the action intention collision information, the average maintenance time, and the identified plurality of locations and a second action intention inferred in a second period consecutive to the first period An action intent can be determined as a composite action intent that can occur simultaneously.

For example, when a user eats a meal in the living room from t ₁ to t ₂ (the first period) and then turns on and watches a TV in the living room from t ₂ to t ₃ (the second period), according to the present embodiment The device is an action intention in which eating and watching TV do not conflict with each other, the first period is within the average holding time of eating, and the location has not moved, so that the previous action intention does not end at t _{2 and} multiple It detects that there is a compound action intent in which user actions occur simultaneously.

According to the present invention, through the determination of the end point of a plurality of action intentions and the detection of complex action intentions as described above, it is possible to accurately grasp action intentions continuously appearing over time. In particular, as shown in FIG. 4, the elderly By checking the daily life pattern of the elderly, it is possible to detect abnormal signs that occur in the elderly in advance.

The embodiments of the present invention described above have been disclosed for illustrative purposes, and those skilled in the art having ordinary knowledge of the present invention will be able to make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions will be considered to fall within the scope of the following claims.

Claims

As an action intention inference device based on user feedback,

processor; and

Including a memory coupled to the processor,

the memory,

Constituting a plurality of event operation rules composed of preconditions corresponding to perceptual information and conclusions corresponding to action intentions;

At least some of the plurality of event operation rules include event operation rules consisting of a plurality of conclusions for the same precondition, each of the plurality of conclusions having the same probability value at an initial time point,

Sequentially collecting perceptual information including at least one of behavior, posture, object, and sound from the user's living environment over time;

When there are a plurality of action intentions inferred through the perception information collected at the first point in time, selecting one of the plurality of action intentions according to a probability distribution of each of the plurality of action intentions;

To adjust a probability value of each of the plurality of action intentions based on feedback received from a user for the selected action intention,

An action intention inference device comprising program instructions executable by the processor.
According to claim 1,

The program instructions control the probability value of each of a plurality of action intentions through the following equation.

[mathematical expression]

Here, P(x) is an existing probability value applied to a predetermined action intention, P(x)' is a new probability value adjusted,
is the probability control value
According to claim 2,

The probability adjustment value is determined according to the number of action intentions having the same precondition and the number of feedbacks.
According to claim 2,

The program instructions are

If there is a first action intention and a second action intention having the same precondition in the initially set event operation rule, the first action intention is inferred through the perception information collected at the first point in time, and positive feedback is obtained from the user. If delivered, the probability of the first action intent
At this time, the probability of the second action intention having the same precondition is
An action intention inference device that reduces as much as
According to claim 1,

The program instructions are

Identifying a plurality of places corresponding to the sequentially collected perception information using an object place mapper that stores information in which a plurality of objects and places where each of the plurality of objects can be located correspond to each other;

Each of the plurality of action intentions is determined by using at least one of action intention collision information defining a relationship between action intentions that cannot be simultaneously generated, an average maintenance time for each of the inferred plurality of action intentions, and the identified plurality of locations. An action intention inference device that determines when to end.
According to claim 1,

The program instructions are

Counting the places corresponding to each of the plurality of objects included in the perception information collected at a first point in time based on the object place mapper,

An action intention inference device for identifying a place having the highest association score according to counting as a place at the first point in time.
According to claim 1,

The program instructions are

An action intention inferred in a first period using at least one of the action intention collision information, the average maintenance time, and the identified plurality of locations may occur simultaneously with an action intention inferred in a second period subsequent to the first period. An action intention inference device that determines a complex action intent that can be performed.
A method of inferring an action intention in a device including a processor and a memory,

A step of constructing a plurality of event operation rules consisting of preconditions corresponding to perceptual information and conclusions corresponding to action intentions - at least some of the plurality of event operation rules are composed of a plurality of conclusions for the same precondition. Including, each of the plurality of conclusions at an initial time point has the same probability value;

sequentially collecting perceptual information including at least one of behavior, posture, object, and sound from a user's living environment according to time;

selecting one of the plurality of action intentions according to a probability distribution of each of the plurality of action intentions when there are multiple action intentions inferred through the perception information collected at the first point in time; and

and adjusting a probability value of each of the plurality of action intentions based on a feedback received from a user for the selected action intention.
According to claim 8,

In the step of adjusting the probability value, the action intention inference method of adjusting the probability value of each of the plurality of action intentions through the following equation.

[mathematical expression]

Here, P(x) is an existing probability value applied to a predetermined action intention, P(x)' is a new probability value adjusted,
is the probability control value
A computer readable program for performing the method according to claim 8 .