KR100982164B1 - Method and device for recognition of human activity based on activity theory - Google Patents

Abstract

Disclosed are a behavior theory based behavior recognition method and apparatus. Behavior theory-based behavior recognition apparatus for recognizing user's behavior according to an aspect of the present invention is a motion recognition unit for recognizing any one of the predefined operation (operation) using the measured external physical data as the user's motion ; And an action determination unit that determines an observation behavior of the user corresponding to one or more cognitive actions recognized by the action recognition unit by using learning data that is information about an action consisting of one or more actions. According to the present invention, an action recognition method and apparatus based on a behavior theory divided into an activity, an action, and an action, and based on an action theory that can be processed in a device by limiting an action that is the most basic action of a person's action Can provide.

Action, theory of action, action, action, learning

Description

Method and device for recognition of human activity based on activity theory

The present invention relates to whether the user's behavior, and more particularly, to a method and apparatus for behavior theory based behavior recognition.

Ubiquitious is a Latin word meaning 'anywhere, anytime', and the ubiquitous computing environment means a computing environment that allows users to use spatially connected devices over the Internet without restriction of communication speed anytime, anywhere. to be. This ubiquitous computing environment is one step further than the existing home network or mobile computing.

As an example of a ubiquitous computing environment, a home network, for example, adds a computing function to all physical spaces and objects related to a house such as a washing machine, a refrigerator, a TV, a table, and intelligently connects objects and objects to each other to connect information to each other. One can give and receive.

Context-aware system, a basic technology of ubiquitous computing, effectively shares various environmental information collected by sensors and computers in various places of daily life in order to provide services that users need. It requires situational awareness processing techniques to find out.

In particular, in order to provide a service corresponding to the current behavior of the user, it is necessary to determine what the current user is doing. In other words, it is because the user's current behavior must be recognized to provide the service accordingly. For example, when a user is preparing dinner, the window can be automatically opened to recognize and ventilate.

As techniques for recognizing the user's behavior, the conventional techniques can be largely classified into a rule-based system and a probability model system. Rule-based systems are easy to implement but do not adequately handle exceptional behavior and suffer from scalability. This is because the rules of knowledge should be specified in the knowledge base if they can be interpreted or for behaviors that require recognition. The probabilistic model recognizes behavior by creating a statistical model based on the user's history of behavior and classifying a given set of behaviors based on it. Most systems use hidden Markov models or Conditional Random Field (CRF) methods. However, such a model works well for sequential situations but does not cope with exceptional behavior. That is, there is a high probability that the user's specific purpose of action (for example, meal preparation, etc.) is not always performed in the same order of action, and thus the exceptional behavior is not accurately recognized in the conventional manner.

Moreover, the above-described conventional method can recognize a user's behavior only in a middleware or an offline system, and thus can not be used in real time because it is implanted in a portable device such as a mobile phone.

Accordingly, the present invention has been made to solve the above-described problems, and to provide a method and apparatus for recognizing a behavior based on a behavioral theory divided into activities, behaviors, and behaviors.

It is also an object of the present invention to provide a method and apparatus for recognizing a faster processing speed to be implemented in a portable device.

In addition, the present invention is to provide a more accurate behavior recognition method and apparatus by adding a penalty according to the action and time in the behavior recognition.

Other objects of the present invention will become more apparent through the preferred embodiments described below.

According to an aspect of the present invention, there is provided a device for recognizing a user's behavior, the apparatus comprising: a motion recognition unit for recognizing any one of a predefined operation as a user's motion by using measured external physical data; And an action determination unit that determines a user's observed behavior corresponding to one or more cognitive actions recognized by the action recognition unit by using learning data that is information about an action consisting of one or more actions. Behavioral awareness devices are provided.

In this case, the operation may be recognized as any one of a stay, a hold, a move, a touch, and a release.

The behavior determining unit may include a behavior learning unit for generating the learning data using at least one learning example data; A learning information storage unit for storing the learning data; And a learning cognition unit for recognizing the observing behavior of the user corresponding to the cognitive motion using the learning data.

, 상기 행동이 수행될 때 임의의 동작이 발생할 확률

, 행동 시간, 행동을 이루는 동작의 개수 및 세트(행동-동작 세트)를 포함하는 학습 데이터를 생성할 수 있다.And, the behavior learning unit using the learning example data using the probability of occurrence according to each action

, The probability that any action will occur when the action is performed.

Learning data can be generated, including the action time, the number of actions that make up the action, and the set (action-action set).

는 복수개의 상기 학습예제 데이터를 이용한 포아송 분포를 이용하여 산출될 수 있다.In addition,

May be calculated using a Poisson distribution using a plurality of the learning example data.

In addition, the behavior recognition unit may recognize the observed behavior by applying a penalty calculated by using the inconsistency between the behavior-action set corresponding to each behavior of the learning data and the behavior set of the cognitive behavior to each behavior. The number of discrepancies can be calculated by subtracting the action set of the cognitive action from the action-action set.

는 상기 학습 데이터의 상기 행동-동작 세트와 상기 인지 동작의 동작 세트의 불일치 수를 이용하여 계산될 수 있다.In addition, the behavior recognition unit recognizes the observed behavior by using the following equation, the penalty function of the following equation

Can be calculated using the number of inconsistencies of the behavior-action set of the learning data and the behavior set of the cognitive action.

The discrepancy number may also use an action set of perceived actions during the same time interval as the action time of the learning data, according to a fixed action time scheme. Alternatively, the number of inconsistencies may be calculated using an action set of recognized actions for a time interval extended within a predetermined range than the action time of the learning data, according to an extended action time scheme.

는 하기의 수학식에 따라 상기 불일치 수에 따른 불일치 패널티와 함께 상기 확장된 시간만큼의 시간 패널티를 이용할 수 있다.In addition, the penalty

Can use the time penalty of the extended time together with the discrepancy penalty according to the discrepancy number according to the following equation.

는 확장 시간에 따른 시간 패널티,

는 불일치 수에 따른 불일치 패널티)(here,

Is the time penalty over expansion time,

Is a penalty for discrepancies based on the number of discrepancies)

The behavior determiner may further include a feedback processor configured to control the behavior learner to update the learning data to correspond to the feedback data input by the user according to the observed behavior.

In addition, the motion recognition unit may recognize the measured external physical data as any one of the defined motions through time series analysis according to the passage of time.

The behavioral theory-based behavioral recognition device may further include a measurement unit for measuring the external physical data of any one or more of RFID tag information, GPS location information, and acceleration information.

According to another aspect of the present invention, there is provided a method for recognizing a user's behavior, the method comprising: recognizing any one of a predefined operation as a user's motion by using measured external physical data; And determining the observed behavior of the user corresponding to one or more cognitive motions recognized by the motion cognition unit by using learning data that is information about an action consisting of one or more motions. There is provided a recording medium on which a recognition method and a program for executing the method are recorded.

, 상기 행동이 수행될 때 임의의 동작이 발생할 확률(

) , 행동 시간, 행동을 이루는 동작의 개수 및 세트(행동-동작 세트)를 포함하는 상기 학습 데이터를 생성하는 단계가 선행될 수 있다.Here, the occurrence probability according to each action using a plurality of learning example data including the action name, the action information, and the action execution time

, The probability that any action will occur when the action is performed (

Generating the learning data, the action time, the number of actions that make up the action, and the set (action-action set).

The determining of the observed behavior of the user may include: calculating a penalty by comparing a number of inconsistencies between an action-action set according to each pre-stored action and an action set recognized by the motion recognition unit; And applying the penalty to recognize the observation behavior corresponding to the cognition operation.

The method may further include displaying information according to the observed behavior. When feedback data is input from a user, the learning data may be updated using the feedback data.

According to the present invention, an action recognition method and apparatus based on a behavior theory divided into an activity, an action, and an action, and based on an action theory that can be processed in a device by limiting an action that is the most basic action of a person's action Can provide.

In addition, according to the present invention, it is possible to simplify the calculation method for behavior recognition by setting the learning data in advance, so that the present invention can be applied to a device having a small processing capability such as a portable device.

In addition, according to the present invention it is possible to provide a more accurate and reliable behavior recognition method by adding a penalty according to the operation and time.

In addition, the present invention can provide a method and apparatus for recognizing a behavior capable of learning its own behavior by receiving feedback from a user from a recognized behavior and updating the learning data.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in describing the present invention with reference to the accompanying drawings, the same or corresponding elements are denoted by the same reference numerals regardless of the reference numerals, and duplicates thereof. The description will be omitted.

1 is a block diagram showing the configuration of a behavior recognition apparatus according to an embodiment of the present invention, Figure 2 is a view showing the structure of a general behavior theory, Figure 3 is a behavior according to an embodiment of the present invention This is a conceptual diagram that defines the operations that make up.

Prior to describing the behavioral recognition apparatus according to the present invention, the behavioral theory will be briefly described.

The general theory of behavior is used to explain why an action is done by a person. Although research on behavioral theories is mainly done by psychologists and anthropologists, traditional behavioral theories are mostly ambiguous and known in declarative form without defining the scope and prerequisites for applying the asserting theory or model. . Therefore, it is virtually impossible to use traditional behavior theory as it is for behavior recognition.

In this embodiment, this behavior theory is transformed into a computer-computable model and data is expressed based on this. Behavior theory is a theory proposed in the 1920s in the former Soviet Union and is designed to include a variety of relationships that can induce behavior to explain various human behaviors.

Referring to Figure 2, the behavioral theory is largely composed of activity (activity), action (action) and operation (operation). Activity models the thawing of humans over long periods of time and consists of a set of actions. An activity can explain why a particular action is triggered. Actions are used to represent actions that occur relatively shortly than activities. Action is always purpose-oriented and performed for this purpose. An action is represented as a sequence of operations. Operation is the lowest act of the theory of action and is the act of unconsciousness in a present state given to a person for some purpose.

This traditional theory of behavior is so vague that it is difficult to apply to the real world. In particular, it is difficult to define an action, and even in the relationship between an action and an action, an action may be any action depending on the situation.

Accordingly, the present invention is to define a behavior that is a basic behavior of a person, and to provide a method of behavior recognition based on behavior theory. According to one embodiment of the present invention, the operation (operation) as shown in Figure 3 the basic operation of the human being represented by five ellipses (stay, hold, move, touch ( touch) and release) and express it as a graph model. A node represented by a rectangle in FIG. 3 means that one operation is finished. Of course, the node is a semantic node for expressing that one action may be determined by one or more operations.

That is, according to an embodiment of the present invention, by using the basic structure of the behavior theory as it is defined as five (operation), the existing behavior theory makes it possible to calculate. Of course, the operation is not necessarily limited to the five operations as described above, which is merely an embodiment. For example, when a textile sensor is used as the measurement device, an operation called a textile using sensed tactile information (roughness, softness, etc.) may be added.

In the present invention, a description will be mainly focused on recognizing the action according to the above operation, but the method of recognizing the activity according to the actions may be performed in the same or similar manner as the action recognition method according to the action. It will be more apparent to those skilled in the art through the following description.

Hereinafter, a device for recognizing a user's behavior according to the present invention will be described.

Referring to FIG. 1, the behavior recognition apparatus according to the present embodiment includes a measurement unit 10, a gesture recognition unit 30, a behavior determination unit 50, an input unit 70, and a display unit 90.

To explain the components of the behavior recognition device and its function schematically, the measurement unit 10 is for acquiring external physical information for determining a user's behavior, and the gesture recognition unit 30 is based on the acquired physical information. Recognizing a motion, the behavior determination unit 50 functions to recognize the user's behavior using the recognized one or more operations. The input unit 70 is for receiving arbitrary information from the user, and the display unit 90 is for informing the user of information about the recognized behavior and may be, for example, a display device such as an LCD.

The function of each component is described in detail below.

The measurement unit 10 is basic data for determining a user's operation, and measures external physical information such as position information by a GPS receiver and tag information of an RFID attached to an arbitrary object by a radio frequency identification (RFID) receiver. It is for. As described above, the measuring unit 10 may include a measuring device such as a GPS receiver, an RFID receiver, an accelerometer, and the like, and thus, the measuring unit 10 may be provided with the position information (x, y) and the RFID by the GPS receiver. External physical information such as tag information (ID of the corresponding object, detailed information (for example, position, etc.)) and a measured value by an accelerometer can be acquired. Of course, the external physical information acquired by the measurement unit 10 may further include other information such as tactile information by the textile sensor in addition to the position information, tag information, and acceleration information described above. Of course. Here, in the present embodiment, the behavior recognition device has been described as including the measurement unit 10, but according to another embodiment, the measurement unit 10 may be implemented as a separate device. That is, the behavior recognition device may be connected to the measurement unit 10 by wire or wirelessly, and receive external physical information measured by the measurement unit 10.

Measured external physical information is encoded based on behavioral theory. According to one embodiment, the rule of representation of measurement data follows Table 1 below. The operation OP may be determined according to the type of the measuring device. For example, "MOVE" and "STAY" of five basic information are recognized using measurement data by a GPS receiver or accelerometer, and "HOLD", "TOUCH" and "RELEASE" are electronic tag information such as RFID tag. Based on the perception.

The function of converting the RFID tag information and the GPS information into characters that can be recognized by the user may be performed by the motion recognition unit 30, and the information may be performed using a predefined conversion table or database (not shown). Can be performed. For the sake of understanding, for example, when the tag information of the RFID tag attached to an object includes 'ID = 0001', which means the knife, the object, 'RFID 0001 = knife' The Iran information may be stored as a conversion table, and it may be recognized by the motion recognition unit 30 that the object is a knife.

OP (RFID, Location) OP = {"MOVE", "HOLD", "TOUCH", "STAY", "RELEASE"} RFID = RFID data conversion table or database Location = GPS data conversion table or database

The measured physical information is represented by any one of five pieces of information through time series analysis by the motion recognition unit 30.

4 is a graph illustrating recognition of motion along a time axis according to an embodiment of the present invention.

를 미리 설정된 조건이 만족할 때 까지 확장한다. 예를 들어 "TOUCH"의 경우는 Δ

를 “2”로 설정한 경우 Δ

가 “3”인 경우에도 계측된 데이터는 표현 규칙에 따라 "TOUCH(ID, Location)"으로 변환된다. 여기서 TOUCH의 속성 값들로는 터치된 해당 물체의 태그 정보인 ID와 위치정보를 가질 수 있다.The measured information may be represented along the time axis as shown in FIG. 4. According to one embodiment of the invention, the motion recognition unit 30 recognizes the motion by analyzing the measurement data through the slide windows method of the time series analysis method. Slide windows method extends the windows from the starting point until the condition is satisfied. As shown in FIG.

Extend until the preset condition is satisfied. For example, in the case of "TOUCH", Δ

Is set to “2” Δ

Even if "3", the measured data is converted to "TOUCH (ID, Location)" according to the expression rule. Here, the attribute values of the TOUCH may have ID and location information which is tag information of the touched object.

5 is an exemplary view showing a recognized operation using the measurement data according to an embodiment of the present invention. FIG. 5 illustrates an embodiment of motion data modified by the motion recognition unit 30 through time series analysis based on data transmitted from the actual measurement device of the measurement unit 10.

As described above, the motion recognition unit 30 determines a specific motion based on the data measured by the measurement unit 10 (hereinafter, measured data). For example, when the location information by the GPS receiver is changed from (x, y) to (a, b), it may be recognized that the user has performed a 'move' operation.

Action data representing each action has attribute information corresponding to the action. As shown in FIG. 5, in the case of MOVE, location information on a location before moving and a location after moving may be included as attribute information, such as 'move (location1, location2)'. The above attribute information will be hereinafter referred to as "operation attribute".

Similarly, according to an embodiment, the TOUCH may have information on the contacted object ID and its location as attribute information, where RELEASE is an object and a location, STAY is a location and a holding time, and finally HOLD is an object. , Location and maintenance time may be included as attribute information.

The behavior determination unit 50 functions to determine what the user's behavior is by using one or more motion information by the motion recognition unit 30.

The behavior determining unit 50 includes a behavior learning unit 52, a learning information storage unit 54, a behavior recognition unit 56, and a feedback processing unit 58 as shown in FIG. 1.

", 특정 행동이 수행될 때 해당 동작이 발생할 확률"

", 행동 시간, 행동에 따른 동작 개수 및 행동을 이루는 동작 세트(즉, 행동-동작 세트)가 있을 수 있다(도 7 또는 도 9 참조). 학습 데이터에 대해서는 도 7 및 도 9를 참조하여 상세히 후술하기로 한다.The behavior learning unit 52 is for learning how each action is made by what actions at what time, and the learning data learned by the behavior learning unit 52 is stored in the learning information storage unit 54. The training data includes a set of one or more actions according to each action (hereinafter, action-action set), a total holding time (hereafter, action time), and performance time information (where time is hours, minutes, and seconds). Absolute time information, hereinafter referred to as execution time), and the number of operations to be performed. And the probability that the action is performed at any time can be used together as learning data. In summary, the training data for each behavior suggests that the probability that a particular behavior will occur during that time is "

", The probability that the action will occur when that action is performed"

", The action time, the number of actions according to the action, and the action set (i.e., action-action set) that constitutes the action (see FIG. 7 or FIG. 9). The learning data is described in detail with reference to FIGS. It will be described later.

The behavior recognizer 56 may recognize a user's behavior (hereinafter, referred to as “observation behavior”) according to one or more recognition operations by the gesture recognizer 30 using the training data stored in the learning information storage 54. do. That is, the behavior recognition unit 56 determines what behaviors one or more operations recognized by the motion recognition unit 30 mean by using previously stored learning data. A function and a method of performing the behavior recognition unit 56 will be described later. Hereinafter, the method will be described in which behaviors are learned to set learning data.

First, the training data according to each initial behavior may be set by one or more training example data (see FIG. 7). In order to more accurately set the training data, it is preferable that a plurality of training example data be used. When there are a plurality of learning example data, the behavior learning unit 52 generates learning data obtained by statisticizing each learning example data. The statistical method will be described later.

According to an embodiment, the learning example data may be input by the user through the input unit 70. For example, “preparing for dinner” is learning data about behavior: {MOVE (A, kitchen), HOLD (knife, 1 minute), TOUCH (fridge, kitchen), STAY (kitchen, 9 minutes) The action set and the time at which the action begins (for example, 7 pm) can be entered, where the total action time of the action can be perceived as the difference between the first action start time and the last action completion time. When a plurality of example data is input for one action, a plurality of learning example data are statistically set to set learning data, which can be stored in the learning information storage unit 54 or separately, although not shown. It may be stored in the learning example storage provided and used by the behavior learning unit 52 to generate and update the learning data.

6 is an exemplary diagram illustrating a user interface for setting learning example data according to an embodiment of the present invention. Referring to FIG. 6, a user may determine a name of an action and select one or more of a plurality of actions provided as a sample to generate learning example data according to one action. In other words, learning example data according to an operation is defined and generated by the user. As shown in the figure, it can be seen that four actions are set by the user for the action of Take a rest.

The reason for setting one learning data by inputting a plurality of learning example data is that the pattern of taking the action may be different for a plurality of users, and even the same user may always be the same pattern of taking the action. Because it is not. Therefore, it is preferable that the behavior learning unit 52 sets learning data using a plurality of learning example data.

Of course, the existing learning data may be changed by using data (hereinafter referred to as “cognitive behavioral data”) that has information about actual behaviors observed and recognized according to the behavior of the user. In other words, the cognitive behavioral data according to the recognized behavior may be fed back and used as learning example data for learning.

Cognitive behavioral data according to the behavior recognized by the feedback processor 58 illustrated in FIG. 1 may be reflected by the behavioral learning unit 52 in the learning data. Of course, the cognitive behavioral data according to the actual perceived behavior may have a higher weight than the learning example data initially input by the user. The detailed function of the feedback processing unit 58 will be described later. Therefore, as the learning progresses more, the user's behavior can be recognized more accurately.

FIG. 7 is a table illustrating a concept of generation of learning data using learning example data according to an embodiment of the present invention.

Referring to FIG. 7, two “dinners” and one “shower” are input to the behavior learning unit 52 as learning example data. First, by observing the action “shower,” actions A, E, F, G, and E were performed for time t minutes. Here, there may be two main methods for setting the action-action set. In the first method, both the order and the number of actions may be reflected in the action-action set. In the second method, the action-action set may be set such that the number and the order are not reflected in the set concept. The behavior-action set of “shower” is {A, E, F, G, E} according to the first method and {A, E, F, G} according to the second method. The reason for using the second method without considering the order and the number of times is that if the order information and the number of occurrences are specified, the system may not be able to perform the correct recognition. For example, since the user does not touch the soap in the same order and the number of times every shower, the learning data can be set in a set concept. Instead, as shown in the figure, the training data includes the number of actions that constitute each action. Using the behavior-action set in the second manner described above facilitates the calculation of the penalty function to be described later.

Of course, it will be apparent to those skilled in the art that the action-action set may be set in consideration of the order and number of actions for more accurate action recognition. However, hereinafter, it will be described on the assumption that the action-action set does not consider the order and the number of times for convenience of description.

Looking at the behavior “dinner”, it can be seen that two learning example data is input to the behavior learning unit 52. In this case, as shown in the figure, the average ((t + t ') / 2) of the action time according to the two learning example data may be set as the action time of the learning data. Likewise, the number of operations can be arithmetic averaged.

"특정 행동에 대해 해당 동작이 발생할 확률 "

", 행동 시간, 행동에 따른 동작 개수 및 행동을 이루는 동작 세트(즉, 행동-동작 세트)에 대해 그 통계값을 산출하여 학습 데이터로써 저장한다.

에 대해서는 도면에 도시되지 않았으나, 행동-동작 세트의 각 동작마다

가 산출되어 함께 학습 데이터로써 저장될 수 있다(관련 도면 도 9참조). 예를 들어, 행동 “샤워”의 동작 E의

는 2/5(0.4)일 수 있다.Referring to the table of the training data of FIG. 7, the behavior learning unit 52 may determine the probability that a specific behavior occurs in a corresponding time zone (where time information is visual information) as described above.

"The probability that the action will occur for a particular action"

", The statistical value is calculated for the action time, the number of actions according to the action, and the action set (that is, the action-action set) which constitutes the action, and stored as learning data.

Although not shown in the figure, for each action in the action-action set

Can be calculated and stored together as learning data (see FIG. 9). For example, the action E of the action “shower”

May be 2/5 (0.4).

의 학습을 위해서 하기의 수학식 1과 같은 포아송 분포를 이용할 수 있다. 포아송 분포를 이용하는 이유는 행동과 시간과의 관계를 유지하기 위해서이다. 즉, 특정 시간대에 이루어지는 특정한 행동을 모델에 반영하고 이를 이용하여 인지의 성능을 높이기 위해서 사용한다. In this embodiment

For learning the Poisson distribution as shown in Equation 1 can be used. The reason for using the Poisson distribution is to maintain a relationship between behavior and time. In other words, the model reflects specific behaviors in specific time zones and uses them to enhance cognitive performance.

행동이 발생한 시간을 의미하며

은 전체 샘플의 개수(즉, 학습예제 데이터의 개수)를 의미한다. here

Means when the action occurred

Means the total number of samples (ie, the number of learning example data).

의 학습을 위해서 수학식 2와 같은 maximum likelihood estimation을 이용한다.And

For learning, we use the maximum likelihood estimation as in Equation 2.

는 가상값이며 일반적으로 샘플의 크기만큼 설정되며

는 작은 값으로 설정된다. In equation (2)

Is an imaginary value, typically set to the size of the sample

Is set to a small value.

를 이루고 있는 동작 중 첫 번째 동작

과 마지막 동작

의 차이를 통해 개별 행동의 유지시간을 산출할 수 있으며 학습 예제에서 같은 개별행동의 산술평균을 통해서 평균 행동 시간을 얻을 수 있다. Action time is a specific action

The first one that makes up

And last action

The difference in time can be calculated from the retention time of individual behaviors, and in the learning example, the average behavior time can be obtained from the arithmetic mean of the same individual behaviors.

를 이루고 있는 평균 동작(operation)의 수는 하나의 행동이 몇 개의 동작으로 이루어져 있는지 셀 수 있고, 평균 행동 시간과 같이 학습예제 전체를 통해서 산술평균값으로 얻을 수 있다(도 7 참조).Also, act

The average number of operations making up the number of operations can be counted as a number of operations, and can be obtained as an arithmetic mean value through the whole learning example as the average time of action (see FIG. 7).

를 이루는 동작의 세트 즉 행동-동작 세트는 학습예제 전체를 통해서 산출할 수 있는데, 그 산출 방법은 다양하게 이용될 수 있다. 하나의 예를 들자면, 학습예제 전체 중 80% 이상 포함되는 동작들의 집합이 행동-동작 세트로 설정될 수 있다. 물론, 이에 한정되는 것은 아니며 다른 예를 하나 더 들면, 각 동작의

이 일정 수치 이상인 동작들로만 구성되도록 행동-동작 세트가 설정될 수도 있다.Also, act as

The set of actions constituting the action, that is, the action-action set may be calculated through the entire learning example, and the calculation method may be used in various ways. As an example, a set of actions included in more than 80% of all learning examples may be set as an action-action set. Of course, the present invention is not limited thereto. For another example,

An action-action set may be set to consist only of actions that are above this certain number.

In summary, the behavior learning unit 52 statistics about each action using a plurality of learning example data, and stores the data according to the learning information storage unit 54 as learning data.

Hereinafter, the format in which the actual learning data is stored will be described.

가 산출될 수 있다.8 is a diagram illustrating a format of learning data according to an embodiment of the present invention. Referring to FIG. 8, the behavior index 810 has all the actions that occurred in one action and time information (action time) of the action, and time information (time information on which the action was performed). Reference numeral 810 defines the action "Prepare dinner" and "#" indicates the start of the action. "TIMEZONE" means the average time zone (visual information) of the time when this action occurred mainly. "AD" is the average action time of the action and "AO" means the number of actions that occurred on average. "PRIOR" is a learned parameter that indicates how frequently an action occurs. "PRIOR P 18.77" indicates that the learned parameter is a parameter according to the Poisson distribution whose value is "18.77". In other words, by using the value of PRIOR,

Can be calculated.

을 가진다. 예를 들어 "plates cupboard"를 주방에서 들고 있는 동작 "HOLD(Plates Cupboard, K)“는 "Prepare Breakfast"와 "Leave House"라는 행동에서 발생했으며 각각의 발생 확률은 대략 0.56, 0.35임을 나타내고 있다. 본 실시예에서 이와 같은 내용을 저장하는 이유는 본 실시예에 따른 행동인지 장치가 모바일 디바이스로 구현되는 경우, 보다 빠르게 데이터 처리를 수행하기 위해서이다. 실제 행동인지 장치의 학습정보 저장부(54)에는 학습 데이터가 도면에 도시된 행동 인덱스(810) 및 동작 인덱스(830)의 "#"뒤에 나타나는 행동 이름과 동작 이름을 키로 사용하는 해쉬 테이블 등으로 구현될 수 있다.The action index 830 is an index file for actions, where "#" indicates the start of an action and a probability value that will occur in an action including each action.

Has For example, the action "HOLD (Plates Cupboard, K)" holding the "plates cupboard" in the kitchen occurred in the actions of "Prepare Breakfast" and "Leave House", indicating that the probability of each occurrence was approximately 0.56 and 0.35. In the present embodiment, the reason for storing such contents is to perform data processing more quickly when the behavior recognition apparatus according to the present embodiment is implemented as a mobile device. The learning data may be implemented as a hash table using the action name and the action name appearing after the "#" of the action index 810 and the action index 830 shown in the figure as keys.

Hereinafter, a method of recognizing a user's observed behavior corresponding to the motion data recognized by the motion recognition unit 30 using the learning data stored in the learning information storage unit 54 will be described. Shall be.

According to one embodiment of the invention, the behavior recognition unit 56 uses a 0-other Markov model. In general, high-dimensional Markov models are known to perform well, but the use of high-dimensional Markov models increases exponentially, making it impossible to calculate on equipment such as mobile devices. However, when using the 0-order Markov model, learning may be possible as mentioned above, but performance may be degraded when the user's behavior is recognized based on the learning result.

In this embodiment, to solve this problem, a penalty function is introduced and used to recognize the behavior.

The behavior recognizer 56 recognizes the observed behavior according to the recognized motions using Equation 3 below. Given the user's situation (o, t), the most likely behavior is to be perceived by the user up to time t-1.

"및 특정 행동 수행 시 해당 동작이 발생할 확률"

"는 학습 데이터로서 미리 저장되어 있으므로,

만을 산출하면 된다.Whether it's a behavior here, the probability that

"And the probability that the action will occur when you perform that action"

"Is pre-stored as training data,

Calculate only.

는 패널티 함수이다. 수학식 3을 해석하자면, 특정 행동의 발생 확률

와 해당 행동에서의 특정 동작의 발생 확률을 이용한 수치

를 패널티 함수

로 나누어 각 행동에 대한 패널티가 부여된 수치를 이용하여 인지된 동작들에 따른 행동을 판단하는 것이다.In Equation 3 above

Is a penalty function. To interpret Equation 3, the probability of occurrence of a specific behavior

And numerical value using probability of occurrence of a specific action in the action

Penalty function

By dividing by, it is used to determine the behavior according to the perceived movements using the numerical value assigned to each action.

In this embodiment, two interpretation methods can be used for the penalty function. The first interpretation method fixes the average progress time (ie behavior time) of the behaviors calculated in the learning phase, calculates the degree of inconsistency between the set of cognitive behaviors that occurred within the behavior time and the behavior-action set learned during the learning, and then It is a penalty. The second method is to extend the average action time to minimize the degree of inconsistency. At this time, as an example, the maximum value of the time that can be extended may use the deviation of the average time (that is, the deviation of the behavior time statistically calculated using the learning example data).

To illustrate how to apply the penalty function described above, reference is made to FIG. 7 together. Referring to the training data shown in Figure 7,

(example)

1. Training data (see Figure 7)

1) Dinner lesson: t minutes of action, action-action set {A, B, C, D, H}

2) Shower Learning: Action Time t Minutes, Action-Action Set {A, E, F, G}

2. Actually observed motion (ie one or more motions recognized by motion recognition unit 30)

Actions recognized for t minutes (A, B, C, D, B, E), actions recognized for t 'minutes after t minutes-(H)]

If the behavior penalty is fixed with the behavior time fixed, the behavior penalty is calculated by subtracting the cognitive behavior set of the behavior data according to the perceived behavior from the behavior-action set of the training data. Applying the example above, the dinner behavior penalty may be one and the shower behavior penalty may be two. Subtracting the observed motion set {A, B, C, D, E} from the dinner set {A, B, C, D} for dinner, 1 ({H}) is the behavioral penalty for dinner, and similarly showers Subtracting the observed motion set {A, B, C, D, E} from the motion set {A, E, F, G}, 2 ({F, G}) is the behavior penalty of the shower. In the following, the penalty of an action set will be referred to as "operation mismatch". In other words, the dinner discrepancy is 1, and the shower discrepancy is 2.

To calculate the behavioral penalty according to the method of extending the behavioral time according to another embodiment, the motion observed during the additional time t 'minutes is H, and accordingly, the motion inconsistency of the dinner becomes 0, and the motion inconsistency of the shower remains 2 It can be seen that ({F, G}).

Even if only a set is used without considering the order and the number of occurrences separately, it is possible to recognize an action because the actions taken in each action are different from each other. For example, you can remove the fork at dinner but rarely hold the fork in the shower. Therefore, each action does not have many overlapping actions, so even if the number of discrepancies is calculated in the set concept, there is no great difficulty in recognizing one action.

9 is a conceptual diagram illustrating a method of calculating a behavior penalty according to an embodiment of the present invention.

의 동작(operation)들을 보여주고 있으며, 참조번호 910은 첫 번째 해석방법(행동 시간 고정)을, 그리고 참조번호 930은 두 번째 해석방법(행동 시간 확장)을 설명하고 있다. 행동 시간 고정 방법(910)에서는 상술한 것과 같이 현재 진행된 동작들이

인지를 판별하기 위해서

의 평균 행동 시간을 고정(

)하고

의 동작과 도면에 나타난 동작 집합과의 불일치한 동작의 수를 통해서 패널티를 받게 된다. 행동 시간 확장 방법(930)은 평균 행동 시간을 확장(

)한 후 도면과 같은 방식으로 불일치 정도를 따져 패널티를 받게 된다.9, reference numeral 950 indicates an already learned behavior.

The operations of are shown, reference numeral 910 describes the first method of interpretation (action time fixation), and reference numeral 930 describes the second method of interpretation (action time extension). In the action time fixing method 910, as currently described,

To determine whether

Fixed average action time of

)and

Penalty is given through the number of inconsistencies between the motion set and the motion set shown in the figure. The behavior time extension method 930 extends the average behavior time (

After that, you will be penalized according to the degree of inconsistency.

Table 2 charts the imposition of penalties according to circumstances.

division No mismatch If there is a mismatch in behavior Action time fixed No penalty # of mismatch Extended action time Time penalty Time penalty + discrepancies

Therefore, although two penalty functions are required, the present invention uses an exponential distribution such as Equation 4 as a penalty function for inconsistency between time and behavior.

는 동작의 불일치된 개수이며 파라미터

는 1.2이하의 작은 값을 사용한다. 시간에 대한 패널티

는 동작 불일치 지수분포를 따르는

와 같이 계산할 수 있다.Of the exponential distribution in equation (4).

Is the mismatched number of actions

Uses a value less than 1.2. Penalty for time

Follows the behavior mismatch exponential distribution

It can be calculated as

The final penalty is weighted to each penalty as shown in Equation 5 so that the specific penalty can be changed as necessary.

In this embodiment, in the case of extending the behavior time, the time penalty and the number of motion inconsistency are used as an action penalty. However, the present invention is not necessarily limited thereto. It can also be used as an action penalty. Since any operation is not likely to have the same action time each time, a predetermined amount of time can be extended without penalty.

Hereinafter, the function of the feedback processing unit 58 of the behavior determination unit 50 will be described. The feedback processor 58 notifies the user of the result recognized by the behavior recognizer 56 through the display unit 90 and receives feedback on whether the result is correct through the input unit 70. Accordingly, the function of controlling the behavior learning unit 52 to be modified is performed.

FIG. 10 illustrates a user interface for updating learning data using behaviors recognized by user feedback according to an embodiment of the present invention.

Referring to FIG. 10, the feedback processor 58 displays a behavior result screen including information according to the recognition behavior data recognized by the behavior recognition unit 56, such as reference numeral 1010, to the user through the display unit 90 and is recognized. Whether the information is correct is received from the user through the input unit 70. In this case, if the user selects that the recognized information is correct, no additional work is required, but if the user selects that the recognized information is not correct, the feedback processor 58 outputs a correction screen as shown by reference numeral 1030. In the modified screen, the behavior recognition unit 56 shows the actions that are determined as the current behavior and receives the wrong part from the user. The feedback processor 58 transmits the modified data (action or behavior name, etc.) to the behavior learning unit 52 to instruct the model correction for the wrongly recognized behavior and the user input. The behavior learning unit 52 updates the learning data using the data received from the feedback processing unit 58 (hereinafter, the feedback data) as the learning example data. Of course, the feedback data received from the feedback processor 58 may be used to update the training data by adding weight to the training example data.

As an example of updating the training data by the feedback data, if the perceived behavior was "preparation for dinner," but the user's feedback actually performed the action "washing dishes," the feedback processing unit 58 is a cognitive behavior. The name of the data and the actual behavior “washing dishes” is provided to the behavior learning unit 52. Accordingly, the behavior learning unit 52 reflects the feedback data to the learning data for the behavior "washing dishes" and modifies it.

In summary, when the behavior is not recognized, the feedback screen shows the observed motions, and when the user removes or adds the wrong motion, the user's learning model is modified based on the feedback information (ie, the training data is modified). .

는 처음 2/5에서 1/4로 수정된다.As another example, it is assumed that there are A and B motions that are observed as dinner behaviors, and 2 and 3 motions are performed. Here, when the first operation is removed by the feedback, the operations A and B of dinner are corrected to appear as 1 and 3 times, respectively. Therefore, the probability that action A will occur in the action

Is modified from the first 2/5 to 1/4.

Hereinafter, a description will be given of the process whether the behavior of the device according to the present invention.

11 is a flowchart illustrating a behavior recognition process according to an embodiment of the present invention.

In describing the behavior cognition process with reference to the drawings, the method of setting learning data using learning example data, user's behavior recognition using learning data, and updating the learning data using feedback data from the use is a function of the behavior recognition device. Since it has been described and described above, duplicate description thereof will be omitted.

Referring to FIG. 11, the behavior recognition apparatus recognizes a user's motion based on external physical information according to the user's motion (S100). As described above, the behavior recognition device includes a measurement device for acquiring external physical information such as RFID tag information and GPS location information, and directly acquires the external physical information as described above, or receives it from a connected external measurement device. An operation corresponding to measurement data according to external physical information may be recognized. Therefore, the behavior recognition device recognizes a user's motion according to any one of MOVE, HOLD, RELEASE, TOUCH, and STAY based on the recognized external physical information.

Thereafter, the behavior recognition apparatus determines whether the behaviors recognized to date are previously learned using stored learning data (S102). If the number of recognized motions is small or no behavior is recognized, the process proceeds to s100 again to recognize the next motion.

If there is learning data on the behavior corresponding to the previously recognized behaviors, the behavior according to the corresponding learning data is recognized as the user's behavior (s104). That is, the behavior recognition device recognizes a behavior corresponding to the user's motions observed to date.

Information about the perceived behavior is displayed to the user (s106).

The behavior determining apparatus determines whether feedback data is input from the user (s108), and when the feedback data is input, updates the learning data according to the feedback data (s110).

The method of the present invention as described above may be implemented in a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.).

In addition, preferred embodiments of the present invention described above are disclosed for the purpose of illustration, those skilled in the art will be able to various modifications, changes, additions within the spirit and scope of the present invention, such modifications, Modifications and additions should be considered to be within the scope of the following claims.

1 is a block diagram showing the configuration of a behavior recognition apparatus according to an embodiment of the present invention.

2 illustrates the structure of a general theory of behavior.

3 is a conceptual diagram defining actions constituting an action according to an embodiment of the present invention.

Figure 4 is a graph for explaining the recognition of the motion along the time axis according to an embodiment of the present invention.

5 is an exemplary view showing a recognized operation using the measurement data according to an embodiment of the present invention.

6 is an exemplary diagram illustrating a user interface for setting learning example data according to an embodiment of the present invention.

7 is a table illustrating a concept of generation of learning data using learning example data according to an embodiment of the present invention.

8 illustrates a format of learning data according to an embodiment of the present invention.

9 is a conceptual diagram illustrating a method of calculating a behavior penalty according to an embodiment of the present invention.

FIG. 10 illustrates a user interface for updating learning data using behaviors perceived by user feedback in accordance with one embodiment of the present invention. FIG.

11 is a flowchart illustrating a behavior recognition process according to an embodiment of the present invention.

Claims (19)

In the device for recognizing the behavior of the user, A motion recognition unit for recognizing any one of predefined operations as a user's motion by using measured external physical data; And It includes a behavior determination unit for determining the observed behavior of the user corresponding to the one or more cognitive actions recognized by the motion recognition unit using the learning data that is information on the action (action) consisting of one or more actions, The behavior determination unit may include: a behavior learning unit for generating the learning data using at least one learning example data; A learning information storage unit for storing the learning data; And a learning cognition unit for recognizing the observed behavior of the user corresponding to the cognitive motion by using the learning data. The method of claim 1, Wherein the action is one of stay theory, action hold, move, touch, and release. delete The method of claim 1, The behavior learning unit generates probability according to each action using the learning example data.

, The probability that any action will occur when the action is performed.

Behavioral theory based behavioral cognition device, characterized in that for generating training data, including the action time, the number of motions and the set of behaviors (action-action set). The method of claim 4, wherein remind

Is a behavior theory based behavior recognition device, characterized in that calculated using a Poisson distribution using a plurality of the learning example data. The method of claim 1 The behavior recognition unit recognizes the observed behavior by applying a penalty calculated by using the inconsistency between the behavior-action set corresponding to each behavior of the learning data and the behavior set of the cognitive behavior to each behavior. Behavioral Cognitive Device Based. The method of claim 6, The discrepancy number is calculated by subtracting the action set of the cognitive action from the action-action set. The method according to claim 4 or 6, The behavior recognition unit recognizes the observed behavior using the following equation, but the penalty function (

Is calculated using the number of inconsistencies between the action-action set of the learning data and the action set of the cognitive action.

The method of claim 6, Wherein the number of discrepancies uses an action set of perceived actions during the same time interval as the action time of the learning data, according to a fixed action time method. The method of claim 6, The discrepancy number may be calculated using an action set of actions recognized for a time interval extended within a predetermined range from the action time of the learning data, according to an extended action time method. . The method of claim 10, The penalty (

) Uses a behavioral theory based behavioral cognition apparatus according to the following equation, using the time penalty for the extended time together with the inequality penalty according to the number of inconsistency.

(here,

Is the time penalty over expansion time,

Is a penalty for discrepancies based on the number of discrepancies) The method of claim 1, The behavior determining unit further includes a feedback processing unit controlling the behavior learning unit to update the learning data to correspond to the feedback data of the user input according to the observed behavior. The method of claim 1, And the motion recognition unit recognizes the measured external physical data as any one of the defined motions through time series analysis over time. The method of claim 1, And a measurement unit for measuring the external physical data of at least one of RFID tag information, GPS location information, and acceleration information. In the method of recognizing the user's behavior, Recognizing any one of a predefined operation as a user's operation using the measured external physical data; And Determining the observed behavior of the user corresponding to the one or more cognitive actions recognized by the motion recognition unit using the learning data that is information on the action consisting of one or more actions, The determining of the observed behavior of the user may include: calculating a penalty by comparing a number of inconsistencies between an action-action set according to each pre-stored action and an action set recognized by the gesture recognition unit; And applying the penalty to recognize the observed behavior corresponding to the cognitive behavior. The method of claim 15, Probability of occurrence for each action using a plurality of learning example data including action name, action information, and action execution time

, The probability that any action will occur when the action is performed.

And generating the learning data comprising a time of action, a number and a set of actions (action-action sets) that constitute an action. delete The method of claim 15, The method may further include displaying information according to the observation behavior. When feedback data is input from a user, behavior theory based behavior recognition method, characterized in that for updating the learning data using the feedback data. 19. A recording medium on which a combination of instructions for performing the method of any one of claims 15 to 16 and 18 is tangibly embodied and in which a program readable by a digital information processing apparatus is recorded.

Images