JP5359414B2 - Action recognition method, apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a behavior recognition method, device, and program for recognizing behavior at a plurality of abstract levels. <P>SOLUTION: A behavior identifier sequence at a first abstract level is recognized and stored by holding model data corresponding to a behavior model and an behavior recognition algorithm for each of plurality of abstract levels, extracting a feature vector from the sensor data, and performing recognition processing that uses the model data corresponding to the first abstract level among the plurality of abstract levels to the feature vector. The behavior identifier sequence at the abstract level is recognized and stored by performing recognition processing that uses the model data corresponding to the abstract level to the sequence of behavior identifiers at a lower level than the abstract level stored in each of the plurality of abstract levels. A second recognition step is repeated with respect to at least one abstract level including a second abstract level. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a behavior recognition method, apparatus, and program for recognizing behavior of an individual such as a person or an animal.

  Conventionally, a behavior recognition method for recognizing the behavior of an individual such as a person or an animal, such as a system for watching the behavior of an elderly person, a system for monitoring the behavior of a driver, or a system for analyzing customer behavior in a store, or the like The system is known. In such a method or system, usually, an individual's action is recognized from image information obtained by a video camera or the like, or an individual's action is recognized from sensor information by RFID (Radio Frequency IDentification) attached to the individual. .

  For example, Patent Document 1 compares “data” obtained from an acceleration sensor, a pulse sensor, and a skin temperature sensor with a statistic obtained from the data, thereby “sleep”, “rest”, “stressed”, “meal”, “ A technique for classifying “movement” is disclosed. Patent Document 2 discloses a technique for recognizing a human action using a video camera image. Here, learning is performed based on a stochastic state transition model such as a Hidden Markov Model (HMM) based on features such as mesh features and optical flow direction distribution in the image, and actions are recognized. It is. Patent Document 3 discloses a technology for tracking movement of a person by cooperating with a plurality of cameras and recognizing actions such as “sitting”, “hurrying”, and “being troubled” from the transition pattern. doing. Furthermore, in Patent Document 4, whether or not a person wearing an RFID tag reader on the wrist has done a hand near the RFID tag, each of the merchandise such as a shelf and a bag in the retail store is attached. Disclosed is a technique for recognizing purchase behavior that represents a willingness to purchase a product by determining the relationship such as the order of approaching a bag, a product, or a shelf tag.

JP 2000-245713 A JP 05-46583 A JP 2006-221379 A JP 2006-209421 A

  However, the conventional action recognition system as described above mainly focuses on recognizing a single action from data that can be sensed from a person's action. A human action cannot be expressed as a single action, but can be expressed in various levels of abstraction. For example, in the case of an action “driving a car”, “driving a car toward the workplace in the process of going to work” or “driving a car toward the store in the process of going to shopping” It can also be understood as an action with a different level of abstraction. Similarly, in the case of the action of “opening the door”, the action can be expressed at various levels depending on “what” the door of “where” is opened. For example, the action of “opening the door” when going to work in the morning is specifically the action of “opening the entrance door”, and when the level of abstraction is further increased, “going out”, specifically, “going to work” It is possible to express at various levels of abstraction such as Thus, even a single action can be expressed as a plurality of abstract level actions. However, the method of the conventional literature can recognize only an action at a single abstract level such as “open the door”.

  The present invention is to provide an action recognition method, apparatus, and program capable of grasping actions at a plurality of abstract levels.

The behavior recognition method according to the present invention is a behavior recognition method in a behavior recognition device that takes in sensor data that measures an individual's behavior and recognizes the behavior of the individual from the sensor data,
A behavior model holding step for holding model data corresponding to a behavior model and a behavior recognition algorithm for each of a plurality of abstract levels; a feature vector is extracted from the sensor data; A first recognition step of recognizing the action identifier of the first abstract level by performing recognition processing using model data corresponding to the first abstract level, and storing the action identifier in association with the acquisition time of the sensor data. And, for each of the plurality of abstract levels, for a sequence that is a temporal transition of action identifiers lower than the abstract level and is a set of one or more action identifiers, by performing recognition processing using the corresponding model data, it recognizes the action identifier of the level of abstraction, the cell Includes a second recognition step of storing in association with acquisition time of Sadeta, and for at least one level of abstraction, including a second level of abstraction, characterized by repeating said second recognition step.

The behavior recognition device according to the present invention is a behavior recognition device that captures sensor data for measuring an individual's behavior and recognizes the behavior of the individual from the sensor data, and includes a behavior model and a behavior recognition algorithm for each of a plurality of abstract levels. Behavior model holding means for holding model data corresponding to the above, and a feature vector is extracted from the sensor data, and model data corresponding to a first abstract level of the plurality of abstract levels is extracted from the feature vector. The first recognition means for recognizing the action identifier of the first abstract level by performing the recognition processing used and storing the action identifier in association with the acquisition time of the sensor data , and stored for each of the plurality of abstract levels Model data corresponding to the abstract level for a sequence of action identifiers lower than the abstract level. By a recognition process performed using recognizes the action identifier of the level of abstraction, wherein the second recognition means for storing in association with acquisition time of the sensor data, the second recognizing means, the second abstraction for at least one level of abstraction including level, characterized by repeating an operation of recognizing and storing the action identifier of the level of abstraction.

The behavior recognition program according to the present invention is a behavior recognition program for fetching sensor data for measuring the behavior of an individual and executing a procedure for recognizing the behavior of the individual from the sensor data on a plurality of abstract levels. A behavior model holding step for holding model data corresponding to a behavior model and a behavior recognition algorithm every time, a feature vector is extracted from the sensor data, and a first of the plurality of abstract levels with respect to the feature vector A first recognition step of recognizing the action identifier of the first abstract level by performing recognition processing using model data corresponding to the abstract level of the sensor data, and storing the action identifier in association with the acquisition time of the sensor data; For each abstract level, it consists of action identifiers stored at a level lower than the abstract level. Relative Sequence, by performing a recognition process using the model data corresponding to the level of abstraction, a second recognition step of recognizing the action identifier of the abstraction level and stored in association with the acquisition time of the sensor data, comprises, for at least one level of abstraction, including a second level of abstraction, characterized in that it comprises a procedure of repeating the second recognition step.

  According to the individual behavior recognition method, apparatus, and program of the present invention, a configuration is provided in which a plurality of detailed behaviors that are temporally related to each other are accumulated to recognize a higher abstract level of behavior. Thereby, it is possible to grasp the behavior at a plurality of abstract levels.

It is a block diagram which shows the concept of the action recognition apparatus by this invention. It is a wave form diagram which shows the example of the sensor data from a 3-axis acceleration sensor. It is an Example of this invention and is a block diagram which shows the internal structure of the action recognition apparatus by this invention. It is a figure which shows the example of the correspondence of the action identifier and action in a 1st abstract level. It is a figure which shows the example of the action identifier sequence used as the correspondence of action identifier and action in a 2nd abstract level, and a reference | standard. Correspondence between action identifiers and actions at the third level of abstraction and the reference row It is a flowchart which shows the process sequence of the action recognition method by this invention. It is a figure which shows the example of the correspondence of sensor data and the recognized action identifier sequence.

  Embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is an embodiment of the present invention and shows a conceptual form of an action recognition apparatus according to the present invention. A person as an individual who is an object of action recognition is equipped with a plurality of three-axis acceleration sensors 11 to 13 on both wrists and waist. The action recognition device 20 takes in sensor data from each of the three-axis acceleration sensors 11 to 13. An example of sensor data from the three-axis acceleration sensors 11 to 13 is shown in FIG. The action recognition device 20 recognizes the action identifier of the first abstract level from the captured sensor data, and then recognizes the action identifier of the higher abstract level from the sequence consisting of one or more action identifiers of the lower abstract level. By repeating this procedure, a plurality of abstract level action identifier sequences are output as a sequence.

  The first action refers to an action having a first abstract level, and has a first action identifier for each first action. The second action refers to an action having the second abstract level, and similarly has a second action identifier for each second action. When generalized, an nth action (n is a positive integer) is an action having an nth abstract level that is one level higher than the (n−1) th action, and for each nth action. It has an nth action identifier.

  In FIG. 1 showing an example of radio gymnastics, the first action is the action at the lowest level of abstraction. For example, the individual action such as “bend the hand from the bottom” or “bend the waist” is fine. It is an action. The second action has a level of abstraction that is one step higher than the first action, and in the example of radio exercises, the first 13 types of exercises of radio exercises, such as “curve the chest”, “bend the body sideways”, etc. It is an action. In addition, the third action has a level of abstraction one level higher than that of the second action, and indicates whether it is “Radio Gymnastic First” or “Radio Gymnastic Second” rather than individual exercises of radio gymnastics. It is a level action. In this way, a higher abstract level action is recognized as a sequence of action identifiers consisting of a series of lower level action actions, that is, a chain of action identifiers.

  Each of the nth actions is output in association with the time. For example, the first action recognized from the sensor data between 10: 10: 10.00 seconds and 10: 10: 11.00 seconds is recorded as the first action at 10: 10: 10.00 seconds. Or output. The third action recognized from the sensor data between 10: 10: 10.02 seconds and 10: 10: 11.02 seconds is recorded or output as the third action at 10: 10: 10.02 seconds. Is done. In the example of this figure, the third action “radio exercises first” associated with 10: 10: 10.02 is output.

  FIG. 3 shows the internal configuration of the action recognition device 20. The behavior recognition device 20 includes an input unit 21, a feature extraction unit 22, a first recognition unit 23, a second recognition unit 24, a behavior storage unit 25, a behavior model storage unit 26, and an output unit 27. Has been. The behavior recognition device 20 can be usually realized as one computer device. In that case, each of the above-described units 21 to 27 can be realized by hardware provided inside or outside the computer and / or software such as a program and data executed by the computer.

  The input unit 21 receives sensor data from a sensor such as an acceleration sensor, samples the sensor data at a predetermined sampling period, and outputs time-series sensor data. For example, sensor data acquired from a 3-axis acceleration sensor with a measurement range of ± 3G (1G = 9.80665m / s2) is sampled 50 times per second and converted to data with a resolution of 10mG, for example. Thus, time-series sensor data for each of the x-, y-, and z-axis directions is obtained at intervals of 0.02 seconds. An example of time-series sensor data for one axis is shown in FIG.

  The feature extraction unit 22 extracts feature vectors from time-series sensor data. The feature vector is generated, for example, as an array of m (m is a positive integer) feature quantities. For example, a plurality of statistical features for time-series sensor data obtained in one second can be used as a plurality of feature amounts. In the case of sensor data sampled 50 times per second, the average value or dispersion value of sensor data obtained per second for each of the x, y, and z axes is used as a feature value, or discrete Fourier transform is performed on the sensor data per second. A spectral frequency value or power value at the peak of the obtained frequency spectrum is used as a feature amount, and an array obtained by combining these amounts can be used as a feature vector.

  The first recognition unit 23 recognizes the first action from the feature vector output by the feature extraction unit 22 and outputs the first action identifier to the action storage unit 25 in a sequence. The first recognition unit 23 recognizes the first behavior from the feature vector based on the first behavior model for identifying the first behavior provided as the model data by the behavior model storage unit 26, The action identifier sequence in which the values of the action identifiers are arranged in a sequence is output to the action storage unit 25.

  As a method for recognizing the first action based on the first action model, for example, a discriminator that performs clustering for classifying an input m-th order feature vector into clusters of an m-th order space, a support vector machine ( A recognizer such as SVM (support vector machine) can be used. For example, in the case of a support vector machine, a correspondence relationship between a feature vector value and a first action identifier can be constructed by preparing a plurality of learning data for each value of the first action identifier and performing machine learning. For the method using the support vector machine, for example, refer to the technical content of Japanese Patent Laid-Open No. 2008-250990 entitled “Model Derivation Method, Recognition System, Information Processing Device, Design Device, Design Method, and Program”. .

  The second recognizing unit 24 uses the nth behavior model from the behavior identifier sequence that is a temporal chain of lower abstract levels output to the behavior storage unit 25, that is, the (n−1) th behavior identifier. n actions are recognized and output to the action storage unit 25. In this case, n is 2 or more, and the second or more actions are recognized by the second recognition unit 24. The n-th behavior model is read as model data from the behavior model storage unit 26, and the second recognition unit 24 recognizes the behavior identifier sequence using a recognition algorithm corresponding to the behavior model or a recognition algorithm-dependent parameter.

  As a recognition method in the second recognition unit 24, in addition to the support vector machine described above, for example, a discriminator based on an action model such as a Hidden Markov Model (HMM) or a conditional random field (CRF). Can be used. Classifiers based on behavior models such as HMM and CRF estimate the most probable behavior from a variable-length behavior identifier sequence using a recognition algorithm such as the maximum likelihood method, the least square method, or the Viterbi method. be able to. For the technology of the discriminator based on the behavior model, for example, refer to the technical content of Japanese Patent Laid-Open No. 9-62647 entitled “Learning Method of Probability Model”.

  When recognizing the n-th action, reference is made not only to the (n-1) th action identifier but also to any of the first to n-1th action identifiers having a lower abstract level. Alternatively, a different identification algorithm may be used for each abstract level or for each action to be identified.

  The behavior storage unit 25 records the first behavior output by the first recognition unit 23 in association with the corresponding time, and records the nth behavior output by the second recognition unit 24 in association with the corresponding time. To do. The first action and the n-th action are recorded as action identifier sequences as time-series data having temporal continuity. For example, as the first action identifier sequence, the first action identifier at time 10: 10: 10.00 is “2”, the first action identifier at 10: 10: 10.02 is “2”, When the first action identifier at the time 10: 10: 10.04 seconds is “3” and the first action identifier at the time 10: 10: 10.06 seconds is “4”, the time 10:10 The first action identifier sequence from 10.00 seconds to 10: 10: 10.06 seconds is expressed as {2, 2, 3, 4}.

  The output unit 27 reads out the first to n-th actions corresponding to any specified abstract level or specified time from the action storage unit 25 and outputs them in response to an operation command input to the action recognition device 20 from the outside. There are various output methods for outputting any of the first to n actions such as displaying on a display device such as a display, printing output to a printing device such as a printer, and providing information to the outside via a network. There can be a form.

  In addition, the output unit 27 outputs an abstract level and time for an application that requires the latest action to some extent in real time, for a method that outputs actions at the latest time for a plurality of abstract levels, or for an application that does not require real-time performance. In addition to specifying, a general search means for the database such as specifying a specific value of the action identifier may be provided.

  The behavior model storage unit 26 holds model data for recognizing the first to nth behaviors. The model data of the behavior model is associated with the abstract level via the behavior model identifier, and includes, for each abstract level, various parameters depending on the identification information of the recognition algorithm and the behavior model. For example, the parameters in the case of a hidden Markov model include three parameters: an internode transition probability, a symbol output probability, and an initial node probability. For the model data such as parameters, it is necessary to obtain an optimum value in advance using learning data and to be held in the behavior model storage unit 26 together with the recognition algorithm.

  The behavior model storage unit 26 also holds a correspondence relationship between behavior identifiers and behaviors at each abstract level, and behavior identifier sequences that serve as reference values or initial values in the recognition algorithm.

FIG. 4 shows an example of the correspondence between action identifiers and actions at the first abstract level.
Here, the content of the first action corresponding to each of the first action identifiers “1, 2, 3, 4,...” Is set. FIG. 5 and FIG. 6 show an example of a behavior identifier sequence serving as a reference and a correspondence relationship between behavior identifiers and behaviors at the second and third abstract levels. Referring to FIG. 5, the content of the second action corresponding to each of the second action identifiers “1, 2, 3, 4,...” Is set, and the reference action identifier sequence is the first. It is set as a chain of action identifiers. The reference action identifier sequence is used as a reference value or an initial value in an estimation process for estimating the most likely second action in the recognition algorithm. Referring to FIG. 6, similarly, the contents of the second action corresponding to each of the third action identifiers “1, 2,...” Are set, and the reference action identifier sequence is the first action identifier sequence. It is set as a chain of two action identifiers.

  FIG. 7 shows a processing procedure of the behavior recognition method according to the present invention. The processing procedure is executed in cooperation with each unit of the action recognition device 20.

  First, the action recognition device 20 takes in sensor data in an arbitrary time range, and generates time-series sensor data at a predetermined sampling rate and resolution (step S101). Next, a plurality of feature amounts are extracted from the obtained sensor data, and a feature vector composed of an array of the plurality of feature amounts is generated (step S102). Next, the first behavior model is read from the behavior model storage unit (step S103).

  Next, the behavior recognition apparatus 20 recognizes the first behavior from the generated feature vector using a recognition algorithm based on the read behavior model (step S104). Next, the recognized first action is stored in the action storage unit (step S105). Next, the abstract level n is initialized to n = 2 (step S106), and the process proceeds to step S107.

  In step S107, the nth behavior model is read from the behavior model storage unit. Next, the success or failure of the reading is determined (step S108). If there is no n-th behavior model, it can not be raised to an abstract level higher than this, and the process proceeds to step S114 where new sensor data is awaited. The n-th behavior model read in step S107 need not be read when there are a plurality of models used for recognition for each behavior. For example, the n-th behavior model can be generated from the (n-1) th behavior identifier sequence. By obtaining n action identifiers, only the nth action model related to the nth action that may occur may be read. For example, the nth behavior model includes two types of “wrist movement” and “ankle movement”, and the (n−1) th action identifier sequence is composed only of those related to “wrist movement”. In this case, it is possible to reduce the calculation amount by limiting the n-th behavior model to be read only to “wrist movement”.

  In step S108, when the reading of the nth behavior model is successful, the behavior recognition apparatus 20 further reads the (n-1) th behavior identifier sequence from the behavior storage unit (step S109). Next, the success or failure of reading is determined (step S110). If there is no (n-1) th action identifier sequence, it is determined that the level of abstraction cannot be increased any more, and new sensor data is obtained. The process proceeds to the waiting step S114.

  Next, the behavior recognition apparatus 20 recognizes the nth behavior from the (n-1) th behavior identifier sequence using a recognition algorithm based on the read behavior model (step S111). Selection of the recognition algorithm based on the read behavior model can be realized by providing the behavior model with an identifier indicating the recognition algorithm. This identifier can be selected by learning independently for each of a plurality of recognition algorithms at the time of learning, and evaluating the recognition rate by those recognition algorithms. For example, the identifier of the recognition algorithm with the best recognition rate and the parameter obtained by learning can be stored in the behavior model storage unit as a behavior model. Next, the recognized nth action is stored in the action storage unit (step S112). Next, the abstract level is updated, that is, initialized to n = n + 1 (step S113), and the process proceeds to step S107. The action recognition device 20 repeats the process for the next n + 1th action. As a result, a plurality of behaviors at the abstract level that follows n = 2, 3, 4,... Are recognized.

  On the other hand, when it progresses to step S114, it waits for the timing at which new sensor data is taken in (S114). If new data is acquired, the process is restarted from step S101, and if not, the process ends (step S115). The resumption of processing may be repeated every time sensor data is sampled. Further, it may be processed after accumulating sensor data to some extent such as several seconds. If the output of the recognition result is not required in real time, the recognition process may be performed after accumulating sensor data for a period of one day or one week, for example. For example, when sensor data is sampled at 50 Hz, the recognition process may be repeated 50 times per second, or the recognition process may be repeated in units of 1 second.

  FIG. 8 shows an example of the correspondence between the sensor data and the recognized action identifier sequence. Here, as the first action identifier sequence for the sensor data from time 0:00:00 to time 0: 0: 20, “1 1 1 1 1 2 2 3 3 4 4... “” Is recognized, “1 2 8 4” is recognized as the second action identifier sequence, and “1 2...” Is recognized as the third action identifier sequence.

  As described above, it is possible to recognize a plurality of actions at a plurality of abstract levels by collecting a plurality of actions and performing a recognition process, recognizing a action with a higher level of abstraction, and repeating this.

  In the above embodiments, by applying the present invention, it is possible to recognize and recognize one action at a plurality of abstraction levels, and it is possible to recognize actions with higher expressive power. In addition, if the level of abstraction is raised, it is possible to recognize not only actions but also human states. For example, when a sequence of actions with a low abstract level changes between when tired and when not tired, the action recognition with a high abstract level can recognize the change and recognize it as another action. In this case, an action with a high level of abstraction can be expressed as a state or situation of being tired / not tired.

  In the above embodiment, a person has been described as an action recognition target individual, but there is no limitation according to the present invention, and not only a person's action but also a movement of an object or robot, an object that can be carried by a plurality of persons Can be recognized at multiple levels of abstraction.

10 individuals
11 to 13 Three-axis acceleration sensor 20 Action recognition device 21 Input unit 22 Feature extraction unit 23 First recognition unit 24 Behavior storage unit 25 Second recognition unit 26 Behavior model storage unit 27 Output unit

Claims (8)

It is a behavior recognition method in a behavior recognition device that captures sensor data for measuring a behavior of an individual and recognizes the behavior of the individual from the sensor data,
An action model holding step for holding model data corresponding to an action model and an action recognition algorithm for each of a plurality of abstract levels;
A feature vector is extracted from the sensor data, and a recognition process using model data corresponding to a first abstract level of the plurality of abstract levels is performed on the feature vector, thereby the first abstract level. A first recognition step of recognizing the action identifier and storing the action identifier in association with the acquisition time of the sensor data ;
For each of the plurality of abstract levels, a recognition process using model data corresponding to the abstract level is performed on a sequence composed of one or more action identifiers at a level lower than the abstract level. A second recognition step of recognizing the action identifier of the abstract level and storing the action identifier in association with the acquisition time of the sensor data ;
Hints, for at least one level of abstraction, including a second level of abstraction, activity recognition method characterized by repeating said second recognition step. Said first recognition step and the second recognition step, that in accordance with the designated time and the designated abstraction level is inputted external, further comprising an output step of outputting an action identifier corresponding to the designated time and the designated abstraction level The action recognition method according to claim 1, wherein:   The behavior recognition method according to claim 1, wherein the behavior model retaining step retains model data corresponding to a discriminator that recognizes a behavior identifier by performing clustering on the feature vector.   The action model holding step holds model data corresponding to a support vector machine for recognizing an action identifier of the abstract level from the feature vector or a sequence of one or more action identifiers at a level lower than the abstract level. The action recognition method according to claim 1, wherein:   In the behavior model holding step, model data corresponding to a recognition algorithm for recognizing a behavior identifier of the abstract level by applying a hidden Markov model to a sequence composed of one or a plurality of behavior identifiers at a level lower than the abstract level. The behavior recognition method according to claim 1, wherein:   The action model holding step holds model data corresponding to a recognition algorithm for recognizing an action identifier of the abstract level by applying a conditional random field model to a sequence composed of action identifiers lower than the abstract level. The action recognition method according to claim 1, wherein: A behavior recognition device that captures sensor data for measuring an individual's behavior and recognizes the behavior of the individual from the sensor data,
Action model holding means for holding model data corresponding to the action model and action recognition algorithm for each of a plurality of abstract levels;
A feature vector is extracted from the sensor data, and a recognition process using model data corresponding to a first abstract level of the plurality of abstract levels is performed on the feature vector, thereby the first abstract level. A first recognition means for recognizing the action identifier and storing the action identifier in association with the acquisition time of the sensor data ;
For each of the plurality of abstract levels, a recognition process using model data corresponding to the abstract level is performed on a sequence composed of one or more action identifiers at a level lower than the abstract level. Second recognition means for recognizing the action identifier of the abstract level and storing it in association with the acquisition time of the sensor data ;
And the second recognition means repeats an operation of recognizing and storing an action identifier of the abstract level for at least one abstract level including the second abstract level. A behavior recognition program for capturing a sensor data for measuring an individual's behavior and executing a procedure for recognizing the behavior of the individual from the sensor data in a computer,
An action model holding step for holding model data corresponding to an action model and an action recognition algorithm for each of a plurality of abstract levels;
A feature vector is extracted from the sensor data, and a recognition process using model data corresponding to a first abstract level of the plurality of abstract levels is performed on the feature vector, thereby the first abstract level. A first recognition step of recognizing the action identifier and storing the action identifier in association with the acquisition time of the sensor data ;
For each of the plurality of abstract levels, a recognition process using model data corresponding to the abstract level is performed on a sequence composed of one or more action identifiers at a level lower than the abstract level. A second recognition step of recognizing the action identifier of the abstract level and storing the action identifier in association with the acquisition time of the sensor data ;
It hints, for at least one level of abstraction, including a second level of abstraction, behavioral recognition program, which comprises the steps of repeating the second recognition step.

Images