US20220240638A9 - Method and system for activity classification - Google Patents
Method and system for activity classification Download PDFInfo
- Publication number
- US20220240638A9 US20220240638A9 US17/173,978 US202117173978A US2022240638A9 US 20220240638 A9 US20220240638 A9 US 20220240638A9 US 202117173978 A US202117173978 A US 202117173978A US 2022240638 A9 US2022240638 A9 US 2022240638A9
- Authority
- US
- United States
- Prior art keywords
- recurrent neural
- neural network
- inner product
- skeleton
- trained
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- A—HUMAN NECESSITIES
- A41—WEARING APPAREL
- A41D—OUTERWEAR; PROTECTIVE GARMENTS; ACCESSORIES
- A41D27/00—Details of garments or of their making
- A41D27/20—Pockets; Making or setting-in pockets
-
- A—HUMAN NECESSITIES
- A45—HAND OR TRAVELLING ARTICLES
- A45C—PURSES; LUGGAGE; HAND CARRIED BAGS
- A45C1/00—Purses; Money-bags; Wallets
- A45C1/02—Purses
- A45C1/024—Purses fitted in handbags or the like
-
- A—HUMAN NECESSITIES
- A45—HAND OR TRAVELLING ARTICLES
- A45C—PURSES; LUGGAGE; HAND CARRIED BAGS
- A45C1/00—Purses; Money-bags; Wallets
- A45C1/06—Wallets; Notecases
-
- A—HUMAN NECESSITIES
- A45—HAND OR TRAVELLING ARTICLES
- A45C—PURSES; LUGGAGE; HAND CARRIED BAGS
- A45C13/00—Details; Accessories
- A45C13/18—Devices to prevent theft or loss of purses, luggage or hand carried bags
-
- A—HUMAN NECESSITIES
- A45—HAND OR TRAVELLING ARTICLES
- A45C—PURSES; LUGGAGE; HAND CARRIED BAGS
- A45C13/00—Details; Accessories
- A45C13/18—Devices to prevent theft or loss of purses, luggage or hand carried bags
- A45C13/185—Devices to prevent theft or loss of purses, luggage or hand carried bags of purses, money-bags or wallets
-
- A—HUMAN NECESSITIES
- A45—HAND OR TRAVELLING ARTICLES
- A45F—TRAVELLING OR CAMP EQUIPMENT: SACKS OR PACKS CARRIED ON THE BODY
- A45F5/00—Holders or carriers for hand articles; Holders or carriers for use while travelling or camping
- A45F5/02—Fastening articles to the garment
- A45F5/022—Fastening articles to the garment within pockets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
- G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Definitions
- This disclosure relates to methods and systems for the identification and classification of human gestures and actions.
- the disclosure relates to the identification and classification using skeletons representing human poses.
- activity classifiers can broadly be split into two categories: a. Methods that directly consume video data as input; and b. Methods that consume skeleton data, representing the positions of human joints.
- the first strategy is somewhat more general, as it requires no assumptions about the representation of humans in each frame. However, methods of this class may be more difficult to train, transfer to new application domains and may over-estimate the relationship between static visual cues and co-occurring human activities.
- the second strategy decouples the problems of identifying humans in a frame and identifying their activities. This decoupling can be advantageous with respect to fast training of compact models and with respect to the portability of these models to other domains.
- the vast majority of these methods require 3D skeleton data, which may be challenging to infer robustly and may require specialized hardware beyond standard RGB cameras.
- Another challenge faced by this strategy is the lack of contextual information available in the input skeleton data.
- This disclosure is directed to an activity classifier system, for classifying human activities using 2D skeleton data.
- the system includes a skeleton preprocessor that transforms the 2D skeleton data into transformed skeleton data, the transformed skeleton data comprising scaled, relative joint positions and relative joint velocities. It also includes a gesture classifier comprising a first recurrent neural network that receives the transformed skeleton data, and is trained to identify the most probable of a plurality of gestures. There is also an action classifier comprising a second recurrent neural network that receives information from the first recurrent neural networks and is trained to identify the most probable of a plurality of actions.
- FIG. 1 is a representation of the high-level architecture of an embodiment of a activity classifier.
- FIG. 2 is an example representation of a 2D skeleton with 19 joints, overlaid onto a still image from which the skeleton had been derived.
- FIGS. 3A to 3D are a series of example representations of a 2D skeleton with gesture probabilities and recognized action for four snapshots from a video, overlaid with frames from the video.
- FIG. 4 represents the flow of information through a video processing system, including an activity classifier.
- An activity classifier generally identifies activity, such as “lifting a ball” from source material, such as a video.
- An activity classifier may use video, or partially analyzed video, such as a skeleton representing the positions of a set of labeled joints of a person's body.
- the skeleton data may be 2D, encoded as an array of X and Y coordinate positions within the video frame, for each joint.
- the number and anatomical locations of labeled joints may vary between implementations as best suits the requirements of the situation and specific task to be performed.
- the X and Y coordinates for the ‘elbow’ joint may be provided for each frame of a video sequence. Similar information may be provided for other joints in the model. This can often be extracted from video sources, even under varying lighting conditions and scene environments.
- 3D skeleton data that includes X, Y, Z location information for joints.
- 3D skeleton data is often more difficult to extract, particularly from standard video and may require specialized capture equipment and higher computational costs to capture.
- 3D skeleton data may be impossible to infer unambiguously from a single RGB video source in some cases.
- skeleton data 50 for a single frame is indicated as dots for each joint and lines connecting associated joints.
- the skeleton is superimposed on the associated frame.
- the skeleton data includes 19 joints: feet, ankles, knees, hips, shoulders, elbows, wrists, eyes, ears and nose.
- the activity classifier 10 may also use contextual information about the existence and/or class of objects that the person is interacting with.
- This information may be encoded as one or more Boolean variables or class index variables representing the existence and/or class of the object being interacted with from different parts of the human body. These parts of the body may include the left hand, the right hand or any other targeted body part which may be expected to interact with the environment.
- the contextual information may be: ‘ornament’/‘left hand’. The contextual information may be helpful to the classifier because activities may be indistinguishable without context.
- This pair of input information, feature sets representing skeleton joint positions and information about contextual objects, may be provided in an order corresponding to a temporal sequence.
- the temporal sequence may be obtained by extracting information from sequential frames of a video.
- the skeleton data may be provided by a different system or system component that extracts labelled joint positions from RGB video or some other source.
- contextual information about objects may also derive from an external system, such as an object detection model.
- the activity classifier 10 may comprise a skeleton preprocessor 20 , a gesture classifier 30 and an action classifier 40 .
- the components may be separate software modules, separate hardware units or portions of one or more software or hardware components.
- the software modules may be written in the Python programming language with suitable modules, such as Caffe.
- the hardware includes sufficient processing and memory capacity to implement the two RNNs for the classifier.
- the activity classifier may operate on a real-time stream of skeleton data such that the output of the activity classifier may indicate the most likely activity being conducted in the skeleton data at approximately the same time as the activity is represented in the skeleton data.
- the activity classifier may operate on video streams and provide gesture and activity information in near real-time.
- the system may operate in associate with broadcast, live video or other video streams to provide gesture and activity metadata in association with the video stream.
- video may be captured 405 , such as using a video camera, antenna, TV receiver or other means of obtaining a video stream.
- the video may be stored or buffered to be streamed and processed at some time after being captured.
- the video stream may be transmitted 410 and displayed 415 such as on a video screen or computer screen.
- the video stream may also be communicated to a skeleton processor 420 for generating 2D skeletons of humans that appear in the screen.
- the skeleton data may be communicated to the activity classifier 425 which, as described in more detail elsewhere, determines the likely activities being represented in the 2D skeleton data and hence in the video stream.
- the output of the activity classifier may be displayed or used in parallel with or instead of the video stream.
- the output of the activity classifier may be provided to other systems, such as robotic interfaces, security systems, frictionless retail systems, software systems or displayed to human users.
- the output of the activity classifier may be displayed without the video stream in order to anonymize persons in the video and display their actions and/or gestures and/or skeletons alone, without revealing their face, body or clothing to a user.
- the gesture classifier component 30 and the action classifier component 40 are both based on deep recurrent neural network (RNN) models with recurrent units, such as long-short term memory (LSTM) units.
- RNN deep recurrent neural network
- LSTM long-short term memory
- An RNN is a kind of deep neural network machine learning model that encodes temporal information through recurrent memory cells.
- the RNN models may be trained on skeleton data that includes manually classified gesture and actions class annotations, or other similar source material.
- the training data may include temporal sequential skeleton data.
- a skeleton preprocessor 20 may perform a set of basic transforms on the input joint position data obtained from the skeleton data.
- the skeleton data may represent the human pose in a way that is expected to be most useful for the subsequent classifiers and invariant to transforms that are irrelevant to identification of activities.
- Such irrelevant factors may include the distance from the camera and the position in frame.
- Transforms completed by the skeleton preprocessor may include: a. Temporally smoothing the positions of the joints. b. Transforming the joint positions from screen space coordinates to coordinates that are relative to the persons' position. For example, the joint positions may be determined relative to the neck or pelvis joint positions. c. Scaling the relative joint positions to be relative to the person's height. The person's height may be estimated from other features of the skeleton data by the skeleton preprocessor. d. Computing the velocity of each scaled joint position.
- the output of the skeleton preprocessor is a set of scaled, relative joint positions and a set of scaled, joint velocities. Some or all of these transforms may not be needed, depending on the quality and processing already completed on the source skeleton data. For example, if all the source skeleton data is taken from a fixed camera distance and location, scaling may not be required.
- a gesture classifier component 30 of the activity classifier 10 takes as input the pre-processed joint features from the skeleton preprocessor.
- a gesture classifier infers a set of probabilities of different gestures.
- a pre-selected set of gesture classes may be determined in advance.
- the set of gesture classes is intended to comprise simple movements of the human body and generalizes specific human movements.
- the gesture classes may include reach arm, turn head, point finger.
- the gesture classes may include a neutral pose, or ‘none’ class.
- the set of classes may also include an ‘other’ class to represent the space of non-neutral gestures that are not of specific interest.
- examples of pre-selected gestures may include ‘none’, ‘retract’, ‘reach’, ‘low retract’, ‘low reach’.
- FIG. 3A a skeleton is shown that has been identified as most likely not having any apparent gesture, reflected in the high probability of ‘none’.
- the skeleton is identified as most probably in the ‘reach’ gesture.
- the most probable gesture is identified as ‘retract’.
- the most probable gesture is identified as ‘retract’ with ‘none’ being the second most probable.
- the action classifier may consider this sequence of gestures, of which the Figures are only representative snapshots of a video sequences and the 2D skeletons from the sequence, as ‘ornament added to tree’.
- the pink boxes represent likely locations of contextual objects being manipulated by the human.
- the label stating ‘holding’ in FIG. 3A and ‘not holding’ in FIGS. 3B to 3D represent the most likely status of whether the human is holding some contextual object.
- gesture classifier component While the gesture classifier component only infers the gesture probabilities of a skeleton at a single point in time, it may rely on a memory of previous states to perform the inference. The component therefore may assume a sequential feed of data.
- the gesture classifier component may operate independently of the action classifier, and without contextual object information.
- a gesture classifier component is based on a deep RNN with inner-product (IP) layers, rectified linear unit (ReLU) layers, and recurrent layers such as LSTM layers.
- IP inner-product
- ReLU rectified linear unit
- a gesture classifier component comprises one or more pairs of IP and ReLU layers, followed by one or more LSTM layers, followed by zero or more pairs of IP and ReLU layers, followed by a final IP layer.
- the result of this final IP layer may be passed to a subsequent action classifier component and/or output as inferred class probabilities, possibly after being normalized, such as with softmax normalization. In the latter case, these gesture class probabilities may be used to infer a most probable gesture class.
- the intention of placing LSTM layers between blocks of IP, ReLU groups is to enable temporally informed inferencing by the LSTM to occur in a transformed space where information about individual joints are combined and/or filtered in a manner found to be most discriminative by the training procedure.
- the gesture classifier may comprise between five to ten stacked IP, ReLU and LSTM layers and each hidden layer may comprise dozens to hundreds of hidden units. The specific value of these parameters may depend on the amount of available training data and the complexity of the gestures and actions being detected.
- An action classifier component of the activity classifier builds on the gesture classifier component.
- the action classifier may accept features extracted from a hidden layer in the gesture classifier, such as the layer prior to the final IP layer of the gesture classifier RNN.
- gesture classifier features are assumed to provide rich information pertinent to inferring gesture probabilities and are also independent of the specific number of classes being predicted by the gesture classifier.
- This architecture is designed to leverage salient features learned by the gesture classifier component. This is done under the assumption that these features are also pertinent for predicting actions.
- an action classifier component may also use contextual information about the existence or class of object or objects that the human is interacting with.
- the action classifier component is a deep RNN and may require a sequential feed of input data in order to leverage temporal cues via the memory units in the LSTM layers.
- an action classifier component comprises one or more pairs of IP and ReLU layers, followed by one or more LSTM layers, followed by zero or more pairs of IP and ReLU layers, followed by a final IP layer with softmax normalization.
- the optimal number of layers and number of hidden units in each layer may be similar to the respective parameters of the gesture classifier component and may depend on the amount of available training data and the complexity of the actions and actions being detected.
- the output of the action classifier component is a set of action class probabilities for the current input, given the history of previous inputs.
- a pre-selected set of action classes may be determined in advance.
- the set of action classes is intended to comprise complex or compound movements of the human body, possibly involving interactions with contextual objects, and generalize specific human actions.
- the gesture classifier component and action classifier component RNNs contain trainable parameters that define the behaviours of their respective components. These parameters may be learned by training the gesture and action component RNNs using labelled gesture and action training data.
- the gesture classifier component RNN may be trained first, independently of the action component RNN.
- Gesture training data may comprise sequences of real 2D skeleton sequences inferred from captured video or synthetically generated sequences of skeletons and corresponding gesture class labels for each skeleton.
- the action classifier component RNN may be trained in the context of the entire activity classifier with a pre-trained gesture classifier component RNN that either has fixed training parameters or tunable training parameters, which are updated based on training of the entire system.
- Action training data comprises sequences of real or synthetic 2D skeleton data and corresponding action class labels for each skeleton.
- Appropriate training data may comprise thousands of sequences demonstrating many examples of each gesture or action to be identified by the classifier.
- the systems and methods described above provide a neural-network based activity classifier to detect activities hierarchically, with a low-level gesture recognition as parts of higher-level action classifier.
- the system network architecture relates to the hierarchy of complexity of the activities being detected.
- This architecture provides: a. simultaneous inference of human activities at multiple scales of complexity; b. Modularity that facilitates evaluation, testing, debugging, and replacement of specific components of the model. This can be important for experimenting or migrating the model to new domains of activity classes; and c. Ease of model training due to the predominantly linear structure of the network as compared to more complex network topologies, which may require many training steps.
Abstract
Description
- This application is a continuation and claims priority under 35 USC 120 of U.S. application Ser. No. 16/267,493, filed Feb. 14, 2019 that is turn claims the benefit under 35 USC 119(b) to Canadian Patent Application 2,995,242 filed Feb. 15, 2018, all of which are incorporated herein by reference.
- This disclosure relates to methods and systems for the identification and classification of human gestures and actions. In particular, the disclosure relates to the identification and classification using skeletons representing human poses.
- Early approaches to recognising and classifying human activities from video data employed a variety of different strategies and inference methods, including simple classifiers such as support-vector machines (SVM) and statistical state models such as hidden Markov models (HMM). These methods were often applicable only to highly constrained use cases and were not robust to the variations present in real-world data.
- Modern robust methods for performing human activity classification largely rely on deep neural networks; machine learning models that have been proven across a variety of application domains. Among these methods, activity classifiers can broadly be split into two categories: a. Methods that directly consume video data as input; and b. Methods that consume skeleton data, representing the positions of human joints.
- The first strategy is somewhat more general, as it requires no assumptions about the representation of humans in each frame. However, methods of this class may be more difficult to train, transfer to new application domains and may over-estimate the relationship between static visual cues and co-occurring human activities.
- The second strategy decouples the problems of identifying humans in a frame and identifying their activities. This decoupling can be advantageous with respect to fast training of compact models and with respect to the portability of these models to other domains. The vast majority of these methods, however, require 3D skeleton data, which may be challenging to infer robustly and may require specialized hardware beyond standard RGB cameras. Another challenge faced by this strategy is the lack of contextual information available in the input skeleton data.
- There is therefore a need for improved methods and systems for classifying activities from skeleton data.
- This disclosure is directed to an activity classifier system, for classifying human activities using 2D skeleton data. The system includes a skeleton preprocessor that transforms the 2D skeleton data into transformed skeleton data, the transformed skeleton data comprising scaled, relative joint positions and relative joint velocities. It also includes a gesture classifier comprising a first recurrent neural network that receives the transformed skeleton data, and is trained to identify the most probable of a plurality of gestures. There is also an action classifier comprising a second recurrent neural network that receives information from the first recurrent neural networks and is trained to identify the most probable of a plurality of actions.
- In drawings which illustrate by way of example only a preferred embodiment of the disclosure.
-
FIG. 1 is a representation of the high-level architecture of an embodiment of a activity classifier. -
FIG. 2 is an example representation of a 2D skeleton with 19 joints, overlaid onto a still image from which the skeleton had been derived. -
FIGS. 3A to 3D are a series of example representations of a 2D skeleton with gesture probabilities and recognized action for four snapshots from a video, overlaid with frames from the video. -
FIG. 4 represents the flow of information through a video processing system, including an activity classifier. - An activity classifier generally identifies activity, such as “lifting a ball” from source material, such as a video.
- An activity classifier may use video, or partially analyzed video, such as a skeleton representing the positions of a set of labeled joints of a person's body.
- The skeleton data may be 2D, encoded as an array of X and Y coordinate positions within the video frame, for each joint. The number and anatomical locations of labeled joints may vary between implementations as best suits the requirements of the situation and specific task to be performed. For example, the X and Y coordinates for the ‘elbow’ joint may be provided for each frame of a video sequence. Similar information may be provided for other joints in the model. This can often be extracted from video sources, even under varying lighting conditions and scene environments.
- In contrast, some systems use 3D skeleton data, that includes X, Y, Z location information for joints. 3D skeleton data is often more difficult to extract, particularly from standard video and may require specialized capture equipment and higher computational costs to capture. Furthermore, 3D skeleton data may be impossible to infer unambiguously from a single RGB video source in some cases.
- Other systems that accept raw video data as input, may require a much larger number of learnable parameters compared to those requirements for 2D skeleton data, making them more difficult and computationally intensive to train, tune and run. Such networks may rely on aspects of the video source, such as background information rather than the human activities, a reliance that is avoided when the network is only provided with 2D skeletons.
- With reference to
FIG. 2 , skeleton data 50 for a single frame is indicated as dots for each joint and lines connecting associated joints. For reference, the skeleton is superimposed on the associated frame. In this example, the skeleton data includes 19 joints: feet, ankles, knees, hips, shoulders, elbows, wrists, eyes, ears and nose. - The activity classifier 10 may also use contextual information about the existence and/or class of objects that the person is interacting with. This information may be encoded as one or more Boolean variables or class index variables representing the existence and/or class of the object being interacted with from different parts of the human body. These parts of the body may include the left hand, the right hand or any other targeted body part which may be expected to interact with the environment. For a particular example, the contextual information may be: ‘ornament’/‘left hand’. The contextual information may be helpful to the classifier because activities may be indistinguishable without context.
- This pair of input information, feature sets representing skeleton joint positions and information about contextual objects, may be provided in an order corresponding to a temporal sequence. The temporal sequence may be obtained by extracting information from sequential frames of a video. The skeleton data may be provided by a different system or system component that extracts labelled joint positions from RGB video or some other source.
- Similarly, contextual information about objects may also derive from an external system, such as an object detection model.
- With reference to
FIG. 1 , the activity classifier 10 may comprise a skeleton preprocessor 20, a gesture classifier 30 and an action classifier 40. - These components may be separate software modules, separate hardware units or portions of one or more software or hardware components. For example, the software modules may be written in the Python programming language with suitable modules, such as Caffe. The hardware includes sufficient processing and memory capacity to implement the two RNNs for the classifier.
- The activity classifier may operate on a real-time stream of skeleton data such that the output of the activity classifier may indicate the most likely activity being conducted in the skeleton data at approximately the same time as the activity is represented in the skeleton data. In this way, the activity classifier may operate on video streams and provide gesture and activity information in near real-time. The system may operate in associate with broadcast, live video or other video streams to provide gesture and activity metadata in association with the video stream.
- With reference to
FIG. 4 , video may be captured 405, such as using a video camera, antenna, TV receiver or other means of obtaining a video stream. The video may be stored or buffered to be streamed and processed at some time after being captured. The video stream may be transmitted 410 and displayed 415 such as on a video screen or computer screen. The video stream may also be communicated to askeleton processor 420 for generating 2D skeletons of humans that appear in the screen. - The skeleton data may be communicated to the
activity classifier 425 which, as described in more detail elsewhere, determines the likely activities being represented in the 2D skeleton data and hence in the video stream. - The output of the activity classifier may be displayed or used in parallel with or instead of the video stream. The output of the activity classifier may be provided to other systems, such as robotic interfaces, security systems, frictionless retail systems, software systems or displayed to human users. The output of the activity classifier may be displayed without the video stream in order to anonymize persons in the video and display their actions and/or gestures and/or skeletons alone, without revealing their face, body or clothing to a user.
- The gesture classifier component 30 and the action classifier component 40 are both based on deep recurrent neural network (RNN) models with recurrent units, such as long-short term memory (LSTM) units. An RNN is a kind of deep neural network machine learning model that encodes temporal information through recurrent memory cells.
- The RNN models may be trained on skeleton data that includes manually classified gesture and actions class annotations, or other similar source material. The training data may include temporal sequential skeleton data.
- A skeleton preprocessor 20 may perform a set of basic transforms on the input joint position data obtained from the skeleton data. As a result of the basic transforms, the skeleton data may represent the human pose in a way that is expected to be most useful for the subsequent classifiers and invariant to transforms that are irrelevant to identification of activities. Such irrelevant factors may include the distance from the camera and the position in frame.
- Transforms completed by the skeleton preprocessor may include: a. Temporally smoothing the positions of the joints. b. Transforming the joint positions from screen space coordinates to coordinates that are relative to the persons' position. For example, the joint positions may be determined relative to the neck or pelvis joint positions. c. Scaling the relative joint positions to be relative to the person's height. The person's height may be estimated from other features of the skeleton data by the skeleton preprocessor. d. Computing the velocity of each scaled joint position.
- The output of the skeleton preprocessor is a set of scaled, relative joint positions and a set of scaled, joint velocities. Some or all of these transforms may not be needed, depending on the quality and processing already completed on the source skeleton data. For example, if all the source skeleton data is taken from a fixed camera distance and location, scaling may not be required.
- A gesture classifier component 30 of the activity classifier 10 takes as input the pre-processed joint features from the skeleton preprocessor. A gesture classifier infers a set of probabilities of different gestures.
- A pre-selected set of gesture classes may be determined in advance. The set of gesture classes is intended to comprise simple movements of the human body and generalizes specific human movements. For example, the gesture classes may include reach arm, turn head, point finger. The gesture classes may include a neutral pose, or ‘none’ class. The set of classes may also include an ‘other’ class to represent the space of non-neutral gestures that are not of specific interest.
- With reference to
FIGS. 3A to 3D , examples of pre-selected gestures may include ‘none’, ‘retract’, ‘reach’, ‘low retract’, ‘low reach’. InFIG. 3A , a skeleton is shown that has been identified as most likely not having any apparent gesture, reflected in the high probability of ‘none’. - In
FIG. 3B , the skeleton is identified as most probably in the ‘reach’ gesture. InFIG. 3C , the most probable gesture is identified as ‘retract’. InFIG. 3D , the most probable gesture is identified as ‘retract’ with ‘none’ being the second most probable. The action classifier may consider this sequence of gestures, of which the Figures are only representative snapshots of a video sequences and the 2D skeletons from the sequence, as ‘ornament added to tree’. - With reference to
FIGS. 3A to 3D , the pink boxes represent likely locations of contextual objects being manipulated by the human. The label stating ‘holding’ inFIG. 3A and ‘not holding’ inFIGS. 3B to 3D represent the most likely status of whether the human is holding some contextual object. - While the gesture classifier component only infers the gesture probabilities of a skeleton at a single point in time, it may rely on a memory of previous states to perform the inference. The component therefore may assume a sequential feed of data. The gesture classifier component may operate independently of the action classifier, and without contextual object information.
- A gesture classifier component is based on a deep RNN with inner-product (IP) layers, rectified linear unit (ReLU) layers, and recurrent layers such as LSTM layers. In particular, a gesture classifier component comprises one or more pairs of IP and ReLU layers, followed by one or more LSTM layers, followed by zero or more pairs of IP and ReLU layers, followed by a final IP layer. The result of this final IP layer may be passed to a subsequent action classifier component and/or output as inferred class probabilities, possibly after being normalized, such as with softmax normalization. In the latter case, these gesture class probabilities may be used to infer a most probable gesture class. The intention of placing LSTM layers between blocks of IP, ReLU groups is to enable temporally informed inferencing by the LSTM to occur in a transformed space where information about individual joints are combined and/or filtered in a manner found to be most discriminative by the training procedure.
- The gesture classifier may comprise between five to ten stacked IP, ReLU and LSTM layers and each hidden layer may comprise dozens to hundreds of hidden units. The specific value of these parameters may depend on the amount of available training data and the complexity of the gestures and actions being detected.
- An action classifier component of the activity classifier builds on the gesture classifier component. The action classifier may accept features extracted from a hidden layer in the gesture classifier, such as the layer prior to the final IP layer of the gesture classifier RNN.
- These features are assumed to provide rich information pertinent to inferring gesture probabilities and are also independent of the specific number of classes being predicted by the gesture classifier. This architecture is designed to leverage salient features learned by the gesture classifier component. This is done under the assumption that these features are also pertinent for predicting actions.
- As mentioned above, an action classifier component may also use contextual information about the existence or class of object or objects that the human is interacting with.
- Similar to a gesture classifier component, the action classifier component is a deep RNN and may require a sequential feed of input data in order to leverage temporal cues via the memory units in the LSTM layers. Specifically, an action classifier component comprises one or more pairs of IP and ReLU layers, followed by one or more LSTM layers, followed by zero or more pairs of IP and ReLU layers, followed by a final IP layer with softmax normalization. The optimal number of layers and number of hidden units in each layer may be similar to the respective parameters of the gesture classifier component and may depend on the amount of available training data and the complexity of the actions and actions being detected.
- Also, similar to a gesture classifier component, the output of the action classifier component is a set of action class probabilities for the current input, given the history of previous inputs. A pre-selected set of action classes may be determined in advance. The set of action classes is intended to comprise complex or compound movements of the human body, possibly involving interactions with contextual objects, and generalize specific human actions.
- The gesture classifier component and action classifier component RNNs contain trainable parameters that define the behaviours of their respective components. These parameters may be learned by training the gesture and action component RNNs using labelled gesture and action training data.
- The gesture classifier component RNN may be trained first, independently of the action component RNN. Gesture training data may comprise sequences of real 2D skeleton sequences inferred from captured video or synthetically generated sequences of skeletons and corresponding gesture class labels for each skeleton.
- The action classifier component RNN may be trained in the context of the entire activity classifier with a pre-trained gesture classifier component RNN that either has fixed training parameters or tunable training parameters, which are updated based on training of the entire system. Action training data comprises sequences of real or synthetic 2D skeleton data and corresponding action class labels for each skeleton.
- Appropriate training data may comprise thousands of sequences demonstrating many examples of each gesture or action to be identified by the classifier.
- The systems and methods described above provide a neural-network based activity classifier to detect activities hierarchically, with a low-level gesture recognition as parts of higher-level action classifier. The system network architecture relates to the hierarchy of complexity of the activities being detected.
- This architecture provides: a. simultaneous inference of human activities at multiple scales of complexity; b. Modularity that facilitates evaluation, testing, debugging, and replacement of specific components of the model. This can be important for experimenting or migrating the model to new domains of activity classes; and c. Ease of model training due to the predominantly linear structure of the network as compared to more complex network topologies, which may require many training steps.
- Various embodiments of the present disclosure having been thus described in detail by way of example, it will be apparent to those skilled in the art that variations and modifications may be made without departing from the disclosure. The disclosure includes all such variations and modifications as fall within the scope of the appended claims.
Claims (16)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/173,978 US11783183B2 (en) | 2018-02-15 | 2021-02-11 | Method and system for activity classification |
US18/477,482 US20240028896A1 (en) | 2018-02-15 | 2023-09-28 | Method and system for activity classification |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CA2995242 | 2018-02-15 | ||
CA2995242A CA2995242A1 (en) | 2018-02-15 | 2018-02-15 | Method and system for activity classification |
US16/276,493 US10949658B2 (en) | 2018-02-15 | 2019-02-14 | Method and system for activity classification |
US17/173,978 US11783183B2 (en) | 2018-02-15 | 2021-02-11 | Method and system for activity classification |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/267,493 Continuation US11241070B2 (en) | 2018-02-07 | 2019-02-05 | Anti-theft pocket insert |
US16/276,493 Continuation US10949658B2 (en) | 2018-02-15 | 2019-02-14 | Method and system for activity classification |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/477,482 Continuation US20240028896A1 (en) | 2018-02-15 | 2023-09-28 | Method and system for activity classification |
Publications (3)
Publication Number | Publication Date |
---|---|
US20210161266A1 US20210161266A1 (en) | 2021-06-03 |
US20220240638A9 true US20220240638A9 (en) | 2022-08-04 |
US11783183B2 US11783183B2 (en) | 2023-10-10 |
Family
ID=67541740
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/276,493 Active 2039-02-27 US10949658B2 (en) | 2018-02-15 | 2019-02-14 | Method and system for activity classification |
US17/173,978 Active US11783183B2 (en) | 2018-02-15 | 2021-02-11 | Method and system for activity classification |
US18/477,482 Pending US20240028896A1 (en) | 2018-02-15 | 2023-09-28 | Method and system for activity classification |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/276,493 Active 2039-02-27 US10949658B2 (en) | 2018-02-15 | 2019-02-14 | Method and system for activity classification |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/477,482 Pending US20240028896A1 (en) | 2018-02-15 | 2023-09-28 | Method and system for activity classification |
Country Status (2)
Country | Link |
---|---|
US (3) | US10949658B2 (en) |
CA (1) | CA2995242A1 (en) |
Families Citing this family (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10354168B2 (en) * | 2016-04-11 | 2019-07-16 | A2Ia S.A.S. | Systems and methods for recognizing characters in digitized documents |
GB2560177A (en) | 2017-03-01 | 2018-09-05 | Thirdeye Labs Ltd | Training a computational neural network |
GB2560387B (en) * | 2017-03-10 | 2022-03-09 | Standard Cognition Corp | Action identification using neural networks |
US10853965B2 (en) | 2017-08-07 | 2020-12-01 | Standard Cognition, Corp | Directional impression analysis using deep learning |
US11200692B2 (en) | 2017-08-07 | 2021-12-14 | Standard Cognition, Corp | Systems and methods to check-in shoppers in a cashier-less store |
US10474988B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Predicting inventory events using foreground/background processing |
US10474991B2 (en) | 2017-08-07 | 2019-11-12 | Standard Cognition, Corp. | Deep learning-based store realograms |
US11250376B2 (en) | 2017-08-07 | 2022-02-15 | Standard Cognition, Corp | Product correlation analysis using deep learning |
US11232687B2 (en) | 2017-08-07 | 2022-01-25 | Standard Cognition, Corp | Deep learning-based shopper statuses in a cashier-less store |
US10650545B2 (en) | 2017-08-07 | 2020-05-12 | Standard Cognition, Corp. | Systems and methods to check-in shoppers in a cashier-less store |
US10691969B2 (en) * | 2017-11-06 | 2020-06-23 | EagleSens Systems Corporation | Asynchronous object ROI detection in video mode |
WO2020000096A1 (en) * | 2018-06-29 | 2020-01-02 | Wrnch Inc. | Human pose analysis system and method |
EP3624021A1 (en) * | 2018-09-17 | 2020-03-18 | Robert Bosch GmbH | Device and method for training an augmented discriminator |
US11232575B2 (en) | 2019-04-18 | 2022-01-25 | Standard Cognition, Corp | Systems and methods for deep learning-based subject persistence |
US11176699B2 (en) * | 2019-05-24 | 2021-11-16 | Tencent America LLC | Augmenting reliable training data with CycleGAN for hand pose estimation |
US10620713B1 (en) * | 2019-06-05 | 2020-04-14 | NEX Team Inc. | Methods and systems for touchless control with a mobile device |
US10706286B1 (en) * | 2019-09-05 | 2020-07-07 | Alphonso Inc. | 3D convolutional neural networks for television advertisement detection |
US20240087353A1 (en) * | 2019-10-31 | 2024-03-14 | Nec Corporation | Image processing apparatus, image processing method, and non-transitory computer readable medium storing image processing program |
SE1951443A1 (en) * | 2019-12-12 | 2021-06-13 | Assa Abloy Ab | Improving machine learning for monitoring a person |
CN111208783B (en) * | 2019-12-30 | 2021-09-17 | 深圳市优必选科技股份有限公司 | Action simulation method, device, terminal and computer storage medium |
US11436267B2 (en) * | 2020-01-08 | 2022-09-06 | International Business Machines Corporation | Contextually sensitive document summarization based on long short-term memory networks |
US11443141B2 (en) * | 2020-02-27 | 2022-09-13 | International Business Machines Corporation | Using video tracking technology to create machine learning datasets for tasks |
JP7413836B2 (en) * | 2020-02-28 | 2024-01-16 | 富士通株式会社 | Behavior recognition method, behavior recognition program, and behavior recognition device |
CN111353447B (en) * | 2020-03-05 | 2023-07-04 | 辽宁石油化工大学 | Human skeleton behavior recognition method based on graph convolution network |
US20210312236A1 (en) * | 2020-03-30 | 2021-10-07 | Cherry Labs, Inc. | System and method for efficient machine learning model training |
US11361468B2 (en) | 2020-06-26 | 2022-06-14 | Standard Cognition, Corp. | Systems and methods for automated recalibration of sensors for autonomous checkout |
US11303853B2 (en) | 2020-06-26 | 2022-04-12 | Standard Cognition, Corp. | Systems and methods for automated design of camera placement and cameras arrangements for autonomous checkout |
EP4238061A4 (en) * | 2020-10-29 | 2024-02-21 | Hinge Health Inc | Three-dimensional mesh generator based on two-dimensional image |
US20220230079A1 (en) * | 2021-01-21 | 2022-07-21 | Microsoft Technology Licensing, Llc | Action recognition |
CN112998652B (en) * | 2021-02-23 | 2022-07-19 | 华南理工大学 | Photoelectric volume pulse wave pressure identification method and system |
CA3225826A1 (en) * | 2021-07-27 | 2023-02-02 | Caroline ROUGIER | Two-dimensional pose estimations |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130278501A1 (en) * | 2012-04-18 | 2013-10-24 | Arb Labs Inc. | Systems and methods of identifying a gesture using gesture data compressed by principal joint variable analysis |
US20140119640A1 (en) * | 2012-10-31 | 2014-05-01 | Microsoft Corporation | Scenario-specific body-part tracking |
US20160042227A1 (en) * | 2014-08-06 | 2016-02-11 | BAE Systems Information and Electronic Systems Integraton Inc. | System and method for determining view invariant spatial-temporal descriptors for motion detection and analysis |
US20170177930A1 (en) * | 2013-11-21 | 2017-06-22 | Mo' Motion Ventures | Jump Shot and Athletic Activity Analysis System |
CN107368182A (en) * | 2016-08-19 | 2017-11-21 | 北京市商汤科技开发有限公司 | Gestures detection network training, gestures detection, gestural control method and device |
US20170344829A1 (en) * | 2016-05-31 | 2017-11-30 | Microsoft Technology Licensing, Llc | Skeleton -based action detection using recurrent neural network |
US20180315200A1 (en) * | 2017-04-28 | 2018-11-01 | Cherry Labs, Inc. | Monitoring system |
US20190294871A1 (en) * | 2018-03-23 | 2019-09-26 | Microsoft Technology Licensing, Llc | Human action data set generation in a machine learning system |
US20200160046A1 (en) * | 2017-06-30 | 2020-05-21 | The Johns Hopkins University | Systems and method for action recognition using micro-doppler signatures and recurrent neural networks |
-
2018
- 2018-02-15 CA CA2995242A patent/CA2995242A1/en active Pending
-
2019
- 2019-02-14 US US16/276,493 patent/US10949658B2/en active Active
-
2021
- 2021-02-11 US US17/173,978 patent/US11783183B2/en active Active
-
2023
- 2023-09-28 US US18/477,482 patent/US20240028896A1/en active Pending
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130278501A1 (en) * | 2012-04-18 | 2013-10-24 | Arb Labs Inc. | Systems and methods of identifying a gesture using gesture data compressed by principal joint variable analysis |
US20140119640A1 (en) * | 2012-10-31 | 2014-05-01 | Microsoft Corporation | Scenario-specific body-part tracking |
US20170177930A1 (en) * | 2013-11-21 | 2017-06-22 | Mo' Motion Ventures | Jump Shot and Athletic Activity Analysis System |
US20160042227A1 (en) * | 2014-08-06 | 2016-02-11 | BAE Systems Information and Electronic Systems Integraton Inc. | System and method for determining view invariant spatial-temporal descriptors for motion detection and analysis |
US20170344829A1 (en) * | 2016-05-31 | 2017-11-30 | Microsoft Technology Licensing, Llc | Skeleton -based action detection using recurrent neural network |
CN107368182A (en) * | 2016-08-19 | 2017-11-21 | 北京市商汤科技开发有限公司 | Gestures detection network training, gestures detection, gestural control method and device |
US20180315200A1 (en) * | 2017-04-28 | 2018-11-01 | Cherry Labs, Inc. | Monitoring system |
US20200160046A1 (en) * | 2017-06-30 | 2020-05-21 | The Johns Hopkins University | Systems and method for action recognition using micro-doppler signatures and recurrent neural networks |
US20190294871A1 (en) * | 2018-03-23 | 2019-09-26 | Microsoft Technology Licensing, Llc | Human action data set generation in a machine learning system |
Non-Patent Citations (1)
Title |
---|
Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields (IEEE 2017) (Year: 2017) * |
Also Published As
Publication number | Publication date |
---|---|
US11783183B2 (en) | 2023-10-10 |
CA2995242A1 (en) | 2019-08-15 |
US20240028896A1 (en) | 2024-01-25 |
US20210161266A1 (en) | 2021-06-03 |
US20190251340A1 (en) | 2019-08-15 |
US10949658B2 (en) | 2021-03-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11783183B2 (en) | Method and system for activity classification | |
Lin et al. | Human hand gesture recognition using a convolution neural network | |
Mici et al. | A self-organizing neural network architecture for learning human-object interactions | |
Boualia et al. | Pose-based human activity recognition: a review | |
Patel et al. | Hand gesture recognition system using convolutional neural networks | |
Sarma et al. | Hand gesture recognition using deep network through trajectory-to-contour based images | |
Wersing et al. | Online learning of objects in a biologically motivated visual architecture | |
Tur et al. | Isolated sign recognition with a siamese neural network of RGB and depth streams | |
Damdoo et al. | Adaptive hand gesture recognition system using machine learning approach | |
Lalitha et al. | A deep learning model for face expression detection | |
Parisi et al. | HandSOM-Neural clustering of hand motion for gesture recognition in real time | |
Howell et al. | Active vision techniques for visually mediated interaction | |
Sun et al. | Automated work efficiency analysis for smart manufacturing using human pose tracking and temporal action localization | |
Moutsis et al. | Fall detection paradigm for embedded devices based on YOLOv8 | |
Ramanathan et al. | Combining pose-invariant kinematic features and object context features for rgb-d action recognition | |
Tazhigaliyeva et al. | Cyrillic manual alphabet recognition in RGB and RGB-D data for sign language interpreting robotic system (SLIRS) | |
Cardenas et al. | Multimodal human action recognition based on a fusion of dynamic images using cnn descriptors | |
Vrigkas et al. | Inferring human activities using robust privileged probabilistic learning | |
Pohlt et al. | Weakly-Supervised Learning for Multimodal Human Activity Recognition in Human-Robot Collaboration Scenarios | |
Basly et al. | Lahar-CNN: human activity recognition from one image using convolutional neural network learning approach | |
Harini et al. | A novel static and dynamic hand gesture recognition using self organizing map with deep convolutional neural network | |
Ferreira et al. | Dimensional emotion recognition using visual and textual cues | |
Butalia et al. | Emotional recognition and towards context based decision | |
Gurav et al. | Vision based hand gesture recognition with haar classifier and AdaBoost algorithm | |
Wilson et al. | Neural networks for sign language translation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: WRNCH INC., CANADA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BROWN, COLIN J.;TOLSTIKHIN, ANDREY;PETERS, THOMAS D.;AND OTHERS;REEL/FRAME:055282/0023 Effective date: 20180320 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: HINGE HEALTH, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WRNCH INC.;REEL/FRAME:058364/0007 Effective date: 20210915 |
|
FEPP | Fee payment procedure |
Free format text: PETITION RELATED TO MAINTENANCE FEES GRANTED (ORIGINAL EVENT CODE: PTGR); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |