CN110796110B - Human behavior identification method and system based on graph convolution network - Google Patents

Human behavior identification method and system based on graph convolution network Download PDF

Info

Publication number
CN110796110B
CN110796110B CN201911070446.0A CN201911070446A CN110796110B CN 110796110 B CN110796110 B CN 110796110B CN 201911070446 A CN201911070446 A CN 201911070446A CN 110796110 B CN110796110 B CN 110796110B
Authority
CN
China
Prior art keywords
graph
node
time
topological
sequence
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911070446.0A
Other languages
Chinese (zh)
Other versions
CN110796110A (en
Inventor
朱光明
张亮
杨露
李洪升
沈沛意
宋娟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xidian University
Original Assignee
Xidian University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xidian University filed Critical Xidian University
Priority to CN201911070446.0A priority Critical patent/CN110796110B/en
Publication of CN110796110A publication Critical patent/CN110796110A/en
Application granted granted Critical
Publication of CN110796110B publication Critical patent/CN110796110B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a human behavior identification method and a human behavior identification system based on a graph convolution network, wherein the identification method comprises the following steps: extracting human body skeleton information from an image containing human body behaviors, acquiring a human body joint point position information sequence, and constructing a topological graph sequence with any length of a human body skeleton; performing feature extraction and adaptive evolution of a topological structure on a topological graph sequence through a space-time graph convolution network based on topological learnable graph convolution to obtain new features of nodes fusing local space-time features and a topological graph sequence with a new topological structure; extracting features through a graph volume long-term and short-term memory neural network; obtaining global space-time characteristics by using global pooling operation; and carrying out human behavior recognition based on the global space-time characteristics through a classifier. The method directly learns the characteristics of the whole graph, expands the weight matrix in graph convolution to the structure of the whole topological graph, learns the relation between any two nodes in the graph without the limitation of the topological structure, and has high identification accuracy.

Description

Human behavior identification method and system based on graph convolution network
Technical Field
The invention belongs to the technical field of artificial intelligence, and relates to a human behavior identification method and a human behavior identification system based on a graph convolution network, which can be used for action identification of a topological graph sequence.
Background
Convolutional neural networks have achieved tremendous success in many areas, but rely on data characterization with a grid structure. However, data in many fields is not in a grid structure, and data in irregular domains usually shows a topological graph structure, so that the convolutional neural network is difficult to popularize in the graph domain. In order to maintain the characteristic that graph convolution products keep the characteristic, usually a transition matrix is defined on each node and a weight matrix is defined for the degree of the node so that graph convolution can learn on different topological subgraphs, and a corresponding space division rule and a rule used for determining are designed according to the number of domain subsets of the nodes of the space-time graph. The existing self-adaptive graph convolution only can learn the topological self-adaptive relationship between adjacent nodes, and the learning capability of the relationship between nodes with longer distance is insufficient. Furthermore, due to the limitations of the transition matrix in graph convolution, there is often a lack of effective modeling of long-term temporal relationships between sequences of topological graphs.
The topology of the topological graph data is usually fixed at all layers of the network, but the natural topology is not necessarily optimal, so the graph convolution network with the ability to learn arbitrary topologies has great significance to the convolutional neural network in the field of topological structure data.
Disclosure of Invention
In order to solve the problems, the invention provides a human behavior identification method based on a graph convolution network, which directly learns the characteristics of the whole graph, expands the weight matrix in the graph convolution to the structure of the whole topological graph, learns the relationship between any two nodes in the graph without the limitation of the topological structure, and has high identification accuracy; meanwhile, a recurrent neural network is introduced to model the long-term time relation of the topological graph sequence, so that the problems in the prior art are solved.
The invention also aims to provide a human behavior recognition method and system based on the graph convolution network.
The invention adopts the technical scheme that a human behavior identification method based on a graph convolution network comprises the following steps:
s1, extracting human skeleton information from the image containing human behavior, obtaining a human joint point position information sequence, and constructing a topological graph sequence with any length of the human skeleton by taking each joint point as a node and the skeleton between the joint points as an edge;
s2, performing feature extraction and adaptive evolution of a topological structure on the topological graph sequence through a space-time graph convolution network based on topological learnable graph convolution to obtain new node features fusing local space-time features and a topological graph sequence with a new topological structure;
s3, extracting the characteristics of the new topological graph sequence through the graph convolution long-short term memory neural network to obtain a topological graph sequence with long-term spatiotemporal characteristics;
s4, further fusing the characteristics of the topological graph sequence by using global pooling operation to obtain global space-time characteristics;
and S5, recognizing human body behaviors by using a classifier based on the global space-time characteristics.
Further, in the step S1, the topological graph sequence of the human skeleton is composed of a plurality of topological graph structures, and the topological graph structures are represented by formula (1-1);
G＝(V,E)＝(f v ,w E ) (1-1)
wherein G is a topological graph structure of a human skeleton, and a node set V ═ V ti 1, …, T, i, 1, …, N } represents human body joints, T is the frame number of the sequence, N is the number of the joints, and the node set V includes all the nodes in the skeleton sequence at each moment; the edge set E consists of two edge sets of a space domain and a time domain, and the edge set E in the space domain S ＝{v ti v tj I (i, j) belongs to H, and represents the edge of the t frame node i and the node j, wherein H is a set of natural connection of human joints; edge set E in time domain T ＝{v ti v (t+1)i Represents the connection between the same node and the previous and next frames; f. of v Feature vector, w, representing a node E Representing the connection weight of the edge.
Further, in step S2, specifically, the step includes:
s21, the space-time graph convolution network based on the topology learnable graph convolution is provided with a plurality of graph convolution blocks, and the space-domain feature and the time-domain feature are learnt for each graph convolution block respectively to obtain a node feature vector fusing local space-time features;
spatial domain feature learning: learning the spatial domain features by using the node feature learning function to obtain a node feature vector fusing the local spatial domain features, wherein the formula (1-2) is as follows:
Figure BDA0002260777730000021
wherein W is a node characteristic learning parameter matrix,
Figure BDA0002260777730000022
is node v i Of the feature vector of (1), node v i Is the ith node in the topological graph, W m Representing the m-th dimension of the matrix W,
Figure BDA0002260777730000023
representing a node v i Corresponding characteristic vectors, namely contents stored in a data structure corresponding to the nodes, wherein M represents the corresponding dimension of the vector or the matrix; normalizing the learned airspace characteristics by using a batch standardization function, and finally activating function processing characteristics by using linear rectification;
learning time domain characteristics: learning time domain features by using a time domain convolution function, and then normalizing the learned time domain features by using a batch standardization function;
s22, after the airspace feature learning, fusing the airspace feature vectors through a node fusion function GFusion (-) to obtain the connection weight of a new edge set; GFusion (. cndot.) is implemented using a matrix multiplication between topology learnable fusion weights and node features with a specific initialization, as shown in equations (1-3):
Figure BDA0002260777730000031
wherein L represents a topology learnable fusion parameter matrix,
Figure BDA0002260777730000032
is a node v i Feature vector of, L ij Is node v i And v j A fusion weight which can be learned in between, the fusion weight is initialized by normalizing the adjacent matrix or the full 0 matrix, v j Is a dividing node v in the topological graph i All nodes outside, "" indicate the product of the elements of two matrices,
Figure BDA0002260777730000033
representing a node v j The topology learnable fusion parameter matrix L is self-adaptive and is realized by utilizing two-dimensional convolution or matrix multiplication with the convolution kernel size of 1 multiplied by 1;
and S23, substituting the node feature vectors fused with the local space-time features and the connection weights of the new edge sets into formula (1-1) to obtain a topological graph sequence with a new topological structure.
Further, the topological graph sequence with long-term spatio-temporal characteristics in step S3 is determined according to equation (1-4):
F vt ＝GCNLSTM(STGCN(I)) (1-4)
wherein, F vt For the long-time space-time characteristics of a node v in a t-th frame, I is a human skeleton topological graph sequence shown in a formula (1-1), STGCN is a space-time graph convolution network based on topological learnable graph convolution, GCNLSTM is a graph convolution long-term and short-term memory network, and the specific implementation mode is shown in a formula (1-5):
Figure BDA0002260777730000034
wherein, W xi And W hi Is the weight of the input and hidden states in the input gate, W xf And W hf Is the weight of the input and hidden states in the forgetting gate, W xo And W ho Is the weight of the input and hidden states in the output gate, W xc Is the weight of the input in the cell state, W hc Is the weight of hidden states in the cell state, "+" in the cell state g "denotes a graph convolution operation, X t Is input at the current time, H t Is the hidden state at the present moment, H t-1 Is a hidden state at the previous moment, b i ，b f ，b o And b c Respectively, the deviations of the input gate, the forgetting gate, the output gate and the cell state, sigma is an S-shaped Sigmoid function,
Figure BDA0002260777730000035
i t ，f t and o t Gate function values of input gate, forgetting gate and output gate, respectively, C t-1 The state of the cell at the previous moment,
Figure BDA0002260777730000037
expressing a Hadamard product, wherein tanh is a hyperbolic tangent function; c t The cell state at the current time t.
Further, the step S4 is specifically performed according to the following steps:
s41, firstly, performing mean pooling operation on all node characteristics at each moment to obtain a characteristic vector at each moment, as shown in the formula (1-6):
Figure BDA0002260777730000036
wherein, F vt Long term spatiotemporal features, F t For the feature vector after the fusion at the time t, GPooling () is a node feature mean pooling function, which represents that mean pooling operation is performed on all nodes of the feature map at each time to obtain the feature vector at each time;
s42, aggregating the feature vectors at each moment by using a time domain mean global pooling operation to obtain global space-time features, as shown in the formula (1-7):
Figure BDA0002260777730000041
wherein, F t And F is the global space-time feature obtained by fusion, and TPoving () is a time domain mean global pooling function, and pooling the feature vectors at all the moments to obtain the global space-time feature.
Further, in the step S5, the concrete formula is shown in (1-8);
Figure BDA0002260777730000042
wherein C is the number of behavior classes, C k Is the k-th behavior class, S k And S i The probability that the global space-time feature F belongs to the k-th behavior class and the i-th behavior class is obtained through the known full-connected layer function calculation, and e is a constant.
Further, the topology learnable fusion parameter matrix L and the node characteristic learnt parameter matrix W are learnt and optimized through back propagation.
Further, the determination of the spatial edge set of the topological graph with the new topological structure comprises: node v when t frame ti And node v tj When the fusion weight therebetween is not 0, it indicates that the node v ti And node v tj Have a spatial relationship between them, form a new edge.
A behavior recognition system based on a spatio-temporal graph convolution and a graph convolution long-term and short-term memory network adopts the human behavior recognition method based on the graph convolution network, and comprises the following steps:
the topological graph sequence construction module is used for extracting human skeleton information from an input image, acquiring a human body joint point position information sequence, and constructing a topological graph sequence of a human body skeleton by taking each joint point as a node and bones between the joint points as edges;
the space-time graph convolution network is used for carrying out feature extraction and adaptive evolution of a topological structure on the topological graph sequence to obtain new node features fusing local space-time features and a topological graph sequence with a new topological structure;
the graph convolution long-short term memory neural network is used for extracting the characteristics of the topological graph sequence of the new topological structure to obtain the topological graph sequence with long-time space characteristics;
the global pooling module is used for further fusing the characteristics of the topological graph sequence to obtain global space-time characteristics;
and the classifier is used for carrying out human behavior identification based on the global space-time characteristics.
The invention has the beneficial effects that:
(1) the graph convolution is separated into two operations of feature learning and node fusion, a new topological graph except a manually set topological structure is learned by expanding the range of node fusion, the self-adaptive relation of the whole topological graph is learned by a specifically initialized topology learnable fusion parameter matrix, and a weight matrix in the graph convolution is expanded to the whole topological graph structure, so that the relation between interconnected nodes can be learned, the relation between two unconnected nodes can be learned, and the graph convolution is flexible to use and good in adaptability; the characteristics of the whole topological graph structure are directly learned, so that a human body skeleton sequence is converted into deep space-time characteristics, the integrity of the characteristics of the topological structure is kept in the whole learning process, the characteristics of the whole topological graph are more effectively extracted, the identification accuracy is improved, and the problem that the learning capability of the topological graph except for the manually set topological structure is insufficient due to the existing self-adaptive graph convolution is solved.
(2) The method is combined with a cyclic neural network to learn the long-term time-space characteristics of the topological graph sequence, is used for human behavior recognition based on the skeleton sequence data, effectively learns the characteristics of topological structure data, and solves the problem that the long-term time characteristic modeling capability of the conventional adaptive graph convolution on the topological graph sequence is insufficient.
(3) The invention can use the latest classifier to improve the performance, has good flexibility and expandability, effectively solves the problem of inconsistent duration time between different actions by converting the action sequence into the global space-time characteristic, effectively models the relation between body parts lacking physical connection, realizes the study of a dynamic topological structure sequence, and can be applied to the applications of human behavior recognition, gesture recognition, facial expression recognition and the like based on a skeleton sequence.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a behavior recognition method based on spatiotemporal graph convolution and graph convolution long-short term memory network according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of obtaining a new topological graph sequence by a graph convolution method capable of topology learning according to an embodiment of the present invention.
FIG. 3 is a block diagram of a behavior recognition system based on spatiotemporal graph convolution and graph convolution long and short term memory networks according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a human behavior recognition method based on a spatio-temporal graph convolution and graph convolution long-short term memory network, which comprises the following steps as shown in figure 1:
s1, extracting human skeleton information from the image containing human behavior, obtaining a human joint point position information sequence, constructing a topological graph sequence of the human skeleton by taking each joint point as a node and bones among the joint points as edges, wherein the skeleton information is usually extracted from a color image or a depth image;
the topological graph sequence of the human skeleton consists of a plurality of topological graph structures, and the topological graph structures are represented by formula (1);
G＝(V,E)＝(f v ,w E ) (1)
wherein G is a topological graph structure of a human skeleton, and a node set V ═ V ti L T1, …, T, i 1, …, N, where T is the number of frames in the sequence, N is the number of joints, and the node set V includes all nodes in the skeleton sequence at each time; the edge set E consists of two edge sets of a space domain and a time domain, and the edge set E in the space domain S ＝{v ti v tj I (i, j) belongs to H, and represents the edge of the t frame node i and the node j, wherein H is a set of natural connection of human joints; edge set E in time domain T ＝{v ti v (t+1)i Represents the connection between the same node and the previous and next frames; f. of v Feature vectors, w, representing nodes E Representing the connection weight of the edge.
S2, performing feature extraction and adaptive evolution of a topological structure on the topological graph sequence through a space-time graph convolution network based on topological learnable Graph Convolution (GCN), and obtaining new node features fusing local space-time features and a topological graph sequence with a new topological structure;
the convolution of the topology learnable graph can be completed by two steps of node feature learning and node feature fusion, and the convolution function of the topology learnable graph is obtained by a node feature learning function and a node fusion function, as shown in formula (2):
Figure BDA0002260777730000061
GraphConv (·) is a convolution function of a topology learnable graph, GFusion (·) is a node fusion function, L represents a topology learnable fusion parameter matrix, and FConv (·) is a node feature learning function; the output result of the topological learnable graph convolution function is a node new feature fusing local space-time features and a topological graph sequence with a new topological structure.
S21, the space-time graph convolution network based on topology learnable graph convolution has a plurality of graph convolution blocks, each of which isLearning the space domain characteristics and the time domain characteristics of each graph volume block to obtain a node characteristic vector f fusing local space-time characteristics v ′ ；
Spatial domain characteristics: learning the spatial domain features by using a node feature learning function to obtain a node feature vector fused with local spatial domain features, wherein the formula (3) is as follows:
Figure BDA0002260777730000062
wherein W is a node characteristic learning parameter matrix,
Figure BDA0002260777730000063
is node v i Of the feature vector of (1), node v i Is the ith node in the topological graph, W m Representing the m-th dimension of the matrix W,
Figure BDA0002260777730000064
representing a node v i The corresponding feature vector, i.e. the content stored in the data structure corresponding to the node, M represents the dimension corresponding to the vector or matrix. Normalizing the learned airspace characteristics by using a Batch Normalization function (BN), accelerating convergence, relieving overfitting, enabling the network to be insensitive to the initialization weight, and allowing a larger learning rate to be used; and finally, processing the characteristics by using a Rectified Linear Unit (ReLU), so that the calculated amount is saved, the gradient disappearance is avoided, and the overfitting is relieved.
Time domain characteristics: learning time domain features by using a time domain convolution function, and then normalizing the learned time domain features by using a batch standardization function; the convolution function operating on the spatial domain feature is a graph convolution function, because the spatial domain feature is a topological graph structure; the convolution operation operating on time domain features is a common convolution function because time domain features are grid structure data, non-topological structures.
S22, after the airspace feature learning, fusing the airspace features through a node fusion function GFuse (-) to obtain the connection weight of a new edge set
Figure BDA0002260777730000071
GFusion (·) is implemented using a matrix multiplication between topology learnable fusion weights and node features with a specific initialization, as shown in equation (4):
Figure BDA0002260777730000072
wherein L represents a topology learnable fusion parameter matrix,
Figure BDA0002260777730000073
is a node v i Feature vector of, L ij Is node v i And v j A fusion weight which can be learned in between, the fusion weight is initialized by normalizing the adjacent matrix or the full 0 matrix, v j Is a dividing node v in the topological graph i All nodes outside (including but not limited to node v) i All the nodes of the adjacent nodes) of the matrix can learn not only the relationship between the nodes connected with the edge but also the relationship between any two nodes, "" indicates the product of elements of two matrices,
Figure BDA0002260777730000074
representing a node v j The topology learnable fusion parameter matrix L is self-adaptive and is realized by utilizing two-dimensional convolution or matrix multiplication with the convolution kernel size of 1 multiplied by 1, so that not only the fusion weight of the existing edge can be learnt, but also the fusion weight between any two nodes can be learnt; for example, in a human skeleton, there is no natural connection between joints of the left and right hands, but there is often a correlation between the two when performing an action, and the implicit relationship can be learned by a topologically learnable graph convolution; the fusion parameter matrix L and the node characteristic learning parameter matrix W can be learned through back propagation learning topology, and parameters are optimized.
In the embodiment, the space-time graph convolution network based on topology learnable graph convolution has 7 graph volume blocks, the number of channels of each graph volume block is 64, 128, 256 and 256, respectively, and the number of the graph volume blocks and the corresponding number of the channels have no specific requirements, and belong to the setting of hyper-parameters in a neural network.
S23, fusing the node feature vector f of the local space-time feature v ' new edge set connection weight
Figure BDA0002260777730000075
Substituting formula (1) to obtain a topological graph sequence with a new topological structure, as shown in formula (5):
Figure BDA0002260777730000076
wherein G is GCN Is a new topological graph sequence, V represents a node set, E' S Is a set of edges, f 'of a topology graph having a new topology' v To fuse the nodal feature vectors of the local spatiotemporal features,
Figure BDA0002260777730000081
the connection weight of the new edge set.
Edge set E 'of topology graph with new topology in air domain' S ＝{v ti v tj |L i,j Not equal to 0}, i.e. the node v of the t-th frame ti And node v tj When the fusion weight between the nodes is not 0, it represents the node v ti And node v tj And a spatial relationship is formed between the two, a new edge is formed, and the topological structure of the topological graph is updated. The time domain convolution operation does not change the topology of the graph, but only updates the characteristics of each graph node. Only the topology learnable graph convolution proposed by the invention can simultaneously update the graph node characteristics and the graph topology structure, and the topology learnable graph convolution is only applied to the spatial domain graph convolution, and the time domain convolution still adopts the known time domain convolution method.
In fig. 2, first, a topological graph sequence of T-th frame is input, wherein the topological graph of T-th frame is shown in formula (1), the topological graph has N nodes, each node has a feature vector f v ，C in Representing the number of input channels of the space-time graph convolutional neural network; learning parameters through node features in a networkThe number matrix W and the learnable fusion weight parameter matrix L are used for learning the characteristics of the input topological graph sequence, wherein,
Figure BDA0002260777730000082
is the link weight value, L, of the edge between the ith joint and the jth joint in the tth frame ij Represents the ith row and the jth column of the matrix L; obtaining a new topological graph sequence through a time-space graph convolutional neural network, wherein each node has a new feature vector f' v ，C out Representing the number of channels output by the network.
The convolution of the existing topological learnable graph only learns the relation between the connected nodes, extracts the characteristics of the subgraph in a bottom-up mode and then fuses the characteristics of the whole graph. The graph convolution is separated into two operations of feature learning and node fusion, a new topological graph except for a topological structure set manually is learned by expanding the range of node fusion, and the adaptive relation of the whole topological graph is learned by a specific initialized topological learnable fusion parameter matrix L; expanding the weight matrix in the graph convolution to the whole topological graph structure, learning the relationship between any two nodes in the graph without the limitation of the topological structure, expanding the learning range from the connected nodes to any two nodes; the method does not adopt a bottom-up mode, but can directly learn the characteristics of the whole graph by learning the relation of any two nodes, the parameter quantity is larger than that of learning the subgraph, but the extracted characteristics are more effective.
The method replaces the feature of the learning sub-topological graph with the feature of the learning whole topological graph for re-fusion, and uses simple topology learnable convolution with specific initialization and shows better performance in order to have equivalent parameters and calculated amount with other self-adaptive graph convolution neural networks.
S3, extracting the characteristics of the new topological graph sequence through a graph convolution long-term short-term memory neural network (GCNLSTM) to obtain a topological graph sequence with long-term space-time characteristics, as shown in formula (6):
F vt ＝GCNLSTM(STGCN(I)) (6)
wherein I is represented by formula (1)A human body skeleton topological graph sequence, wherein STGCN is a time-space graph convolution network based on topological learnable graph convolution, GCNLSTM is a graph convolution long-term and short-term memory network, and the specific implementation mode is shown in formula (7); f vt The long-term space-time characteristics of the node v in the t frame, namely a new characteristic vector of the node.
The graph volume long short term memory neural network is shown as a formula (7);
Figure BDA0002260777730000091
wherein, W xi And W hi Is the weight of the input and hidden states in the input gate, W xf And W hf Is the weight of the input and hidden states in the forgetting gate, W xo And W ho Is the weight of the input and hidden states in the output gate, W xc Is the weight of the input in the cell state, W hc Is the weight of the hidden state in the cell state, "+ g "denotes a graph convolution operation, X t Is the current time input, H t Is a hidden state at the present time, H t-1 Is a hidden state at the previous moment, b i ，b f ，b o And b c Respectively, the deviations of the input gate, the forgetting gate, the output gate and the cell state, sigma is an S-shaped Sigmoid function,
Figure BDA0002260777730000092
)；i t ，f t and o t Gate function values of input gate, forgetting gate and output gate, respectively, C t-1 The state of the cell at the previous moment,
Figure BDA0002260777730000095
representing a Hadamard product, and tanh is a hyperbolic tangent function; c t The cell state at the current time t.
The convolutional network of the space-time diagram can only learn short-term high-level space-time characteristics, and for a data sequence with a time relation, the short-term high-level characteristics are insufficient for pattern recognition. The cyclic neural network carries out long-term time modeling on the learned short-term high-level space-time characteristics, fully learns the time relation on the sequence, and obviously improves the effect of mode identification.
S4, further fusing the features of the topological graph sequence by using global pooling operation to obtain global space-time features, effectively solving the problem of inconsistent duration time between different actions, and effectively modeling the relationship between body parts lacking physical connection;
s41, first performing mean pooling on all node features at each time to obtain a feature vector at each time, as shown in equation (8):
Figure BDA0002260777730000093
wherein, F vt For long-term spatiotemporal characteristics, F t For the feature vector after the fusion at the time t, GPooling () is a node feature mean pooling function, which represents that mean pooling operation is performed on all nodes of the feature map at each time to obtain the feature vector at each time.
S42, aggregating the feature vectors at each moment by using a time domain mean global pooling operation to obtain global space-time features, as shown in formula (9):
Figure BDA0002260777730000094
wherein, F t And F is the global space-time feature obtained by fusion, and TPooling () is a time domain mean global pooling function, and the feature vectors at all the moments are pooled to obtain the global space-time feature.
S5, recognizing human body behaviors by using a Softmax classifier based on global space-time characteristics, wherein the Softmax classifier has good flexibility and expandability and improves the performance;
specifically, as shown in formula (10);
Figure BDA0002260777730000101
where C is the number of behavior classes, C k Is the kth behavior category (common behavior categories are drinking, eating, brushing teeth, combing head, reading and the like actions), S k And S i The probabilities that the global space-time feature F belongs to the k-th behavior class and the i-th behavior class are obtained through the known full-connection layer function calculation, and e is a constant and has a value of about 2.718.
The embodiment of the invention discloses a behavior recognition system based on a spatiotemporal graph convolution and graph convolution long-short term memory network, which is shown in figure 3, and adopts the human behavior recognition method based on the graph convolution network, and comprises the following steps:
the topological graph sequence construction module is used for extracting human skeleton information from an input image, acquiring a human body joint point position information sequence, and constructing a topological graph sequence of a human body skeleton by taking each joint point as a node and bones between the joint points as edges;
the space-time graph convolution network is used for carrying out feature extraction and adaptive evolution of a topological structure on the topological graph sequence to obtain new node features fusing local space-time features and a topological graph sequence with a new topological structure;
the graph convolution long-short term memory neural network is used for extracting the characteristics of the topological graph sequence of the new topological structure to obtain the topological graph sequence with long-term space-time characteristics;
the global pooling module is used for further fusing the characteristics of the topological graph sequence to obtain global space-time characteristics;
and the classifier is used for carrying out human behavior identification based on the global space-time characteristics.
The present invention compares on both data sets with a space-time graph convolutional neural network (ST-GCN), one of the most advanced current graph-convolutional neural networks. On the Kinetics-Skeleton data set, the optimal recognition accuracy rate of the method reaches 36.2 percent, and is 5.5 percent higher than that of ST-GCN; on an NTU-RGBD data set, the optimal recognition accuracy of the method reaches 89.2 percent, which is 7.7 percent higher than that of ST-GCN. Compared with the existing human behavior recognition method based on the time-space diagram convolution, the method can directly learn the characteristics of the whole diagram, the learned characteristics can better represent human skeleton information, the relationship between two joints without physical connection can be learned, the human behavior recognition is facilitated, the long-term time characteristics can be learned by using the time relationship of the diagram convolution long-term and short-term memory network learning sequence, and the method has better superiority than the time relationship of the common convolution learning sequence.
The method is used for human behavior recognition, comprises action recognition, gesture recognition and facial expression recognition, can form a topological graph sequence by taking human joints as nodes of a topological graph based on human skeleton information, and adopts the method to recognize; for example, the method can be applied to purchasing behavior recognition in unmanned supermarket, better man-machine interaction can be realized by recognizing the behaviors of people by the intelligent robot in home life, the behaviors of specific people in specific places can be recognized in the field of security monitoring, and the like; the method can also be applied to data analysis and other applications with a relational model data structure.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims (5)

1. A human behavior recognition method based on graph convolution network is characterized by comprising the following steps:
s1, extracting human skeleton information from the image containing human behavior, obtaining a human joint point position information sequence, and constructing a topological graph sequence with any length of the human skeleton by taking each joint point as a node and the skeleton between the joint points as an edge;
s2, performing feature extraction and adaptive evolution of a topological structure on the topological graph sequence through a space-time graph convolution network based on topological learnable graph convolution to obtain new node features fusing local space-time features and a topological graph sequence with a new topological structure;
s3, extracting the characteristics of the new topological graph sequence through the graph convolution long-term and short-term memory neural network to obtain a topological graph sequence with long-term space-time characteristics;
s4, further fusing the features of the topological graph sequence by using global pooling operation to obtain global space-time features;
s5, recognizing human body behaviors by using a classifier based on global space-time characteristics;
the step S2 specifically includes:
s21, the space-time graph convolution network based on topology learnable graph convolution is provided with a plurality of graph convolution blocks, and the space-domain feature and the time-domain feature are learnt for each graph convolution block respectively to obtain a node feature vector fusing local space-time features;
spatial domain feature learning: learning the spatial domain features by using a node feature learning function to obtain a node feature vector fusing local spatial domain features, wherein the formula is shown as (1-2):
Figure FDA0003687355350000011
wherein W is a node characteristic learning parameter matrix,
Figure FDA0003687355350000012
is node v i Of the feature vector of (1), node v i Is the ith node in the topological graph, W m Representing the m-th dimension of the matrix W,
Figure FDA0003687355350000013
representing a node v i The corresponding feature vector, namely the content stored in the data structure corresponding to the node, M represents the corresponding dimension of the vector or the matrix; normalizing the learned airspace features by using a batch normalization function, and finally processing the features by using a linear rectification activation function;
learning time domain features: learning time domain features by using a time domain convolution function, and then normalizing the learned time domain features by using a batch standardization function;
s22, after the airspace feature learning, fusing the airspace feature vectors through a node fusion function GFusion (-) to obtain the connection weight of a new edge set; GFusion (. cndot.) is implemented using a matrix multiplication between topology learnable fusion weights and node features with a specific initialization, as shown in equations (1-3):
Figure FDA0003687355350000014
wherein L represents a topology learnable fusion parameter matrix,
Figure FDA0003687355350000015
is a node v i Characteristic vector of (D), L ij Is node v i And v j A fusion weight which can be learned in between, the fusion weight is initialized by normalizing the adjacent matrix or the full 0 matrix, v j Is a dividing node v in the topological graph i All nodes outside, "", indicates the product of the elements of two matrices,
Figure FDA0003687355350000016
representing a node v j The topological learnable fusion parameter matrix L is self-adaptive and is realized by utilizing two-dimensional convolution or matrix multiplication with convolution kernel size of 1 multiplied by 1;
s23, substituting the node feature vectors fused with local space-time features and the connection weights of the new edge sets into formula (1-1) to obtain a topological graph sequence with a new topological structure;
determining the spatial edge set of the topological graph with the new topological structure: node v when t frame ti And node v tj When the fusion weight between the nodes is not 0, it represents the node v ti And node v tj Have a spatial relationship between them, form a new edge;
the step S4 is specifically performed according to the following steps:
s41, firstly, performing mean value pooling operation on all node characteristics at each moment to obtain a characteristic vector at each moment, wherein the formula is shown as (1-6):
Figure FDA0003687355350000021
wherein, F vt For long-term spatiotemporal characteristics, F t For the feature vector after fusion at the time t, GPoving () is a node feature mean pooling function, and means that mean pooling operation is performed on all nodes of the feature graph at each time to obtain the feature vector at each time;
s42, aggregating the feature vectors at each moment by using a time domain mean global pooling operation to obtain global space-time features, as shown in the formula (1-7):
Figure FDA0003687355350000022
wherein, F t The feature vectors at the time t are obtained, F is the global space-time feature obtained by fusion, TPooling () is a time domain mean value global pooling function, and the feature vectors at all the times are pooled to obtain the global space-time feature;
in the step S1, the topological graph sequence of the human skeleton is composed of a plurality of topological graph structures, and the topological graph structures are represented by the formula (1-1);
G＝(V，E)＝(f v ，w E ) (1-1)
wherein G is a topological graph structure of a human skeleton, and a node set V is { V ═ V } ti 1, ·, T, i ═ 1,. N, N } denotes human joints, T is the frame number of the sequence, N is the number of the joints, and the node set V includes all the nodes in the skeleton sequence at each time; the edge set E consists of two edge sets of a space domain and a time domain, and the edge set E in the space domain S ＝{v ti v tj I (i, j) belongs to H, and represents the edge of the t frame node i and the node j, wherein H is a set of natural connection of human joints; edge set E in time domain T ＝{v ti v (t+1)i Represents the connection between the same node and the previous and next frames; f. of v Feature vectors, w, representing nodes E Representing the connection weight of the edge.
2. The method for recognizing human body behaviors based on graph convolution network as claimed in claim 1, wherein the topological graph sequence with long-term spatiotemporal features in step S3 is determined according to equation (1-4):
F vt ＝GCNLSTM(STGCN(I)) (1-4)
wherein, F vt For the long-time space-time characteristics of a node v in a t-th frame, I is a human skeleton topological graph sequence shown in a formula (1-1), STGCN is a space-time graph convolution network based on topological learnable graph convolution, GCNLSTM is a graph convolution long-term and short-term memory network, and the specific implementation mode is shown in a formula (1-5):
Figure FDA0003687355350000031
wherein, W xi And W hi Is the weight of the input and hidden states in the input gate, W xf And W hf Is the weight of the input and hidden states in the forgetting gate, W xo And W ho Is the weight of the input and hidden states in the output gate, W xc Is the weight of the input in the cell state, W hc Is the weight of the hidden state in the cell state, "+ g "represents a graph convolution operation, X t Is the current time input, H t Is a hidden state at the present time, H t-1 Is a hidden state at the previous moment, b i ，b f ，b o And b c Respectively, the deviation of the input gate, the forgetting gate, the output gate and the cell state, sigma is a Sigmoid function,
Figure FDA0003687355350000032
i t ，f t and o t Gate function values, C, for input, forgetting and output gates, respectively t-1 The state of the cell at the previous moment,
Figure FDA0003687355350000033
representing a Hadamard product, and tanh is a hyperbolic tangent function; c t The cell state at the current time t.
3. The method for recognizing human body behaviors based on graph convolution network according to claim 1, wherein the step S5 is specifically represented by formula (1-8);
Figure FDA0003687355350000034
where C is the number of behavior classes, C k For the kth behavior class, S k And S i The probability that the global space-time feature F belongs to the k-th behavior class and the i-th behavior class is obtained through the known full-connected layer function calculation, and e is a constant.
4. The human behavior recognition method based on the graph convolution network as claimed in claim 1, wherein the topology learnable fusion parameter matrix L and the node feature learning parameter matrix W are both learned and optimized through back propagation.
5. A behavior recognition system based on spatio-temporal graph convolution and graph convolution long-short term memory network, characterized in that, a human behavior recognition method based on graph convolution network as claimed in any one of claims 1-4 is adopted, which includes:
the topological graph sequence construction module is used for extracting human skeleton information from an input image, acquiring a human joint point position information sequence, and constructing a topological graph sequence of a human skeleton by taking all joint points as nodes and bones among the joint points as edges;
the space-time graph convolution network is used for carrying out feature extraction and adaptive evolution of a topological structure on the topological graph sequence to obtain new node features fusing local space-time features and the topological graph sequence with the new topological structure;
the graph convolution long-short term memory neural network is used for extracting the characteristics of the topological graph sequence of the new topological structure to obtain the topological graph sequence with long-term space-time characteristics;
the global pooling module is used for further fusing the characteristics of the topological graph sequence to obtain global space-time characteristics;
and the classifier is used for carrying out human behavior identification based on the global space-time characteristics.
CN201911070446.0A 2019-11-05 2019-11-05 Human behavior identification method and system based on graph convolution network Active CN110796110B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911070446.0A CN110796110B (en) 2019-11-05 2019-11-05 Human behavior identification method and system based on graph convolution network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911070446.0A CN110796110B (en) 2019-11-05 2019-11-05 Human behavior identification method and system based on graph convolution network

Publications (2)

Publication Number Publication Date
CN110796110A CN110796110A (en) 2020-02-14
CN110796110B true CN110796110B (en) 2022-07-26

Family

ID=69442713

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911070446.0A Active CN110796110B (en) 2019-11-05 2019-11-05 Human behavior identification method and system based on graph convolution network

Country Status (1)

Country Link
CN (1) CN110796110B (en)

Families Citing this family (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111291693B (en) * 2020-02-17 2023-03-31 安徽工程大学 Deep integration method based on skeleton motion recognition
CN111310707B (en) * 2020-02-28 2023-06-20 山东大学 Bone-based graph annotation meaning network action recognition method and system
CN111709321B (en) * 2020-05-28 2022-08-16 西安交通大学 Human behavior recognition method based on graph convolution neural network
CN111812450B (en) * 2020-06-01 2022-03-18 复旦大学 Method for identifying dangerous faults of power grid
CN113761286B (en) * 2020-06-01 2024-01-02 杭州海康威视数字技术股份有限公司 Knowledge graph embedding method and device and electronic equipment
CN111737909B (en) * 2020-06-10 2021-02-09 哈尔滨工业大学 Structural health monitoring data anomaly identification method based on space-time graph convolutional network
CN111814842B (en) * 2020-06-17 2023-11-03 北京邮电大学 Object classification method and device based on multichannel graph convolution neural network
CN111783692A (en) * 2020-07-06 2020-10-16 广东工业大学 Action recognition method and device, electronic equipment and storage medium
CN111723779B (en) * 2020-07-20 2023-05-02 浙江大学 Chinese sign language recognition system based on deep learning
CN112069877B (en) * 2020-07-21 2022-05-03 北京大学 Face information identification method based on edge information and attention mechanism
CN111950485B (en) * 2020-08-18 2022-06-17 中科人工智能创新技术研究院(青岛)有限公司 Human body behavior identification method and system based on human body skeleton
CN112084891B (en) * 2020-08-21 2023-04-28 西安理工大学 Cross-domain human body action recognition method based on multi-modal characteristics and countermeasure learning
CN111814921B (en) * 2020-09-04 2020-12-18 支付宝(杭州)信息技术有限公司 Object characteristic information acquisition method, object classification method, information push method and device
CN112052816B (en) * 2020-09-15 2022-07-12 山东大学 Human behavior prediction method and system based on adaptive graph convolution countermeasure network
CN112149618B (en) * 2020-10-14 2022-09-09 紫清智行科技(北京)有限公司 Pedestrian abnormal behavior detection method and device suitable for inspection vehicle
CN112329562B (en) * 2020-10-23 2024-05-14 江苏大学 Human interactive action recognition method based on skeleton characteristics and slicing recurrent neural network
US20220138536A1 (en) * 2020-10-29 2022-05-05 Hong Kong Applied Science And Technology Research Institute Co., Ltd Actional-structural self-attention graph convolutional network for action recognition
CN112543936B (en) * 2020-10-29 2021-09-28 香港应用科技研究院有限公司 Motion structure self-attention-drawing convolution network model for motion recognition
CN112380955B (en) * 2020-11-10 2023-06-16 浙江大华技术股份有限公司 Action recognition method and device
CN112464808B (en) * 2020-11-26 2022-12-16 成都睿码科技有限责任公司 Rope skipping gesture and number identification method based on computer vision
CN112598021A (en) * 2020-11-27 2021-04-02 西北工业大学 Graph structure searching method based on automatic machine learning
CN112633153A (en) * 2020-12-22 2021-04-09 天津大学 Facial expression motion unit identification method based on space-time graph convolutional network
CN112686304B (en) * 2020-12-29 2023-03-24 山东大学 Target detection method and device based on attention mechanism and multi-scale feature fusion and storage medium
CN112800903B (en) * 2021-01-19 2022-08-26 南京邮电大学 Dynamic expression recognition method and system based on space-time diagram convolutional neural network
CN113610156A (en) * 2021-01-20 2021-11-05 廖彩红 Artificial intelligence model machine learning method and server for big data analysis
CN112836670A (en) * 2021-02-24 2021-05-25 复旦大学 Pedestrian action detection method and device based on adaptive graph network
CN112906604B (en) * 2021-03-03 2024-02-20 安徽省科亿信息科技有限公司 Behavior recognition method, device and system based on skeleton and RGB frame fusion
CN112818942B (en) * 2021-03-05 2022-11-18 清华大学 Pedestrian action recognition method and system in vehicle driving process
CN112801060A (en) * 2021-04-07 2021-05-14 浙大城市学院 Motion action recognition method and device, model, electronic equipment and storage medium
CN113408455B (en) * 2021-06-29 2022-11-29 山东大学 Action identification method, system and storage medium based on multi-stream information enhanced graph convolution network
CN113705542A (en) * 2021-10-27 2021-11-26 北京理工大学 Pedestrian behavior state identification method and system
CN114550308B (en) * 2022-04-22 2022-07-05 成都信息工程大学 Human skeleton action recognition method based on space-time diagram
CN115471908A (en) * 2022-08-16 2022-12-13 深圳先进技术研究院 Attitude processing method, apparatus, system and storage medium
CN115797841B (en) * 2022-12-12 2023-08-18 南京林业大学 Quadruped behavior recognition method based on self-adaptive space-time diagram attention transducer network
CN116434339B (en) * 2023-04-13 2023-10-27 江南大学 Behavior recognition method based on space-time characteristic difference and correlation of skeleton data

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109754605A (en) * 2019-02-27 2019-05-14 中南大学 A kind of traffic forecast method based on attention temporal diagram convolutional network
CN110059620A (en) * 2019-04-17 2019-07-26 安徽艾睿思智能科技有限公司 Bone Activity recognition method based on space-time attention
CN110222653A (en) * 2019-06-11 2019-09-10 中国矿业大学(北京) A kind of skeleton data Activity recognition method based on figure convolutional neural networks

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109754605A (en) * 2019-02-27 2019-05-14 中南大学 A kind of traffic forecast method based on attention temporal diagram convolutional network
CN110059620A (en) * 2019-04-17 2019-07-26 安徽艾睿思智能科技有限公司 Bone Activity recognition method based on space-time attention
CN110222653A (en) * 2019-06-11 2019-09-10 中国矿业大学(北京) A kind of skeleton data Activity recognition method based on figure convolutional neural networks

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition";Chenyang Si etal.;《arXiv》;20190329;第1-10页 *
"基于姿态和骨架信息的行为识别方法研究与实现";马静;《中国优秀硕士学位论文全文数据库 信息科技辑》;20190115;第1-5章 *

Also Published As

Publication number Publication date
CN110796110A (en) 2020-02-14

Similar Documents

Publication Publication Date Title
CN110796110B (en) Human behavior identification method and system based on graph convolution network
Salman et al. Weather forecasting using deep learning techniques
CN108846384A (en) Merge the multitask coordinated recognition methods and system of video-aware
Sethi Entropy nets: from decision trees to neural networks
Pronobis et al. Learning deep generative spatial models for mobile robots
CN112395945A (en) Graph volume behavior identification method and device based on skeletal joint points
CN109858390A (en) The Activity recognition method of human skeleton based on end-to-end space-time diagram learning neural network
Felzenszwalb et al. Object detection grammars.
CN109753897B (en) Behavior recognition method based on memory cell reinforcement-time sequence dynamic learning
CN110570035B (en) People flow prediction system for simultaneously modeling space-time dependency and daily flow dependency
CN110826698A (en) Method for embedding and representing crowd moving mode through context-dependent graph
CN109344992B (en) Modeling method for user control behavior habits of smart home integrating time-space factors
CN104537684A (en) Real-time moving object extraction method in static scene
CN111199216A (en) Motion prediction method and system for human skeleton
CN107341471B (en) A kind of Human bodys' response method based on Bilayer condition random field
CN113627326A (en) Behavior identification method based on wearable device and human skeleton
CN117373116A (en) Human body action detection method based on lightweight characteristic reservation of graph neural network
Quenet et al. The dynamic neural filter: a binary model of spatiotemporal coding
CN111325336B (en) Rule extraction method based on reinforcement learning and application
CN114254214A (en) Traffic prediction method and system based on space-time hierarchical network
Chander Deep learning network: deep neural networks
Géczy et al. Rule Extraction from Trained Artifical Neural Networks
Tsang et al. Convergence analysis of a discrete Hopfield neural network with delay and its application to knowledge refinement
Hwang et al. Adaptive reinforcement learning in box-pushing robots
Gamal et al. A fuzzy rough rule based system enhanced by fuzzy cellular automata

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant