CN104850846B - A kind of Human bodys' response method and identifying system based on deep neural network - Google Patents
A kind of Human bodys' response method and identifying system based on deep neural network Download PDFInfo
- Publication number
- CN104850846B CN104850846B CN201510293654.2A CN201510293654A CN104850846B CN 104850846 B CN104850846 B CN 104850846B CN 201510293654 A CN201510293654 A CN 201510293654A CN 104850846 B CN104850846 B CN 104850846B
- Authority
- CN
- China
- Prior art keywords
- neural network
- human
- joint point
- deep neural
- human body
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The Human bodys' response method based on deep neural network that the present invention provides a kind of, including：Obtain the original depth data stream of involved party；The skeleton joint point data of human body is extracted by the original depth data stream of involved party；Using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire Human Modeling；By carrying out feature extraction to entire Human Modeling, characteristic is sent into limitation Boltzmann machine network to pre-process, the weight initialization BP neural network parameter that will be obtained trains deep neural network model, and carries out Activity recognition to the result of feature extraction accordingly；Using multi-threading parallel process, the human skeleton joint point data extracted is overlapped with actual human body, and the behavior recognized is subjected to real-time display；It establishes abnormal behaviour template library and alarms the abnormal behaviour detected.The present invention can detect the variation of human body behavior in real time, be alarmed (as fallen) the abnormal behaviour of human body.
Description
Technical field
The present invention relates to video identification field more particularly to a kind of Human bodys' response methods based on deep neural network
And identifying system.
Background technology
It is well known that being exactly the epoch of human body astogeny now, China is the country that elderly population are most in the world, there is report
Announcement points out that arriving the year two thousand twenty elderly population will be to 2.48 hundred million, in addition the implementation of family planning, and only child is busy with work, many old
The most of the time of people is all independent life.Many abnormal behaviours cannot be found in time, so that old man's delay treatment loses
The tragedy of life happens occasionally, and seriously affects the quality of life of the elderly.For example, tumble behavior is China over-65s the elderly
The immediate cause of injury scope.If can accompany and attend to the daily life of Empty nest elderly, its behavior is effectively identified,
The abnormal behaviour in life is found in time, this will greatly improve the quality of life of old man, have to the development of family harmony, society
Great meaning.And Human bodys' response is the core accompanied and attended at home, the behavior of human body is very complicated, and the same behavior is to different people
For have different performances, carrying out effective identification to people's behavior becomes the task of top priority.
Currently, the prior art is all to use conventional accelerometers, gyroscope etc. to the mode of Human bodys' response, such as
Sensor etc. is put at some position of human body, and this mode identifies that the behavior type of human body is very limited, and accuracy of identification is not high, is known
Other speed is slow, and identification behavior is single, poor expandability.
Therefore, there is an urgent need for designing a kind of intelligence of Human bodys' response to accompany and attend to system, to improve type, precision and the speed of identification
Degree.
Invention content
In view of this, the embodiment of the present invention is designed to provide a kind of Human bodys' response based on deep neural network
Method and identifying system, it is intended to solve to identify that the type of human body behavior is limited, accuracy of identification is not high, recognition speed in the prior art
Slow problem.
The embodiment of the present invention is achieved in that a kind of Human bodys' response method based on deep neural network, including：
Obtain the original depth data stream of involved party；
The skeleton joint point data of human body is extracted by the original depth data stream of the involved party；
Using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire Human Modeling；
By carrying out feature extraction to entire Human Modeling, characteristic is sent into limitation Boltzmann machine network and is carried out
Pretreatment, the weight initialization BP neural network parameter that will be obtained train deep neural network model, trained depth are used in combination
It spends neural network model and Activity recognition is carried out to the result of feature extraction；
Using multi-threading parallel process, the human skeleton joint point data extracted is overlapped with actual human body, and
The behavior recognized is subjected to real-time display；
Abnormal behaviour template library is established, and is alarmed the abnormal behaviour detected.
Preferably, the step of original depth data stream for obtaining involved party includes：
The involved party in the visual field is monitored using kinect sensors, and obtains collected rgb video stream in real time,
And the original depth flow data of involved party is obtained using pumped FIR laser.
Preferably, the original depth data stream by the involved party extracts the step of the skeleton joint point data of human body
Suddenly include：
On the basis of the original depth data stream of the involved party, extracted by coordinate transform and smothing filtering mode
The skeleton joint point data of human body, and shown by visual processes technology.
Preferably, the human skeleton artis data include 20 skeleton joint point datas of human body.
Preferably, described by carrying out feature extraction to entire Human Modeling, characteristic is sent into limitation bohr hereby
Graceful machine network is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains deep neural network model, and
The step of carrying out Activity recognition to the result of feature extraction with trained deep neural network model include：
By extracting the scalar information characteristics and Vector Message feature of involved party's skeleton joint point to entire Human Modeling,
Wherein, the scalar information characteristics include the relative distance feature of two joint point, body joint point coordinate axis opposite variation characteristic with
And the angle feature constituted between joint and joint, the Vector Message feature include the velocity variations feature of artis, a pass
Node is relative to the changes in coordinates feature of another artis and the acceleration change feature of special artis；
The scalar information characteristics of the involved party's skeleton joint point extracted and Vector Message feature are integrally encoded
Input matrix is constituted, while human body behavior is encoded；
The coding carried out by the input matrix and to human body behavior is sent into limitation Boltzmann machine network and is carried out together
Data prediction；
Pretreated data addition back-propagation algorithm neural network is learnt；
Deep neural network system model is established by study, and utilizes the deep neural network system model into every trade
For identification.
On the other hand, the Human bodys' response system based on deep neural network that the present invention also provides a kind of, including：
Data acquisition module, the original depth data stream for obtaining involved party；
Node extraction module, the skeleton joint for extracting human body by the original depth data stream of the involved party are counted
According to；
Data modeling module, the three-dimensional coordinate corresponding to the point data of human skeleton joint extracted for utilization, next pair
Entire Human Modeling；
Activity recognition module, for by carrying out feature extraction to entire Human Modeling, characteristic being sent into and is limited
Boltzmann machine network is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains deep neural network
Model is used in combination trained deep neural network model to carry out Activity recognition to the result of feature extraction；
Data processing module, for using multi-threading parallel process, the human skeleton joint point data and reality that will be extracted
Border human body is overlapped, and the behavior recognized is carried out real-time display；
Template establishes module, alarms for establishing abnormal behaviour template library, and to the abnormal behaviour detected.
Preferably, the data acquisition module is specifically used for carrying out the involved party in the visual field using kinect sensors
Monitoring, and collected rgb video stream in real time is obtained, and the original depth flow data of involved party is obtained using pumped FIR laser.
Preferably, the Node extraction module is specifically used on the basis of the original depth data stream of the involved party,
The skeleton joint point data of human body is extracted by coordinate transform and smothing filtering mode, and is shown by visual processes technology
Show.
Preferably, the human skeleton artis data include 20 skeleton joint point datas of human body.
Preferably, the Activity recognition module includes：
Feature extraction submodule, it is special for the scalar information by extracting involved party's skeleton joint point to entire Human Modeling
Sign and Vector Message feature, wherein the scalar information characteristics include relative distance feature, the body joint point coordinate of two joint point
The angle feature constituted between the opposite variation characteristic of axis and joint and joint, the Vector Message feature includes artis
Velocity variations feature, an artis become relative to the changes in coordinates feature of another artis and the acceleration of special artis
Change feature；
Feature coding submodule, scalar information characteristics and the vector letter of involved party's skeleton joint point for will extract
Breath feature integrally carries out coding and constitutes input matrix, while being encoded to human body behavior；
Submodule is pre-processed, the coding for being carried out by the input matrix and to human body behavior is sent into limitation wave together
The graceful machine network of Wurz carries out data prediction；
Deep learning submodule, for learning pretreated data addition back-propagation algorithm neural network；
Model foundation submodule establishes deep neural network system model for passing through study, and utilizes depth god
Activity recognition is carried out through network system model.
In embodiments of the present invention, technical solution provided by the invention does not need involved party and dresses any sensor device,
It is compared with traditional video surveillance behavior, technical scheme of the present invention has only used depth depth of field number using kinect sensors
According to, and then protection privacy of user is can be very good, technical scheme of the present invention can identify a variety of human body behaviors, such as common
It the human bodies behavior such as falls, make a phone call, bending over, reading a book, sitting, squatting, technical scheme of the present invention passes through to there are many same behaviors
Feature representation greatly improves the accuracy rate of Activity recognition, and by a large amount of human body behavioral data pre-process automatically, drop
Dimension, and then Activity recognition speed is greatly improved, while the depth data to extracting does not have to carry out at the modes such as complicated image
Reason, but only do the extraction of simple data characteristics, so complete complicated human body behavioral data analysis, determine human body behavior,
And alert process is carried out to the abnormal behaviour of setting.
Description of the drawings
Fig. 1 is the Human bodys' response method flow diagram based on deep neural network in an embodiment of the present invention；
The sub-step flow chart that Fig. 2 is step S14 shown in Fig. 1 in an embodiment of the present invention；
Fig. 3 is the Human bodys' response system structure diagram based on deep neural network in an embodiment of the present invention；
Fig. 4 is the internal structure schematic diagram of Activity recognition module 14 shown in Fig. 3 in an embodiment of the present invention.
Specific implementation mode
In order to make the purpose , technical scheme and advantage of the present invention be clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.A kind of Human bodys' response method and identifying system based on deep neural network
The specific embodiment of the invention provides a kind of Human bodys' response method based on deep neural network, main to wrap
Include following steps：
S11, the original depth data stream for obtaining involved party；
S12, the skeleton joint point data that human body is extracted by the original depth data stream of the involved party；
S13, using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire Human Modeling；
S14, by carrying out feature extraction to entire Human Modeling, by characteristic be sent into limitation Boltzmann machine network
It is pre-processed, the weight initialization BP neural network parameter that will be obtained trains deep neural network model, is used in combination and trains
Deep neural network model Activity recognition is carried out to the result of feature extraction；
S15, using multi-threading parallel process, the human skeleton joint point data extracted and actual human body are subjected to weight
It closes, and the behavior recognized is subjected to real-time display；
S16, abnormal behaviour template library is established, and alarmed the abnormal behaviour detected.
A kind of Human bodys' response method based on deep neural network provided by the present invention does not need involved party's wearing
Any sensor device is compared with traditional video surveillance behavior, and technical scheme of the present invention is only used using kinect sensors
Depth depth of field data has been arrived, and then can be very good protection privacy of user, technical scheme of the present invention can identify a variety of human bodies
Behavior, such as common tumble, human bodies behavior, the technical scheme of the present invention such as make a phone call, bend over, reading a book, sitting, squatting pass through to same
A kind of behavior greatly improves the accuracy rate of Activity recognition there are many feature representation, and by a large amount of human body behavioral data into
The automatic pretreatment of row, dimensionality reduction, and then Activity recognition speed is greatly improved, while the depth data to extracting is complicated without carrying out
The modes such as image handle, but only do simple data characteristics extraction, so complete complicated human body behavioral data analysis,
It determines human body behavior and alert process is carried out to the abnormal behaviour of setting.
It below will be detailed to a kind of Human bodys' response method progress based on deep neural network provided by the present invention
Explanation.
Referring to Fig. 1, for the Human bodys' response method flow based on deep neural network in an embodiment of the present invention
Figure.
In step s 11, the original depth data stream of involved party is obtained.
In the present embodiment, the step S11 for obtaining the original depth data stream of involved party includes：
The involved party in the visual field is monitored using kinect sensors, and obtains collected rgb video stream in real time,
And the original depth flow data of involved party is obtained using pumped FIR laser.
In the present embodiment, kinect sensors can monitor all things in its visual field, wherein also wrapping
It includes and involved party is monitored, and detect involved party with the presence or absence of in its visual field.
In step s 12, the skeleton joint point data of human body is extracted by the original depth data stream of the involved party.
In the present embodiment, the skeleton joint point data of human body is extracted by the original depth data stream of the involved party
Step S12 include：
On the basis of the original depth data stream of the involved party, extracted by coordinate transform and smothing filtering mode
The skeleton joint point data of human body, and shown by visual processes technology.
In the present embodiment, it is mainly the skeleton for showing people using the human body that visual processes technology is shown, shows skeleton
Artis.
In step s 13, using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire people
Volume modeling.
In the present embodiment, human skeleton is divided into 5 skeleton branches of four limbs and backbone using body curve, then
The major joint point for extracting each skeleton branches forms human skeleton by connecting the major joint point of each skeleton branches
Figure, usually, each limbs in four limbs include 4 artis, for example, arm include shoulder joint point, elbow joint point,
This 4 artis of wrist joint point, hand point, remaining 4 artis include three artis and joint of head of backbone
Point, therefore, human skeleton artis data include 20 skeleton joint point datas of human body.
In step S14, by carrying out feature extraction to entire Human Modeling, characteristic is sent into limitation bohr hereby
Graceful machine network is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains deep neural network model, and
Activity recognition is carried out to the result of feature extraction with trained deep neural network model.
In the present embodiment, step S14 respectively includes five sub-steps S141-S145, as shown in Figure 2.
Referring to Fig. 2, for the sub-step flow chart of step S14 shown in Fig. 1 in an embodiment of the present invention.
In step s 141, by entire Human Modeling extract involved party's skeleton joint point scalar information characteristics and
Vector Message feature, wherein the scalar information characteristics include the relative distance feature of two joint point, the phase of body joint point coordinate axis
The angle feature constituted between variation characteristic and joint and joint, the Vector Message feature include that the speed of artis becomes
It is special relative to the changes in coordinates feature of another artis and the acceleration change of special artis to change feature, an artis
Sign.
In the present embodiment, the processing for carrying out the modes such as complicated image is not had to the depth data extracted, only
Do the extraction of simple data characteristics, these data characteristicses include just mainly involved party's skeleton joint point scalar information characteristics and
Vector Message feature, system completes complicated behavioral data analysis later by extracting these data characteristicses, and then determines human body
Behavior.
In step S142, by the scalar information characteristics and Vector Message feature of the involved party's skeleton joint point extracted
Entirety carries out coding and constitutes input matrix, while being encoded to human body behavior.
In the present embodiment, the object that system is encoded is not only that the scalar information of involved party's skeleton joint point is special
Sign and Vector Message feature, but also determining human body behavior is encoded.In the present embodiment, step S142 can be with
Very easily feature and behavior are extended.
In step S143, the coding carried out by the input matrix and to human body behavior is sent into limitation bohr hereby together
Graceful machine network carries out data prediction.
In the present embodiment, by limiting Boltzmann machine network (Restrict Boltzmann Machine, RBM)
Pretreatment, dimensionality reduction automatically can be carried out to a large amount of somatic data, to improve training speed.
In step S144, pretreated data addition back-propagation algorithm neural network is learnt.
In the present embodiment, a large amount of somatic data does automatic pretreatment not only by RBM, but also by reversed
Propagation algorithm (Back Propagation algorithm, referred to as：BP algorithm) neural network learnt.
In step S145, deep neural network system model is established by study, and utilize the deep neural network
System model carries out Activity recognition.
In the present embodiment, deep neural network system model is established by study, can identifies a variety of human body behaviors,
Such as common tumble, a variety of human body behaviors such as make a phone call, bend over, reading a book, sitting, squatting.
Please continue to refer to Fig. 1, in step S15, using multi-threading parallel process, the human skeleton artis that will be extracted
Data are overlapped with actual human body, and the behavior recognized is carried out real-time display.In the present embodiment, step S15 is used
Start in quickening network training, data extraction, equipment, the speed of Activity recognition, and then improves the processing speed of whole system.
In step s 16, abnormal behaviour template library is established, and is alarmed the abnormal behaviour detected.In this implementation
In mode, step S16 can also be set as establishing abnormal behaviour template library as needed, and the abnormal behaviour to detecting into
Row alarm.
A kind of Human bodys' response method based on deep neural network provided by the present invention does not need involved party's wearing
Any sensor device is compared with traditional video surveillance behavior, and technical scheme of the present invention is only used using kinect sensors
Depth depth of field data has been arrived, and then can be very good protection privacy of user, technical scheme of the present invention can identify a variety of human bodies
Behavior, such as common tumble, human bodies behavior, the technical scheme of the present invention such as make a phone call, bend over, reading a book, sitting, squatting pass through to same
A kind of behavior greatly improves the accuracy rate of Activity recognition there are many feature representation, and by a large amount of human body behavioral data into
The automatic pretreatment of row, dimensionality reduction, and then Activity recognition speed is greatly improved, while the depth data to extracting is complicated without carrying out
The modes such as image handle, but only do simple data characteristics extraction, so complete complicated human body behavioral data analysis,
It determines human body behavior and alert process is carried out to the abnormal behaviour of setting.
The specific embodiment of the invention also provides a kind of Human bodys' response system 10 based on deep neural network, mainly
Including：
Data acquisition module 11, the original depth data stream for obtaining involved party；
Node extraction module 12, the skeleton joint point for extracting human body by the original depth data stream of the involved party
Data；
Data modeling module 13, for using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, coming
To entire Human Modeling；
Activity recognition module 14, for by carrying out feature extraction to entire Human Modeling, characteristic being sent into and is limited
Boltzmann machine network processed is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains depth nerve net
Network model is used in combination trained deep neural network model to carry out Activity recognition to the result of feature extraction；
Data processing module 15, for use multi-threading parallel process, by the human skeleton joint point data extracted with
Actual human body is overlapped, and the behavior recognized is carried out real-time display；
Template establishes module 16, alarms for establishing abnormal behaviour template library, and to the abnormal behaviour detected.
A kind of Human bodys' response system 10 based on deep neural network provided by the present invention, does not need involved party and wears
It wears any sensor device to compare with traditional video surveillance behavior, technical scheme of the present invention utilizes kinect sensors
Depth depth of field data has been used, and then can be very good protection privacy of user, technical scheme of the present invention can identify a variety of people
Body behavior, such as common tumble, the human bodies behavior such as make a phone call, bend over, reading a book, sitting, squatting, it is right that technical scheme of the present invention passes through
There are many feature representations for same behavior, greatly improve the accuracy rate of Activity recognition, and by a large amount of human body behavioral data
Pretreatment, dimensionality reduction automatically are carried out, and then greatly improves Activity recognition speed, while the depth data to extracting does not have to answer
The modes such as miscellaneous image are handled, but only do simple data characteristics extraction, and then complete complicated human body behavioral data point
It analyses, determine human body behavior and alert process is carried out to the abnormal behaviour of setting.
Referring to Fig. 3, showing in an embodiment of the present invention the Human bodys' response system based on deep neural network
10 structural schematic diagram.In the present embodiment, the Human bodys' response system 10 based on deep neural network is adopted including data
Collect module 11, Node extraction module 12, data modeling module 13, Activity recognition module 14, data processing module 15 and template
Establish module 16.
Data acquisition module 11, the original depth data stream for obtaining involved party.
In the present embodiment, data acquisition module 11 are specifically used for using kinect sensors to the behavior in the visual field
People monitors, and obtains collected rgb video stream in real time, and the original depth fluxion of involved party is obtained using pumped FIR laser
According to.
In the present embodiment, kinect sensors can monitor all things in its visual field, wherein also wrapping
It includes and involved party is monitored, and detect involved party with the presence or absence of in its visual field.
Node extraction module 12, the skeleton joint point for extracting human body by the original depth data stream of the involved party
Data.
In the present embodiment, Node extraction module 12, specifically for the original depth data stream in the involved party
On the basis of, the skeleton joint point data of human body is extracted by coordinate transform and smothing filtering mode, and pass through visual processes skill
Art is shown.
In the present embodiment, it is mainly the skeleton for showing people using the human body that visual processes technology is shown, shows skeleton
Artis.
Data modeling module 13, for using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, coming
To entire Human Modeling.
In the present embodiment, human skeleton is divided into 5 skeleton branches of four limbs and backbone using body curve, then
The major joint point for extracting each skeleton branches forms human skeleton by connecting the major joint point of each skeleton branches
Figure, usually, each limbs in four limbs include 4 artis, for example, arm include shoulder joint point, elbow joint point,
This 4 artis of wrist joint point, hand point, remaining 4 artis include three artis and joint of head of backbone
Point, therefore, human skeleton artis data include 20 skeleton joint point datas of human body.
Activity recognition module 14, for by carrying out feature extraction to entire Human Modeling, characteristic being sent into and is limited
Boltzmann machine network processed is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains depth nerve net
Network model is used in combination trained deep neural network model to carry out Activity recognition to the result of feature extraction.
In the present embodiment, Activity recognition module 14 includes mainly feature extraction submodule 141, feature coding submodule
142, submodule 143, deep learning submodule 144 and model foundation submodule 145 are pre-processed, as shown in Figure 4.
Referring to Fig. 4, showing in an embodiment of the present invention the internal structure signal of Activity recognition module shown in Fig. 3 14
Figure.
Feature extraction submodule 141 is believed for the scalar by extracting involved party's skeleton joint point to entire Human Modeling
Cease feature and Vector Message feature, wherein the scalar information characteristics include relative distance feature, the artis of two joint point
The angle feature constituted between the opposite variation characteristic of reference axis and joint and joint, the Vector Message feature includes joint
Acceleration of the velocity variations feature, an artis of point relative to the changes in coordinates feature and special artis of another artis
Spend variation characteristic.
In the present embodiment, the processing for carrying out the modes such as complicated image is not had to the depth data extracted, only
Do the extraction of simple data characteristics, these data characteristicses include just mainly involved party's skeleton joint point scalar information characteristics and
Vector Message feature, system completes complicated behavioral data analysis later by extracting these data characteristicses, and then determines human body
Behavior.
Feature coding submodule 142, the scalar information characteristics and arrow of involved party's skeleton joint point for will extract
Amount information characteristics integrally carry out coding and constitute input matrix, while being encoded to human body behavior.
In the present embodiment, the object that system is encoded is not only that the scalar information of involved party's skeleton joint point is special
Sign and Vector Message feature, but also determining human body behavior is encoded.In the present embodiment, feature coding is set
Submodule 142 can very easily be extended feature and behavior.
Submodule 143 is pre-processed, for the input matrix and the coding carried out to human body behavior to be sent into limit together
Boltzmann machine network processed carries out data prediction.
In the present embodiment, by limiting Boltzmann machine network (Restrict Boltzmann Machine, RBM)
Pretreatment, dimensionality reduction automatically can be carried out to a large amount of somatic data, to improve training speed.
Deep learning submodule 144, for back-propagation algorithm neural network to be added in pretreated data
It practises.
In the present embodiment, a large amount of somatic data does automatic pretreatment not only by RBM, but also by reversed
Propagation algorithm (Back Propagation algorithm, referred to as：BP algorithm) neural network learnt.
Model foundation submodule 145 establishes deep neural network system model for passing through study, and utilizes the depth
Nerve network system model carries out Activity recognition.
In the present embodiment, deep neural network system model is established by study, can identifies a variety of human body behaviors,
Such as common tumble, a variety of human body behaviors such as make a phone call, bend over, reading a book, sitting, squatting.
Please continue to refer to Fig. 3, data processing module 15, for using multi-threading parallel process, the human body bone that will be extracted
Frame joint point data is overlapped with actual human body, and the behavior recognized is carried out real-time display.In the present embodiment, if
The speed of network training, data extraction, equipment startup, Activity recognition can be accelerated by setting data processing module 15, and then be improved whole
The processing speed of a system.
Template establishes module 16, alarms for establishing abnormal behaviour template library, and to the abnormal behaviour detected.
In present embodiment, template establishes module 16 can establish abnormal behaviour template library, and the abnormal row to detecting as needed
To alarm.
A kind of Human bodys' response system 10 based on deep neural network provided by the present invention, does not need involved party and wears
It wears any sensor device to compare with traditional video surveillance behavior, technical scheme of the present invention utilizes kinect sensors
Depth depth of field data has been used, and then can be very good protection privacy of user, technical scheme of the present invention can identify a variety of people
Body behavior, such as common tumble, the human bodies behavior such as make a phone call, bend over, reading a book, sitting, squatting, it is right that technical scheme of the present invention passes through
There are many feature representations for same behavior, greatly improve the accuracy rate of Activity recognition, and by a large amount of human body behavioral data
Pretreatment, dimensionality reduction automatically are carried out, and then greatly improves Activity recognition speed, while the depth data to extracting does not have to answer
The modes such as miscellaneous image are handled, but only do simple data characteristics extraction, and then complete complicated human body behavioral data point
It analyses, determine human body behavior and alert process is carried out to the abnormal behaviour of setting.
It is worth noting that, in above-described embodiment, included each unit is only divided according to function logic,
But it is not limited to above-mentioned division, as long as corresponding function can be realized；In addition, the specific name of each functional unit
Only to facilitate mutually distinguishing, the protection domain being not intended to restrict the invention.
In addition, one of ordinary skill in the art will appreciate that realizing all or part of step in the various embodiments described above method
It is that relevant hardware can be instructed to complete by program, corresponding program can be stored in a computer-readable storage and be situated between
In matter, the storage medium, such as ROM/RAM, disk or CD.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all essences in the present invention
All any modification, equivalent and improvement etc., should all be included in the protection scope of the present invention made by within refreshing and principle.
Claims (8)
1. a kind of Human bodys' response method based on deep neural network, which is characterized in that described to be based on deep neural network
Human bodys' response method include：
Obtain the original depth data stream of involved party；
The skeleton joint point data of human body is extracted by the original depth data stream of the involved party；
Using the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire Human Modeling；
By carrying out feature extraction to entire Human Modeling, characteristic is sent into limitation Boltzmann machine network and is located in advance
Reason, the weight initialization BP neural network parameter that will be obtained train deep neural network model, and trained depth god is used in combination
Activity recognition is carried out to the result of feature extraction through network model；
Using multi-threading parallel process, the human skeleton joint point data extracted is overlapped with actual human body, and will known
The behavior being clipped to carries out real-time display；
Abnormal behaviour template library is established, and is alarmed the abnormal behaviour detected；
It is described by carrying out feature extraction to entire Human Modeling, by characteristic be sent into limitation Boltzmann machine network carry out
Pretreatment, the weight initialization BP neural network parameter that will be obtained train deep neural network model, trained depth are used in combination
Spending the step of neural network model carries out Activity recognition to the result of feature extraction includes：
By extracting the scalar information characteristics and Vector Message feature of involved party's skeleton joint point to entire Human Modeling,
In, the scalar information characteristics include the relative distance feature of two joint point, body joint point coordinate axis opposite variation characteristic and
The angle feature constituted between joint and joint, the Vector Message feature include the velocity variations feature of artis, a joint
Point is relative to the changes in coordinates feature of another artis and the acceleration change feature of special artis；
The scalar information characteristics of the involved party's skeleton joint point extracted and Vector Message feature are integrally subjected to coding composition
Input matrix, while human body behavior is encoded；
The coding carried out by the input matrix and to human body behavior is sent into limitation Boltzmann machine network and carries out data together
Pretreatment；
Pretreated data addition back-propagation algorithm neural network is learnt；
Deep neural network system model is established by study, and behavior knowledge is carried out using the deep neural network system model
Not.
2. the Human bodys' response method based on deep neural network as described in claim 1, which is characterized in that the acquisition
The step of original depth data stream of involved party includes：
The involved party in the visual field is monitored using kinect sensors, and obtains collected rgb video stream in real time, and
The original depth data stream of involved party is obtained using pumped FIR laser.
3. the Human bodys' response method based on deep neural network as described in claim 1, which is characterized in that described to pass through
The involved party original depth data stream extraction human body skeleton joint point data the step of include：
On the basis of the original depth data stream of the involved party, human body is extracted by coordinate transform and smothing filtering mode
Skeleton joint point data, and shown by visual processes technology.
4. the Human bodys' response method based on deep neural network as described in claim 1, which is characterized in that the human body
Skeleton joint point data includes 20 skeleton joint point datas of human body.
5. a kind of Human bodys' response system based on deep neural network, which is characterized in that described to be based on deep neural network
Human bodys' response system include：
Data acquisition module, the original depth data stream for obtaining involved party；
Node extraction module, the skeleton joint point data for extracting human body by the original depth data stream of the involved party；
Data modeling module, for utilizing the three-dimensional coordinate corresponding to the human skeleton joint point data extracted, to entire
Human Modeling；
Activity recognition module, for by carrying out feature extraction to entire Human Modeling, characteristic to be sent into limitation bohr
Hereby graceful machine network is pre-processed, and the weight initialization BP neural network parameter that will be obtained trains deep neural network model,
It is used in combination trained deep neural network model to carry out Activity recognition to the result of feature extraction；
Data processing module, for using multi-threading parallel process, the human skeleton joint point data and actual persons that will be extracted
Body is overlapped, and the behavior recognized is carried out real-time display；
Template establishes module, alarms for establishing abnormal behaviour template library, and to the abnormal behaviour detected；
The Activity recognition module includes：
Feature extraction submodule, for by entire Human Modeling extract involved party's skeleton joint point scalar information characteristics with
And Vector Message feature, wherein the scalar information characteristics include the relative distance feature of two joint point, body joint point coordinate axis
The opposite angle feature constituted between variation characteristic and joint and joint, the Vector Message feature includes the speed of artis
Variation characteristic, an artis are special relative to the changes in coordinates feature of another artis and the acceleration change of special artis
Sign；
Feature coding submodule, the scalar information characteristics and Vector Message of involved party's skeleton joint point for will extract are special
The whole coding that carries out of sign constitutes input matrix, while being encoded to human body behavior；
Submodule is pre-processed, the coding for being carried out by the input matrix and to human body behavior is sent into limitation bohr hereby together
Graceful machine network carries out data prediction；
Deep learning submodule, for learning pretreated data addition back-propagation algorithm neural network；
Model foundation submodule establishes deep neural network system model for passing through study, and utilizes the depth nerve net
Network system model carries out Activity recognition.
6. the Human bodys' response system based on deep neural network as claimed in claim 5, which is characterized in that the data
Acquisition module specifically for being monitored to the involved party in the visual field using kinect sensors, and is obtained collected in real time
Rgb video stream, and the original depth data stream using pumped FIR laser acquisition involved party.
7. the Human bodys' response system based on deep neural network as claimed in claim 5, which is characterized in that the node
Extraction module is specifically used on the basis of the original depth data stream of the involved party, passes through coordinate transform and smooth filter
Wave mode extracts the skeleton joint point data of human body, and is shown by visual processes technology.
8. the Human bodys' response system based on deep neural network as claimed in claim 6, which is characterized in that the human body
Skeleton joint point data includes 20 skeleton joint point datas of human body.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510293654.2A CN104850846B (en) | 2015-06-02 | 2015-06-02 | A kind of Human bodys' response method and identifying system based on deep neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510293654.2A CN104850846B (en) | 2015-06-02 | 2015-06-02 | A kind of Human bodys' response method and identifying system based on deep neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104850846A CN104850846A (en) | 2015-08-19 |
| CN104850846B true CN104850846B (en) | 2018-08-24 |
Family
ID=53850481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510293654.2A Expired - Fee Related CN104850846B (en) | 2015-06-02 | 2015-06-02 | A kind of Human bodys' response method and identifying system based on deep neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104850846B (en) |
Families Citing this family (59)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105303563B (en) * | 2015-09-22 | 2018-04-17 | 北京格灵深瞳信息技术有限公司 | A kind of fall detection method and device |
| CN105125220A (en) * | 2015-10-20 | 2015-12-09 | 重庆软汇科技股份有限公司 | Falling-down detection method |
| CN105787469B (en) * | 2016-03-25 | 2019-10-18 | 浩云科技股份有限公司 | The method and system of pedestrian monitoring and Activity recognition |
| CN105913559B (en) * | 2016-04-06 | 2019-03-05 | 南京华捷艾米软件科技有限公司 | A kind of ATM in bank intelligent control method based on body-sensing technology |
| CN106056035A (en) * | 2016-04-06 | 2016-10-26 | 南京华捷艾米软件科技有限公司 | Motion-sensing technology based kindergarten intelligent monitoring method |
| CN106127733B (en) * | 2016-06-14 | 2019-02-22 | 湖南拓视觉信息技术有限公司 | The method and apparatus of human body target identification |
| CN106203503B (en) * | 2016-07-08 | 2019-04-05 | 天津大学 | A kind of action identification method based on bone sequence |
| CN106203363A (en) * | 2016-07-15 | 2016-12-07 | 中国科学院自动化研究所 | Human skeleton motion sequence Activity recognition method |
| CN106504266B (en) | 2016-09-29 | 2019-06-14 | 北京市商汤科技开发有限公司 | The prediction technique and device of walking behavior, data processing equipment and electronic equipment |
| CN106600000A (en) * | 2016-12-05 | 2017-04-26 | 中国科学院计算技术研究所 | Method and system for human-robot motion data mapping |
| CN107045623B (en) * | 2016-12-30 | 2020-01-21 | 厦门瑞为信息技术有限公司 | Indoor dangerous condition warning method based on human body posture tracking analysis |
| CN106909891B (en) * | 2017-01-24 | 2019-11-15 | 华南农业大学 | A kind of Human bodys' response method based on self feed back gene expression programming |
| GB2560177A (en) | 2017-03-01 | 2018-09-05 | Thirdeye Labs Ltd | Training a computational neural network |
| GB2560387B (en) | 2017-03-10 | 2022-03-09 | Standard Cognition Corp | Action identification using neural networks |
| GB2603640B (en) * | 2017-03-10 | 2022-11-16 | Standard Cognition Corp | Action identification using neural networks |
| CN106951852A (en) * | 2017-03-15 | 2017-07-14 | 深圳汇创联合自动化控制有限公司 | A kind of effective Human bodys' response system |
| CN107220604A (en) * | 2017-05-18 | 2017-09-29 | 清华大学深圳研究生院 | A kind of fall detection method based on video |
| CN107220608B (en) * | 2017-05-22 | 2021-06-08 | 华南理工大学 | Basketball action model reconstruction and defense guidance system and method |
| CN107506781A (en) * | 2017-07-06 | 2017-12-22 | 浙江工业大学 | A kind of Human bodys' response method based on BP neural network |
| CN107563281A (en) * | 2017-07-24 | 2018-01-09 | 南京邮电大学 | A kind of construction site personal security hidden danger monitoring method based on deep learning |
| US11200692B2 (en) | 2017-08-07 | 2021-12-14 | Standard Cognition, Corp | Systems and methods to check-in shoppers in a cashier-less store |
| CN109558785A (en) * | 2017-09-25 | 2019-04-02 | 北京缤歌网络科技有限公司 | Safety defense monitoring system and the unmanned convenience store for applying it |
| CN107943276A (en) * | 2017-10-09 | 2018-04-20 | 广东工业大学 | Based on the human body behavioral value of big data platform and early warning |
| CN107679522B (en) * | 2017-10-31 | 2020-10-13 | 内江师范学院 | Multi-stream LSTM-based action identification method |
| CN107818312A (en) * | 2017-11-20 | 2018-03-20 | 湖南远钧科技有限公司 | A kind of embedded system based on abnormal behaviour identification |
| CN107837087A (en) * | 2017-12-08 | 2018-03-27 | 兰州理工大学 | A kind of human motion state recognition methods based on smart mobile phone |
| CN109934881B (en) * | 2017-12-19 | 2022-02-18 | 华为技术有限公司 | Image coding method, motion recognition method and computer equipment |
| CN108875523B (en) * | 2017-12-28 | 2021-02-26 | 北京旷视科技有限公司 | Human body joint point detection method, device, system and storage medium |
| CN108182416A (en) * | 2017-12-30 | 2018-06-19 | 广州海昇计算机科技有限公司 | A kind of Human bodys' response method, system and device under monitoring unmanned scene |
| CN108764059B (en) * | 2018-05-04 | 2021-01-01 | 南京邮电大学 | Human behavior recognition method and system based on neural network |
| CN108875614A (en) * | 2018-06-07 | 2018-11-23 | 南京邮电大学 | It is a kind of that detection method is fallen down based on deep learning image procossing |
| CN108830215B (en) * | 2018-06-14 | 2021-07-13 | 南京理工大学 | Dangerous behavior identification method based on personnel skeleton information |
| CN109299646B (en) * | 2018-07-24 | 2021-06-25 | 北京旷视科技有限公司 | Crowd abnormal event detection method, device, system and storage medium |
| CN109344705B (en) * | 2018-08-27 | 2023-05-23 | 广州烽火众智数字技术有限公司 | Pedestrian behavior detection method and system |
| CN109389041B (en) * | 2018-09-07 | 2020-12-01 | 南京航空航天大学 | Fall detection method based on joint point characteristics |
| CN109145868A (en) * | 2018-09-11 | 2019-01-04 | 广州杰赛科技股份有限公司 | A kind of Activity recognition method and apparatus assisting running training |
| CN109409209A (en) * | 2018-09-11 | 2019-03-01 | 广州杰赛科技股份有限公司 | A kind of Human bodys' response method and apparatus |
| CN109522793B (en) * | 2018-10-10 | 2021-07-23 | 华南理工大学 | Method for detecting and identifying abnormal behaviors of multiple persons based on machine vision |
| CN109325469B (en) * | 2018-10-23 | 2022-06-14 | 北京工商大学 | Human body posture recognition method based on deep neural network |
| CN109472895A (en) * | 2018-10-31 | 2019-03-15 | 广州畅驿智能科技有限公司 | A kind of security protection integrated application management system and its management implementation method |
| CN109567839B (en) * | 2018-11-20 | 2022-04-26 | 北京中科研究院 | Automatic analysis method for hip joint X-ray image |
| CN109740091B (en) * | 2018-12-26 | 2021-08-03 | 武汉大学 | Behavior cognition-based user network behavior prediction system and method |
| CN109920208A (en) * | 2019-01-31 | 2019-06-21 | 深圳绿米联创科技有限公司 | Tumble prediction technique, device, electronic equipment and system |
| CN110087099B (en) * | 2019-03-11 | 2020-08-07 | 北京大学 | Monitoring method and system for protecting privacy |
| CN110070003B (en) * | 2019-04-01 | 2021-07-30 | 浙江大华技术股份有限公司 | Abnormal behavior detection and optical flow autocorrelation determination method and related device |
| CN110075524B (en) * | 2019-05-10 | 2020-11-13 | 腾讯科技（深圳）有限公司 | Abnormal behavior detection method and device |
| CN110390303B (en) * | 2019-07-24 | 2022-04-08 | 达闼机器人有限公司 | Tumble alarm method, electronic device, and computer-readable storage medium |
| CN110826401B (en) * | 2019-09-26 | 2023-12-26 | 广州视觉风科技有限公司 | Human body limb language identification method and system |
| CN111932828A (en) * | 2019-11-05 | 2020-11-13 | 上海中侨健康智能科技有限公司 | Intelligent old-age care monitoring and early warning system based on digital twin technology |
| CN111209793A (en) * | 2019-12-05 | 2020-05-29 | 重庆特斯联智慧科技股份有限公司 | Region shielding human body security check method and system based on artificial intelligence |
| CN111723633B (en) * | 2019-12-09 | 2022-05-20 | 深圳市鸿逸达科技有限公司 | Personnel behavior pattern analysis method and system based on depth data |
| US11574468B2 (en) * | 2020-03-31 | 2023-02-07 | Toyota Research Institute, Inc. | Simulation-based learning of driver interactions through a vehicle window |
| US11361468B2 (en) | 2020-06-26 | 2022-06-14 | Standard Cognition, Corp. | Systems and methods for automated recalibration of sensors for autonomous checkout |
| US11303853B2 (en) | 2020-06-26 | 2022-04-12 | Standard Cognition, Corp. | Systems and methods for automated design of camera placement and cameras arrangements for autonomous checkout |
| CN111783711B (en) * | 2020-07-09 | 2022-11-08 | 中国科学院自动化研究所 | Skeleton behavior identification method and device based on body component layer |
| CN112507870A (en) * | 2020-12-08 | 2021-03-16 | 南京代威科技有限公司 | Behavior recognition method and system through human skeleton extraction technology |
| CN112686208B (en) * | 2021-01-22 | 2022-11-08 | 上海喵眼智能科技有限公司 | Motion recognition characteristic parameter algorithm based on machine vision |
| CN114155555B (en) * | 2021-12-02 | 2022-06-10 | 北京中科智易科技有限公司 | Human behavior artificial intelligence judgment system and method |
| CN115988130A (en) * | 2022-12-22 | 2023-04-18 | 山西警察学院 | Method for identifying input content in motion state based on mobile phone sensor |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104217214A (en) * | 2014-08-21 | 2014-12-17 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | Configurable convolutional neural network based red green blue-distance (RGB-D) figure behavior identification method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5713159B2 (en) * | 2010-03-24 | 2015-05-07 | 独立行政法人産業技術総合研究所 | Three-dimensional position / orientation measurement apparatus, method and program using stereo images |
-
2015
- 2015-06-02 CN CN201510293654.2A patent/CN104850846B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104217214A (en) * | 2014-08-21 | 2014-12-17 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | Configurable convolutional neural network based red green blue-distance (RGB-D) figure behavior identification method |
Non-Patent Citations (1)
| Title |
|---|
| 基于Kinect传感器骨骼信息的人体动作识别;朱国刚等;《计算机仿真》;20141231;第31卷(第12期);第2-5节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104850846A (en) | 2015-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104850846B (en) | A kind of Human bodys' response method and identifying system based on deep neural network | |
| EP3596656B1 (en) | Monitoring and analyzing body language with machine learning, using artificial intelligence systems for improving interaction between humans, and humans and robots | |
| CN106956271B (en) | Predict the method and robot of affective state | |
| CN109620185B (en) | Autism auxiliary diagnosis system, device and medium based on multi-modal information | |
| CN107153871B (en) | Falling detection method based on convolutional neural network and mobile phone sensor data | |
| KR102039848B1 (en) | Personal emotion-based cognitive assistance systems, methods of providing personal emotion-based cognitive assistance, and non-transitory computer readable media for improving memory and decision making | |
| CN103678417B (en) | Human-machine interaction data treating method and apparatus | |
| CN109765991A (en) | Social interaction system is used to help system and non-transitory computer-readable storage media that user carries out social interaction | |
| CN107862270A (en) | Face classification device training method, method for detecting human face and device, electronic equipment | |
| Permana et al. | Classification of bird sounds as an early warning method of forest fires using Convolutional Neural Network (CNN) algorithm | |
| US20160104385A1 (en) | Behavior recognition and analysis device and methods employed thereof | |
| KR102089002B1 (en) | Method and wearable device for providing feedback on action | |
| WO2022067871A1 (en) | Vr-interaction-technology-based auditory stimulation training method for children with autism | |
| CN110916689A (en) | Cognitive and attention-strengthening intelligent evaluation training system and method for autism | |
| KR102372976B1 (en) | Method for providing cognitive reinforcement training game | |
| KR101988334B1 (en) | a mobile handset and a method of analysis efficiency for multimedia content displayed on the mobile handset | |
| CN107704881A (en) | A kind of data visualization processing method and processing device based on animal electroencephalogramrecognition recognition | |
| Miranda et al. | WEMAC: Women and emotion multi-modal affective computing dataset | |
| CN112860213B (en) | Audio processing method and device, storage medium and electronic equipment | |
| KR20170036927A (en) | System for building social emotion network and method thereof | |
| Jianwattanapaisarn et al. | Emotional characteristic analysis of human gait while real-time movie viewing | |
| CN107363862A (en) | Social intercourse system based on robot | |
| US20190008466A1 (en) | Life log utilization system, life log utilization method, and recording medium | |
| Gupta et al. | REDE-Detecting human emotions using CNN and RASA | |
| JP7452558B2 (en) | Processing equipment, processing method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180824 Termination date: 20210602 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |