CN109711277A - Behavioural characteristic extracting method, system, device based on space-time frequency domain blended learning - Google Patents
Behavioural characteristic extracting method, system, device based on space-time frequency domain blended learning Download PDFInfo
- Publication number
- CN109711277A CN109711277A CN201811494799.9A CN201811494799A CN109711277A CN 109711277 A CN109711277 A CN 109711277A CN 201811494799 A CN201811494799 A CN 201811494799A CN 109711277 A CN109711277 A CN 109711277A
- Authority
- CN
- China
- Prior art keywords
- time
- behavioural characteristic
- space
- domain
- space domain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003542 behavioural effect Effects 0.000 title claims abstract description 147
- 238000000034 method Methods 0.000 title claims abstract description 52
- 230000006399 behavior Effects 0.000 claims abstract description 58
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 30
- 230000001360 synchronised effect Effects 0.000 claims abstract description 21
- 230000003044 adaptive effect Effects 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims abstract description 15
- 239000000284 extract Substances 0.000 claims abstract description 8
- 230000009466 transformation Effects 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 19
- 230000003416 augmentation Effects 0.000 claims description 15
- 108091006146 Channels Proteins 0.000 claims description 14
- 238000013528 artificial neural network Methods 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 210000004218 nerve net Anatomy 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 238000003780 insertion Methods 0.000 claims description 3
- 230000037431 insertion Effects 0.000 claims description 3
- 210000005036 nerve Anatomy 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 10
- 230000002123 temporal effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 239000004615 ingredient Substances 0.000 description 3
- 238000009412 basement excavation Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000003475 lamination Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- NBQKINXMPLXUET-UHFFFAOYSA-N Pranlukast Chemical compound C=1C=C(OCCCCC=2C=CC=CC=2)C=CC=1C(=O)NC1=CC=CC(C(C=2)=O)=C1OC=2C=1N=NNN=1 NBQKINXMPLXUET-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001680 brushing effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007620 mathematical function Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Image Analysis (AREA)
- Complex Calculations (AREA)
Abstract
The invention belongs to Activity recognition fields, and in particular to a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning, system, device, it is intended to for the low problem of skeleton behavioural characteristic extraction accuracy.The method of the present invention includes: to obtain the video behavior sequence based on skeleton, extracts time-space domain behavioural characteristic figure by converting network；Time-space domain is gone back in inversion after input frequency domain pays attention to network progress frequency selection, is added with time-space domain behavioural characteristic figure；It is synchronous to carry out part and non local reasoning, and carry out high-rise local reasoning；The time-space domain behavioural characteristic figure global pool that reasoning is obtained, obtains the behavioural characteristic vector of video behavior sequence, can be applied to behavior classification and behavioral value etc..The present invention selects effective frequency mode in frequency domain adaptive, uses in time-space domain while there is the network of local and non local affine field to carry out spatio temporal reasoning, can synchronize and excavate local detail and non local semantic information, to effectively raise the precision of Activity recognition.
Description
Technical field
The invention belongs to Activity recognition fields, and in particular to a kind of behavioural characteristic extraction based on space-time frequency domain blended learning
Method, system, device.
Background technique
Activity recognition has a wide range of applications in fields such as intelligent monitoring, human-computer interaction and automatic Pilots, Activity recognition packet
Behavior classification and behavioral value are included, is specifically exactly with dedicated acquisition equipment acquisition based on information such as RGB, depth, skeletons
Behavior video, classified to it, positioned and detected.Based on the Activity recognition of skeleton since computing cost is small, indicate succinct,
And to the variations such as environment, appearance more robust, cause the broad interest of academia and industrial circle in recent years.Specifically, skeleton row
It is exactly the video sequence of the 2D 3D coordinate composition of the artis according to target object in the environment for identification, to realize to row
For identification.
Existing skeleton Activity recognition method mainly stacks the localized network only with the affine field in part using in time-space domain
Hierarchically to extract the space-time characteristic of behavior sequence, and then behavior is identified and detected.As clapping hands, brushing teeth, shaking hands etc. this
A little behaviors are rich in the inherent frequecy characteristic for having distinction, and existing method is confined to excavate spatiotemporal mode, has ignored behavior
The frequency domain mode of middle inherence, and previously the hierarchical in time-space domain stacks localized network, so that semantic information can only be in high level
Extract, detailed information is mainly extracted in bottom again, detailed information and semantic information cannot synchronous extraction and fusion, be unfavorable for excavating
Effective behavioural characteristic is unable to satisfy requirement so that skeleton Activity recognition precision is low.
Summary of the invention
In order to solve the above problem in the prior art, in order to solve the problems, such as that behavioural characteristic extraction accuracy is low, this hair
It is bright to provide a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning, comprising:
Step S1 obtains the video behavior sequence based on skeleton, and as original video behavior sequence, it is adaptive to carry out time-space domain
It should convert, obtain the first time-space domain behavioural characteristic figure；
First time-space domain behavioural characteristic figure is sent into inversion after frequency domain carries out frequency selection and gains time-space domain by step S2, with
First time-space domain behavioural characteristic figure is added in a manner of residual error, obtains the second time-space domain behavioural characteristic figure；
Step S3, it is synchronous to the second time-space domain behavioural characteristic figure to carry out part and non local reasoning, and with the first time-space domain
Behavioural characteristic figure is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure；
Step S4 carries out high-rise local reasoning to third time-space domain behavioural characteristic figure, obtains the 4th time-space domain behavioural characteristic
Figure；
4th time-space domain behavioural characteristic figure global pool is obtained behavioural characteristic vector by step S5.
In some preferred embodiments, in step S1 " time-space domain self-adaptive transformation ", it the steps include:
Step S11 uses core oblique at K to the original video behavior sequence for 1 convolutional network or fully-connected network
The adaptive augmentation that coordinate system is carried out under coordinate system, obtains the augmentation video behavior sequence under K coordinate system, and K is super ginseng
Number.
Step S12, using using core to be 1 convolutional network or fully-connected network in the augmentation video behavior sequence
Skeleton carries out joint number and joint arrangement sequence is converted, and obtains the augmentation optimization video behavior sequence comprising structural information
Characteristic pattern, be the first time-space domain behavioural characteristic figure.
In some preferred embodiments, " the first time-space domain behavioural characteristic figure is sent into frequency domain in step S2 and carries out frequency
Inversion gains time-space domain after selection, is added in a manner of residual error with the first time-space domain behavioural characteristic figure ", it the steps include:
The characteristic pattern in each channel is transformed to frequency domain using two dimension discrete fourier transform by step S21 respectively, comprising just
String frequency domain character figure and cosine frequency domain character figure；
In view of computational efficiency, two-dimensional discrete Fast Fourier Transform can be used and realize characteristic pattern transformation.
Step S22 respectively learns the sinusoidal frequency domain character figure and cosine frequency domain character figure out just by paying attention to network
String ingredient attention weight and cosinusoidal component attention weight；
Wherein, notice that network, including a channel average layer, two full articulamentums, a softmax function and one lead to
Road duplicating layer.
Step S23 carries out dot product, cosinusoidal component with the sine component attention weight learnt and sinusoidal frequency domain character figure
Attention weight and cosine frequency domain character figure carry out dot product, sine and cosine frequency domain character figure after obtaining frequency selection.
Sinusoidal and cosine frequency domain character figure is transformed to time-space domain using two-dimensional discrete inverse fourier transform by step S24, with
The mode of residual error is added with the first time-space domain behavioural characteristic figure, obtains the second time-space domain behavioural characteristic figure；
In view of computational efficiency, two-dimensional discrete fast Fourier inverse transformation can be used and realize characteristic pattern inverse transformation.
It is " local and non-to the synchronous progress of the second time-space domain behavioural characteristic figure in step S3 in some preferred embodiments
Local reasoning ", the steps include:
Step S31 constructs the neural network submodule y with the affine field in parti, the nerve net with non local affine field
String bag module y 'i:
Wherein, xiRepresent the feature vector of the time-space domain characteristic pattern of current layer network；yiWith y 'iRespectively represent next layer of net
The feature vector of the time-space domain characteristic pattern of the part of network and non local affine field；A(xi,xj) it is parent between calculating position i and j
With the binary transformation matrix of degree；g(xi) it is to calculate xjFeature insertion unitary transforming function transformation function, by convolution kernel be 1 or 1 × 1 volume
Lamination is realized；ZiIt (X) is normalization factor, Ω enumerates all feature locations, δiFor local domain.
The feature that local and non local affine field neural network submodule extracts is had the right to be superimposed and obtains characteristic pattern, and to institute
It states characteristic pattern and carries out batch normalization reduction feature drift, introduce non-linear unit, then carry out down-sampling to reduce the resolution of characteristic pattern
Rate；
Step S32 is led using the M1 parts and non local affine field neural network submodule calculating position i and part
Domain δiThe affinity of all possible positions in affinity and i and Ω between interior neighbours, M1 are the nature more than or equal to 1
Number；
Step S33, will be by the characteristic pattern and first of M1 part and non local affine field neural network submodule reasoning
Time-space domain characteristic pattern is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure.
In some preferred embodiments, " carry out high-rise part to third time-space domain behavioural characteristic figure to push away in step S4
Reason ", method are as follows:
The part constructed using M2 is affine, and neural network submodule in field calculates the third time-space domain behavioural characteristic figure group
Position i and local domain δiAffinity between interior neighbours, M2 are the natural number more than or equal to 1；Characteristic pattern after reasoning
For the 4th time-space domain behavioural characteristic figure.
Another aspect of the present invention proposes a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning, packet
It includes:
Velocity information is obtained to the original difference on time dimension of the video behavior sequence based on skeleton, construction includes position
With the behavior sequence of speed；
The described in any item step S1- steps of claim 1-5 are used to the behavior sequence comprising position and speed respectively
S5 is handled, and the feature vector of corresponding speed and the feature vector of corresponding position are obtained；
Described eigenvector is spliced to obtain splicing feature vector, the behavioural characteristic vector of extraction be velocity characteristic vector,
Position feature vector sum splices feature vector.
Third aspect present invention proposes a kind of behavioural characteristic extraction system based on space-time frequency domain blended learning, including
Video sequence obtain module, adaptive transformation module, frequency-selecting module, it is local with non local synchronous reasoning module, high-rise office
Portion's reasoning module, global pool module, splicing module, output module；
The video sequence obtains module, is configured to obtain the video behavior sequence based on skeleton, as original video row
For sequence；
The adaptive transformation module is configured in time-space domain in such a way that augmentation optimizes, extracts the first time-space domain row
It is characterized figure；
The frequency-selecting module is configured to the first time-space domain behavioural characteristic figure feeding frequency domain noticing that network carries out frequency
Selection, transforms to time-space domain and first time-space domain behavioural characteristic figure phase in a manner of residual error for the frequency domain behavioural characteristic figure of acquisition
Add, obtains the second time-space domain behavioural characteristic figure；
The part and non local synchronous reasoning module are configured to synchronous to the second time-space domain behavioural characteristic figure carry out part
With non local reasoning, and it is added to obtain third time-space domain behavioural characteristic in a manner of residual error with the first time-space domain behavioural characteristic figure
Figure；
The high-rise local reasoning module is configured to carry out third time-space domain behavioural characteristic figure high-rise local reasoning, obtain
To the 4th time-space domain behavioural characteristic figure；
The global pool module is configured to the 4th time-space domain behavioural characteristic figure group global pool obtaining corresponding row
For feature vector；
The splicing module is configured to splice multi-channel feature, obtains splicing feature vector accordingly；
The output module is configured to the behavioural characteristic vector that will be extracted output.
Fourth aspect present invention proposes a kind of storage device, wherein be stored with a plurality of program, described program be suitable for by
Reason device is loaded and is executed to realize the above-mentioned behavioural characteristic extracting method based on space-time frequency domain blended learning.
Fifth aspect present invention proposes a kind of processing unit, including processor, is adapted for carrying out each program；And it deposits
Storage device is suitable for storing a plurality of program；Described program is suitable for being loaded by processor and being executed above-mentioned based on time space frequency to realize
The behavioural characteristic extracting method of domain blended learning.
Beneficial effects of the present invention:
(1) it is only sufficiently dug with the limitation of the spatiotemporal mode of depth Web Mining behavior frame sequence before the present invention is broken through
In pick behavior the frequency mode for having judgement index frequency domain character is carried out to frequency domain character figure in frequency domain using attention mechanism
Automobile driving, by learning end to end, effective frequency mode is adaptive selected in final association.
(2) detailed information and semanteme can only be asynchronously extracted respectively in low layer and upper layer network compared to pervious localized network
Information, synchronization proposed by the present invention have part and the network module of non local affine field each layer can be synchronous extraction with
Local detail and overall situation semanteme are merged, the number of plies and parameter of network can be effectively reduced in relatively traditional localized network.
(3) adaptive transformation network proposed by the present invention, coordinate transform network can be by original in single rectangular co-ordinate
The lower skeleton indicated of system is transformed under multiple oblique coordinates systems by study, obtains richer expression；Skeleton transformation network simultaneously
Optimal joint number and joint arrangement sequence, which can also be relearned, can acquire and more tie compared to previous structureless expression
The feature of structure, and then improve feature extraction precision.
Detailed description of the invention
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is that the present invention is based on the flow diagrams of the behavioural characteristic extracting method of space-time frequency domain blended learning；
Fig. 2 is the overall framework signal of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning
Figure；
Fig. 3 is that the frequency domain of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning pays attention to network
Structural schematic diagram；
Fig. 4 is the non-office of two-dimension time-space of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning
Portion's network plug-in schematic diagram；
Fig. 5 is the localized network module of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning
Schematic diagram；
Fig. 6 is the part of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning and non local
Synchronization module schematic diagram；
Fig. 7 is the part of the behavioural characteristic extracting method embodiment the present invention is based on space-time frequency domain blended learning and non local
The affine field schematic diagram of synchronization module.
Specific embodiment
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is only used for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to just
Part relevant to related invention is illustrated only in description, attached drawing.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Existing Activity recognition method is mainly layered using the localized network only with the affine field in part is stacked in time-space domain
Ground extracts the space-time characteristic of behavior sequence, and then behavior is identified and detected, and is confined to excavate spatiotemporal mode, has ignored row
For the frequency domain mode of middle inherence, and localized network is stacked in the hierarchical of time-space domain, so that semantic information can only be mentioned in high level
It takes, detailed information is mainly extracted in bottom again, and detailed information and semantic information are unable to synchronous fusion, are unfavorable for excavating effective row
It is characterized.The technical solution of the present invention selection effective frequency mode adaptive using attention mechanism in frequency domain, in time-space domain
Using having the network of local and non local affine field to carry out spatio temporal reasoning simultaneously, network is made to synchronize excavation in each layer module
Local detail and non local semantic information, to effectively raise the precision of skeleton behavioural characteristic extraction.
A kind of behavioural characteristic extracting method based on space-time frequency domain blended learning of the invention, comprising:
Step S1 obtains the video behavior sequence based on skeleton, and as original video behavior sequence, it is adaptive to carry out time-space domain
It should convert, obtain the first time-space domain behavioural characteristic figure；
The first time-space domain behavioural characteristic figure is sent into inversion after frequency domain carries out frequency selection and gains space-time by step S2
Domain is added in a manner of residual error with the first time-space domain behavioural characteristic figure, obtains the second time-space domain behavioural characteristic figure；
Step S3, it is synchronous to the second time-space domain behavioural characteristic figure to carry out part and non local reasoning, and with described the
One time-space domain behavioural characteristic figure is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure；
Step S4 carries out high-rise local reasoning to the third time-space domain behavioural characteristic figure, obtains the 4th time-space domain behavior
Characteristic pattern；
The 4th time-space domain behavioural characteristic figure global pool is obtained behavioural characteristic vector by step S5.
In order to more clearly to the present invention is based on the Activity recognition methods of space-time frequency domain blended learning to be illustrated, tie below
It closes Fig. 1-Fig. 7 and expansion detailed description is carried out to each step in a kind of embodiment of our inventive method.
We invent a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning of embodiment, including step
S1- step S5, each step are described in detail as follows:
Step S1 obtains the video behavior sequence based on skeleton, and as original video behavior sequence, it is adaptive to carry out time-space domain
It should convert, obtain the first time-space domain behavioural characteristic figure.
Step S11, note original video behavior sequence are X, and dimension C0*T0*N0, C0 are port number, and T0 is time dimension,
N0 is space artis number；
Use core for 1 convolutional network or fully-connected network to the original video behavior sequence under K oblique coordinates system
The adaptive augmentation for carrying out coordinate system, obtains the augmentation video behavior sequence under K coordinate system, K is hyper parameter；
Step S12 carries out joint number to the skeleton in the augmentation video behavior sequence using multilayer fully-connected network
It is converted with joint arrangement sequence, obtains the characteristic pattern of the augmentation optimization video behavior sequence comprising structural information, be first
Time-space domain behavioural characteristic figure X', dimension C'*T'*N', C' are port number, and T' is time dimension, and N' is space artis number.
The first time-space domain behavioural characteristic figure is sent into inversion after frequency domain carries out frequency selection and gains space-time by step S2
Domain is added in a manner of residual error with the first time-space domain behavioural characteristic figure, obtains the second time-space domain behavioural characteristic figure.
Step S21 is utilized two dimension discrete fourier transform (2D-DFT, 2D-Discrete Fourier Transform)
The characteristic pattern in each channel is transformed into frequency domain respectively, is denoted as Y, as shown in formula (1):
Wherein, c, u, v represent the channel of frequency domain character figure, temporal frequency dimension, spatial frequency dimension；When c, t, n are represented
The channel of spatial feature figure, time dimension, Spatial Dimension；T is the port number of the first time-space domain characteristic pattern；N is frequency domain character figure
Spatial Dimension is always counted.
In view of computational efficiency, two-dimensional discrete Fast Fourier Transform (2D-FFT, 2D-Fast Fourier can be used
Transformation characteristic pattern transformation) is realized.
Finally obtained frequency domain character figure Y includes two ingredients, a sinusoidal frequency domain character figure F altogethersin, a cosine frequency
Characteristic of field figure Fcos。
Step S22, building frequency domain pay attention to network, as shown in figure 3, including a channel average layer, two full articulamentums, one
A softmax function and a channel duplicating layer.
Respectively by sinusoidal frequency domain character figure FsinWith cosine frequency domain character figure FcosBy paying attention to network, learn sine component out
Attention weight MsinWith cosinusoidal component attention weight Mcos。
Step S23, with the sinusoidal attention weight M learntsinWith sinusoidal frequency domain character figure FsinCarry out dot product, cosinusoidal component
Attention weight McosWith cosine frequency domain character figure FcosDot product is carried out, the frequency component for having judgement index is selected, is denoted as F 'i, such as formula
(2) shown in:
Step S24 utilizes two-dimensional discrete inverse fourier transform (2D-IDFT, 2D-Inverse Discrete Fourier
Transform sinusoidal and cosine frequency domain character figure) is switched back into time-space domain, obtains time-space domain characteristic pattern X ", as shown in formula (3):
X "=X'+iift2 (F 'sin+F′cos), X " ∈ RC”×T”×N”Formula (3)
Wherein, C ", T " and N " are respectively the port number of time-space domain characteristic pattern X ", and time dimension is always counted and Spatial Dimension is total
Points.
In view of computational efficiency, two-dimensional discrete fast Fourier inverse transformation (2D-IFFT, 2D-Inverse Fast can be used
Fourier Transformation) realize characteristic pattern inverse transformation.
X " is added with the first time-space domain behavioural characteristic figure in a manner of residual error, obtains the second time-space domain behavioural characteristic figure.
Step S3, it is synchronous to the second time-space domain behavioural characteristic figure to carry out part and non local reasoning, and with described the
One time-space domain behavioural characteristic figure is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure.
Step S31 constructs the neural network submodule y with the affine field in parti, the nerve net with non local affine field
String bag module y 'i, as shown in formula (4) and formula (5):
Wherein, xiRepresent the feature vector of the time-space domain characteristic pattern of current layer network；yiWith y 'iRespectively represent next layer of net
The feature vector of the time-space domain characteristic pattern of the part of network and non local affine field；A(xi,xj) it is parent between calculating position i and j
With the binary transformation matrix of degree；g(xi) it is to calculate xjFeature insertion unitary transforming function transformation function, by convolution kernel be 1 or 1 × 1 volume
Lamination is realized；ZiIt (X) is normalization factor, Ω enumerates all feature locations, δiFor local domain.
The feature that local and non local affine field neural network submodule extracts is had the right to be superimposed, as shown in formula (6):
O=wonon-local+olocalFormula (6)
Wherein, O is superimposed characteristic pattern；onon-localAnd olocalFor same layer part and non local affine field nerve net
The output of string bag module；W is linear transformation function, by convolution kernel be 1 or 1 × 1 convolutional layer realize, for measure it is non local at
Significance level of the split-phase to local part.
Obtained characteristic pattern is subjected to batch normalization and reduces feature drift, introduces non-linear unit, then carry out down-sampling drop
The resolution ratio of low characteristic pattern.
Step S32 is led using the M1 parts and non local affine field neural network submodule calculating position i and part
Domain δiThe affinity of all possible positions in affinity and i and Ω between interior neighbours, M1 are the nature more than or equal to 1
Number.
Step S33, will be by the characteristic pattern and first of M1 part and non local affine field neural network submodule reasoning
Time-space domain characteristic pattern is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure.
The localized network prototype of the present embodiment is three convolutional neural networks, affinity matrix A (xi,xj)=1, g (xi) letter
Number is linear transformation function.Localized network module is as shown in figure 5, include time part plug-in unit (tLocal), space part plug-in unit
(sLocal) and 3 plug-in units of space-time part plug-in unit (stLocal), the convolution kernel size of three plug-in units are respectively k × 1,1 × k, k
×k.Similarly, non-local network also includes 3 plug-in units, respectively, time non local plug-in unit (tNon-Local), the non-office in space
Portion's plug-in unit (sNon-Local) and the non local plug-in unit of space-time (stNon-Local)；Wherein, the non local plug-in unit of two-dimensional space-time
(stNon-Local) mode is specifically completed as shown in figure 4, in figureψ, g, w are the convolutional layer that different core is 1 × 1,
ψ completes the function that affinity calculates, and g completes the function of linear transformation, and w measures the relative importance of non local ingredient；One-dimensional
Similar completion can be used in the non local plug-in unit of time non local plug-in unit (tNon-Local) and one-dimensional space (sNon-Local)
Mode.It is combined and office as shown in Figure 6 can be obtained by 3 plug-in units of localized network module and 3 plug-in units of non-local network's module
Portion and non local synchronization module (SLnL), corresponding affine field figure are as shown in Figure 7.
After M1 part carries out time-space domain reasoning with non local synchronous time-space network module, the parent of local submodule
Constantly increase with field, characteristic pattern resolution ratio constantly reduces, and semantic information has been obtained to be extracted well.Next it only needs to adopt
The excavation of high-rise spatiotemporal mode feature is carried out with local space time's network module.
Step S4 carries out high-rise local reasoning to the third time-space domain behavioural characteristic figure, obtains the 4th time-space domain behavior
Characteristic pattern, method are as follows:
The affine field nerve submodule in part constructed using M2 calculate the third time-space domain behavioural characteristic figure position i with
Local domain δiAffinity between interior neighbours, M2 are the natural number more than or equal to 1；When characteristic pattern after reasoning is the 4th
Airspace behavioural characteristic figure.
Using M1 part and non local synchronous time-space network module and the M2 affine field nerve submodule in part, C
× T × N is dimension signal, and the input for representing network is the three-dimensional tensor being made of channel C, tri- dimensions of time T and space N, C
× TN, TN × TN represents dimension as the two-dimensional matrix of C × TN, TN × TN, and the value of C, T, N is not identical in each submodule.
The 4th time-space domain behavioural characteristic figure global pool is obtained feature vector f by step S5p。
The behavioural characteristic extracting method based on space-time frequency domain blended learning of second embodiment of the invention, comprising:
Velocity information is obtained to the original difference on time dimension of the video behavior sequence based on skeleton, construction includes position
With the behavior sequence of speed.
The described in any item step S1- steps of claim 1-5 are used to the behavior sequence channel of position and speed respectively
S5 is handled, and the feature vector f of corresponding speed is obtainedpWith the feature vector f of corresponding positionv。
Described eigenvector is spliced to obtain splicing feature vector fc, the behavioural characteristic vector of extraction is velocity characteristic vector
fp, position feature vector fvWith splicing feature vector fc。
Behavioural characteristic extracting method in order to further illustrate the present invention based on space-time frequency domain blended learning, below with reference to spy
Application of the vector in terms of behavior classification is levied, the present invention is described further:
By described eigenvector fp、fvAnd fcBy the speed in virtual multitask network, position, splicing feature branch, obtain
Belong to the prediction probability p of each classification to behaviorp、pvAnd pc.Training stage utilizes prediction probability and true behavior classification, meter
Calculate the loss L of three respective predictions of branchp、LvAnd Lc.The present embodiment is calculated using cross entropy loss function, such as formula (7) institute
Show:
Wherein, b is the true one-hot class label of behavior, NCFor total behavior class number.
Shown in the total losses of multitask network such as formula (8):
L=λpLp+λvLv+λcLcFormula (8)
Wherein, λp、λvAnd λcFor three hyper parameters, the weight of each information channel is controlled.Optimized using total loss entire
Network is until being optimal.
Prediction probability p of (application) stage of test according only to splicing tunnelcClassification results are obtained, i.e., directly take pcIn have
The classification of maximum predicted probability is as the behavior classification results exported to the video behavior.
Person of ordinary skill in the field can be understood that, for convenience and simplicity of description, foregoing description
The specific works mistake of the step S1- step S5 of the behavioural characteristic extracting method based on space-time frequency domain blended learning of second embodiment
Journey and related explanation, can be with reference to the behavioural characteristic extracting method step based on space-time frequency domain blended learning of aforementioned first embodiment
Rapid corresponding process, details are not described herein.
The behavioural characteristic extraction system based on space-time frequency domain blended learning of the third embodiment of the present invention, including video sequence
Column obtain module, adaptive transformation module, frequency-selecting module, it is local with non local synchronous reasoning module, high-rise local reasoning
Module, global pool module, splicing module, multitask network module, output module；
The video sequence obtains module, is configured to obtain the video behavior sequence based on skeleton, as original video row
For sequence；
The adaptive transformation module is configured in time-space domain in such a way that augmentation optimizes, extracts the first time-space domain row
It is characterized figure；
The frequency-selecting module is configured to the first time-space domain behavioural characteristic figure feeding frequency domain noticing that network carries out frequency
Selection, transforms to time-space domain and first time-space domain behavioural characteristic figure phase in a manner of residual error for the frequency domain behavioural characteristic figure of acquisition
Add, obtains the second time-space domain behavioural characteristic figure；
The part and non local synchronous reasoning module are configured to synchronous to the second time-space domain behavioural characteristic figure carry out part
With non local reasoning, and it is added to obtain third time-space domain behavioural characteristic in a manner of residual error with the first time-space domain behavioural characteristic figure
Figure；
The high-rise local reasoning module is configured to carry out third time-space domain behavioural characteristic figure high-rise local reasoning, obtain
To the 4th time-space domain behavioural characteristic figure；
The global pool module is configured to the 4th time-space domain behavioural characteristic figure group global pool obtaining corresponding row
For feature vector；
The splicing module is configured to splice multi-channel feature, obtains splicing feature vector accordingly；
The output module is configured to the behavioural characteristic vector that will be extracted output.
It should be noted that the behavioural characteristic extraction system provided by the above embodiment based on space-time frequency domain blended learning,
Only the example of the division of the above functional modules, in practical applications, it can according to need and divide above-mentioned function
With being completed by different functional modules, i.e., by the embodiment of the present invention module or step decompose or combine again, for example,
The module of above-described embodiment can be merged into a module, can also be further split into multiple submodule, to complete above retouch
The all or part of function of stating.For module involved in the embodiment of the present invention, the title of step, it is only for distinguish each
A module or step, are not intended as inappropriate limitation of the present invention.
A kind of storage device of 4th example of the invention, wherein being stored with a plurality of program, described program is suitable for by handling
Device is loaded and is executed to realize the above-mentioned behavioural characteristic extracting method based on space-time frequency domain blended learning.
A kind of processing unit of 5th example of the invention, including processor, storage device；The processor, suitable for holding
Each program of row；The storage device is suitable for storing a plurality of program；Described program is suitable for being loaded by processor and being executed to realize
The above-mentioned behavioural characteristic extracting method based on space-time frequency domain blended learning.
Person of ordinary skill in the field can be understood that, for convenience and simplicity of description, foregoing description
The specific work process and related explanation of storage device, processing unit, can refer to corresponding processes in the foregoing method embodiment,
Details are not described herein
Those skilled in the art should be able to recognize that, mould described in conjunction with the examples disclosed in the embodiments of the present disclosure
Block, method and step, can be realized with electronic hardware, computer software, or a combination of the two, software module, method and step pair
The program answered can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electric erasable and can compile
Any other form of storage well known in journey ROM, register, hard disk, moveable magnetic disc, CD-ROM or technical field is situated between
In matter.In order to clearly demonstrate the interchangeability of electronic hardware and software, in the above description according to function generally
Describe each exemplary composition and step.These functions are executed actually with electronic hardware or software mode, depend on technology
The specific application and design constraint of scheme.Those skilled in the art can carry out using distinct methods each specific application
Realize described function, but such implementation should not be considered as beyond the scope of the present invention.
Term " space-time frequency domain " is " time-space domain " and " frequency domain ", and " time-space domain " is description mathematical function or physical signal to pure
Time, pure space or when space relationship a kind of coordinate system, " frequency domain " description signal used in characteristic in terms of frequency
A kind of coordinate system.
Term " first ", " second " etc. are to be used to distinguish similar objects, rather than be used to describe or indicate specific suitable
Sequence or precedence.
Term " includes " or any other like term are intended to cover non-exclusive inclusion, so that including a system
Process, method, article or equipment/device of column element not only includes those elements, but also including being not explicitly listed
Other elements, or further include the intrinsic element of these process, method, article or equipment/devices.
So far, it has been combined preferred embodiment shown in the drawings and describes technical solution of the present invention, still, this field
Technical staff is it is easily understood that protection scope of the present invention is expressly not limited to these specific embodiments.Without departing from this
Under the premise of the principle of invention, those skilled in the art can make equivalent change or replacement to the relevant technologies feature, these
Technical solution after change or replacement will fall within the scope of protection of the present invention.
Claims (9)
1. a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning characterized by comprising
Step S1 obtains the video behavior sequence based on skeleton, as original video behavior sequence, carries out time-space domain self-adaptive change
It changes, obtains the first time-space domain behavioural characteristic figure；
The first time-space domain behavioural characteristic figure is sent into inversion after frequency domain carries out frequency selection and gains time-space domain by step S2, with
The first time-space domain behavioural characteristic figure is added in a manner of residual error, obtains the second time-space domain behavioural characteristic figure；
Step S3, it is synchronous to the second time-space domain behavioural characteristic figure to carry out part and non local reasoning, and when with described first
Airspace behavioural characteristic figure is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure；
Step S4 carries out high-rise local reasoning to the third time-space domain behavioural characteristic figure, obtains the 4th time-space domain behavioural characteristic
Figure；
The 4th time-space domain behavioural characteristic figure global pool is obtained behavioural characteristic vector by step S5.
2. the behavioural characteristic extracting method according to claim 1 based on space-time frequency domain blended learning, which is characterized in that step
In rapid S1 " time-space domain self-adaptive transformation ", it the steps include:
Step S11 carries out the original video behavior sequence using convolutional network or fully-connected network under K oblique coordinates system
The adaptive augmentation of coordinate system obtains the augmentation video behavior sequence under K coordinate system, and K is hyper parameter；
Step S12 carries out joint number and pass to the skeleton in the augmentation video behavior sequence using multilayer fully-connected network
Section, which puts in order, to be converted, and is obtained the characteristic pattern of the augmentation optimization video behavior sequence comprising structural information, is the first space-time
Domain behavioural characteristic figure.
3. the behavioural characteristic extracting method according to claim 1 based on space-time frequency domain blended learning, which is characterized in that step
" the first time-space domain behavioural characteristic figure is sent into inversion after frequency domain carries out frequency selection in rapid S2 and gains time-space domain, with the first space-time
Domain behavioural characteristic figure is added in a manner of residual error ", method are as follows:
The characteristic pattern in each channel is transformed to frequency domain using two dimension discrete fourier transform by step S21 respectively, includes sinusoidal frequency
Characteristic of field figure and cosine frequency domain character figure；
Step S22, respectively by the sinusoidal frequency domain character figure and cosine frequency domain character figure by paying attention to network, learn out it is sinusoidal at
Divide attention weight and cosinusoidal component attention weight；
The attention network, including a channel average layer, two full articulamentums, a softmax function and a channel are multiple
Preparative layer；
Step S23 carries out dot product with the sine component attention weight learnt and sinusoidal frequency domain character figure, and cosinusoidal component pays attention to
Power weight and cosine frequency domain character figure carry out dot product, sine and cosine frequency domain character figure after obtaining frequency selection；
Sinusoidal and cosine frequency domain character figure is transformed to time-space domain using two-dimensional discrete inverse fourier transform, with residual error by step S24
Mode be added with the first time-space domain behavioural characteristic figure, obtain the second time-space domain behavioural characteristic figure.
4. the feature extraction recognition methods according to claim 1 based on space-time frequency domain blended learning, which is characterized in that step
In rapid S3 " synchronous to the second time-space domain behavioural characteristic figure to carry out part and non local reasoning ", it the steps include:
Step S31 constructs the neural network submodule y with the affine field in parti, the nerve net string bag with non local affine field
Module y 'i:
Wherein, xiRepresent the feature vector of the time-space domain characteristic pattern of current layer network；yiWith y 'iRespectively represent next layer network
The feature vector of the time-space domain characteristic pattern of local and non local affine field；A(xi,xj) it is affinity between calculating position i and j
Binary transformation matrix；g(xi) it is to calculate xjFeature insertion unitary transforming function transformation function, by convolution kernel be 1 or 1 × 1 convolutional layer
It realizes；ZiIt (X) is normalization factor, Ω enumerates all feature locations, δiFor local domain；
The feature that local and non local affine field neural network submodule extracts is had the right to be superimposed and obtains characteristic pattern, and to the spy
Sign figure carries out batch normalization reduction feature drift, introduces non-linear unit, then carry out down-sampling to reduce the resolution ratio of characteristic pattern；
Step S32, using the M1 parts and non local affine field neural network submodule calculating position i and local domain δi
The affinity of all possible positions in affinity and i and Ω between interior neighbours, M1 are the natural number more than or equal to 1；
Step S33, will be by the characteristic pattern and the first space-time of M1 part and non local affine field neural network submodule reasoning
Characteristic of field figure is added in a manner of residual error, obtains third time-space domain behavioural characteristic figure.
5. the behavioural characteristic extracting method according to claim 4 based on space-time frequency domain blended learning, which is characterized in that step
" high-rise local reasoning is carried out to third time-space domain behavioural characteristic figure " in rapid S4, method are as follows:
The part constructed using M2 is affine, and nerve submodule in field calculates the third time-space domain behavioural characteristic figure group position i and office
Portion field δiAffinity between interior neighbours, M2 are the natural number more than or equal to 1；Characteristic pattern after reasoning is the 4th space-time
Domain behavioural characteristic figure.
6. a kind of behavioural characteristic extracting method based on space-time frequency domain blended learning characterized by comprising
Velocity information is obtained to the original difference on time dimension of the video behavior sequence based on skeleton, construction includes position and speed
The behavior sequence of degree；
Respectively to the behavior sequence channel of position and speed using the described in any item step S1- step S5 of claim 1-5 into
Row processing, obtains the feature vector of corresponding speed and the feature vector of corresponding position；
Described eigenvector is spliced to obtain splicing feature vector, the behavioural characteristic vector of extraction is velocity characteristic vector, position
Feature vector and splicing feature vector.
7. a kind of behavioural characteristic extraction system based on space-time frequency domain blended learning, which is characterized in that obtained including video sequence
Module, adaptive transformation module, frequency-selecting module, it is local with non local synchronous reasoning module, it is high-rise local reasoning module, complete
Office's pond module, splicing module, output module；
The video sequence obtains module, is configured to obtain the video behavior sequence based on skeleton, as original video behavior sequence
Column；
The adaptive transformation module is configured in time-space domain in such a way that augmentation optimizes, it is special to extract the first time-space domain behavior
Sign figure；
The frequency-selecting module is configured to the first time-space domain behavioural characteristic figure feeding frequency domain noticing that network carries out frequency choosing
It selects, the frequency domain behavioural characteristic figure of acquisition is transformed into time-space domain and is added with the first time-space domain behavioural characteristic figure, the second space-time is obtained
Domain behavioural characteristic figure；
The part and non local synchronous reasoning module are configured to the part and non-of carrying out synchronous to the second time-space domain behavioural characteristic figure
Local reasoning, and be added to obtain third time-space domain behavioural characteristic figure in a manner of residual error with the first time-space domain behavioural characteristic figure；
The high-rise local reasoning module is configured to carry out third time-space domain behavioural characteristic figure high-rise local reasoning, obtains the
Four time-space domain behavioural characteristic figures；
The global pool module is configured to the 4th time-space domain behavioural characteristic figure group global pool it is special to obtain corresponding behavior
Levy vector；
The splicing module is configured to splice multi-channel feature, obtains splicing feature vector accordingly；
The output module is configured to the behavioural characteristic vector that will be extracted output.
8. a kind of storage device, wherein being stored with a plurality of program, which is characterized in that described program is suitable for being loaded and being held by processor
Row is to realize the behavioural characteristic extracting method described in any one of claims 1-6 based on space-time frequency domain blended learning.
9. a kind of processing unit, including
Processor is adapted for carrying out each program；And
Storage device is suitable for storing a plurality of program；
It is characterized in that, described program is suitable for being loaded by processor and being executed to realize:
Behavioural characteristic extracting method described in any one of claims 1-6 based on space-time frequency domain blended learning.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811494799.9A CN109711277B (en) | 2018-12-07 | 2018-12-07 | Behavior feature extraction method, system and device based on time-space frequency domain hybrid learning |
| PCT/CN2019/083357 WO2020113886A1 (en) | 2018-12-07 | 2019-04-19 | Behavior feature extraction method, system and apparatus based on time-space/frequency domain hybrid learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811494799.9A CN109711277B (en) | 2018-12-07 | 2018-12-07 | Behavior feature extraction method, system and device based on time-space frequency domain hybrid learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109711277A true CN109711277A (en) | 2019-05-03 |
| CN109711277B CN109711277B (en) | 2020-10-27 |
Family
ID=66254092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811494799.9A Active CN109711277B (en) | 2018-12-07 | 2018-12-07 | Behavior feature extraction method, system and device based on time-space frequency domain hybrid learning |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109711277B (en) |
| WO (1) | WO2020113886A1 (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222653A (en) * | 2019-06-11 | 2019-09-10 | 中国矿业大学(北京) | A kind of skeleton data Activity recognition method based on figure convolutional neural networks |
| CN110287836A (en) * | 2019-06-14 | 2019-09-27 | 北京迈格威科技有限公司 | Image classification method, device, computer equipment and storage medium |
| CN110378208A (en) * | 2019-06-11 | 2019-10-25 | 杭州电子科技大学 | A kind of Activity recognition method based on depth residual error network |
| CN110516599A (en) * | 2019-08-27 | 2019-11-29 | 中国科学院自动化研究所 | Group behavior identification model and its training method based on gradual relational learning |
| CN110826462A (en) * | 2019-10-31 | 2020-02-21 | 上海海事大学 | Human body behavior identification method of non-local double-current convolutional neural network model |
| CN115100740A (en) * | 2022-06-15 | 2022-09-23 | 东莞理工学院 | Human body action recognition and intention understanding method, terminal device and storage medium |
| JP2022542676A (en) * | 2019-08-27 | 2022-10-06 | エヌイーシー ラボラトリーズ アメリカ インク | Shuffle, Attend, and Adapt: Video Domain Adaptation with Clip Order Prediction and Clip Attention Adjustment |
| CN117576467A (en) * | 2023-11-22 | 2024-02-20 | 安徽大学 | Crop disease image identification method integrating frequency domain and spatial domain information |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111260774B (en) * | 2020-01-20 | 2023-06-23 | 北京百度网讯科技有限公司 | Method and device for generating 3D joint point regression model |
| CN111815604B (en) * | 2020-07-08 | 2023-07-28 | 讯飞智元信息科技有限公司 | Blast furnace tuyere monitoring method and device, electronic equipment and storage medium |
| CN112653899B (en) * | 2020-12-18 | 2022-07-12 | 北京工业大学 | Network live broadcast video feature extraction method based on joint attention ResNeSt under complex scene |
| CN113269218B (en) * | 2020-12-30 | 2023-06-09 | 威创集团股份有限公司 | Video classification method based on improved VLAD algorithm |
| CN114913565B (en) * | 2021-01-28 | 2023-11-17 | 腾讯科技（深圳）有限公司 | Face image detection method, model training method, device and storage medium |
| CN113516028B (en) * | 2021-04-28 | 2024-01-19 | 南通大学 | Human body abnormal behavior identification method and system based on mixed attention mechanism |
| CN113468954B (en) * | 2021-05-20 | 2023-04-18 | 西安电子科技大学 | Face counterfeiting detection method based on local area features under multiple channels |
| CN113177528B (en) * | 2021-05-27 | 2024-05-03 | 南京昊烽信息科技有限公司 | License plate recognition method and system based on multi-task learning strategy training network model |
| CN113408448A (en) * | 2021-06-25 | 2021-09-17 | 之江实验室 | Method and device for extracting local features of three-dimensional space-time object and identifying object |
| CN114039871B (en) * | 2021-10-25 | 2022-11-29 | 中山大学 | Method, system, device and medium for cellular traffic prediction |
| CN115375980B (en) * | 2022-06-30 | 2023-05-09 | 杭州电子科技大学 | Digital image certification system and certification method based on blockchain |
| CN117176270B (en) * | 2023-09-05 | 2024-03-19 | 浙江畅能数智科技有限公司 | Indoor antenna with signal monitoring function and monitoring method thereof |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8968091B2 (en) * | 2010-09-07 | 2015-03-03 | Microsoft Technology Licensing, Llc | Scalable real-time motion recognition |
| US20160042227A1 (en) * | 2014-08-06 | 2016-02-11 | BAE Systems Information and Electronic Systems Integraton Inc. | System and method for determining view invariant spatial-temporal descriptors for motion detection and analysis |
| CN106056135A (en) * | 2016-05-20 | 2016-10-26 | 北京九艺同兴科技有限公司 | Human body motion classification method based on compression perception |
| US20170228587A1 (en) * | 2016-02-05 | 2017-08-10 | University Of Central Florida Research Foundation, Inc. | System and method for human pose estimation in unconstrained video |
| CN107330362A (en) * | 2017-05-25 | 2017-11-07 | 北京大学 | A kind of video classification methods based on space-time notice |
| CN107680119A (en) * | 2017-09-05 | 2018-02-09 | 燕山大学 | A kind of track algorithm based on space-time context fusion multiple features and scale filter |
| CN108022254A (en) * | 2017-11-09 | 2018-05-11 | 华南理工大学 | A kind of space-time contextual target tracking based on sign point auxiliary |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107292247A (en) * | 2017-06-05 | 2017-10-24 | 浙江理工大学 | A kind of Human bodys' response method and device based on residual error network |
| CN108021889A (en) * | 2017-12-05 | 2018-05-11 | 重庆邮电大学 | A kind of binary channels infrared behavior recognition methods based on posture shape and movable information |
| CN108921087A (en) * | 2018-06-29 | 2018-11-30 | 国家计算机网络与信息安全管理中心 | video understanding method |
-
2018
- 2018-12-07 CN CN201811494799.9A patent/CN109711277B/en active Active
-
2019
- 2019-04-19 WO PCT/CN2019/083357 patent/WO2020113886A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8968091B2 (en) * | 2010-09-07 | 2015-03-03 | Microsoft Technology Licensing, Llc | Scalable real-time motion recognition |
| US20160042227A1 (en) * | 2014-08-06 | 2016-02-11 | BAE Systems Information and Electronic Systems Integraton Inc. | System and method for determining view invariant spatial-temporal descriptors for motion detection and analysis |
| US20170228587A1 (en) * | 2016-02-05 | 2017-08-10 | University Of Central Florida Research Foundation, Inc. | System and method for human pose estimation in unconstrained video |
| CN106056135A (en) * | 2016-05-20 | 2016-10-26 | 北京九艺同兴科技有限公司 | Human body motion classification method based on compression perception |
| CN107330362A (en) * | 2017-05-25 | 2017-11-07 | 北京大学 | A kind of video classification methods based on space-time notice |
| CN107680119A (en) * | 2017-09-05 | 2018-02-09 | 燕山大学 | A kind of track algorithm based on space-time context fusion multiple features and scale filter |
| CN108022254A (en) * | 2017-11-09 | 2018-05-11 | 华南理工大学 | A kind of space-time contextual target tracking based on sign point auxiliary |
Non-Patent Citations (2)
| Title |
|---|
| CYRILLE BEAUDRY 等: "Action recognition in videos using frequency analysis of critical point trajectories", 《2014 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP)》 * |
| YONG DU 等: "Skeleton Based Action Recognition with Convolutional Neural Network", 《2015 3RD IAPR ASIAN CONFERENCE ON PATTERN RECOGNITION (ACPR)》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222653B (en) * | 2019-06-11 | 2020-06-16 | 中国矿业大学(北京) | Skeleton data behavior identification method based on graph convolution neural network |
| CN110378208A (en) * | 2019-06-11 | 2019-10-25 | 杭州电子科技大学 | A kind of Activity recognition method based on depth residual error network |
| CN110222653A (en) * | 2019-06-11 | 2019-09-10 | 中国矿业大学(北京) | A kind of skeleton data Activity recognition method based on figure convolutional neural networks |
| CN110287836A (en) * | 2019-06-14 | 2019-09-27 | 北京迈格威科技有限公司 | Image classification method, device, computer equipment and storage medium |
| CN110287836B (en) * | 2019-06-14 | 2021-10-15 | 北京迈格威科技有限公司 | Image classification method and device, computer equipment and storage medium |
| CN110516599A (en) * | 2019-08-27 | 2019-11-29 | 中国科学院自动化研究所 | Group behavior identification model and its training method based on gradual relational learning |
| JP2022542676A (en) * | 2019-08-27 | 2022-10-06 | エヌイーシー ラボラトリーズ アメリカ インク | Shuffle, Attend, and Adapt: Video Domain Adaptation with Clip Order Prediction and Clip Attention Adjustment |
| JP7286003B2 (en) | 2019-08-27 | 2023-06-02 | エヌイーシー ラボラトリーズ アメリカ インク | Shuffle, Attend, and Adapt: Video Domain Adaptation with Clip Order Prediction and Clip Attention Adjustment |
| CN110826462A (en) * | 2019-10-31 | 2020-02-21 | 上海海事大学 | Human body behavior identification method of non-local double-current convolutional neural network model |
| CN115100740A (en) * | 2022-06-15 | 2022-09-23 | 东莞理工学院 | Human body action recognition and intention understanding method, terminal device and storage medium |
| CN115100740B (en) * | 2022-06-15 | 2024-04-05 | 东莞理工学院 | Human motion recognition and intention understanding method, terminal equipment and storage medium |
| CN117576467A (en) * | 2023-11-22 | 2024-02-20 | 安徽大学 | Crop disease image identification method integrating frequency domain and spatial domain information |
| CN117576467B (en) * | 2023-11-22 | 2024-04-26 | 安徽大学 | Crop disease image identification method integrating frequency domain and spatial domain information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109711277B (en) | 2020-10-27 |
| WO2020113886A1 (en) | 2020-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109711277A (en) | Behavioural characteristic extracting method, system, device based on space-time frequency domain blended learning | |
| Hazirbas et al. | Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture | |
| Qiu et al. | Local climate zone-based urban land cover classification from multi-seasonal Sentinel-2 images with a recurrent residual network | |
| CN109614981A (en) | The Power System Intelligent fault detection method and system of convolutional neural networks based on Spearman rank correlation | |
| CN107037881A (en) | The interactive demonstration method and system of GIS and BIM augmented realities in piping lane, subway work | |
| CN109709603A (en) | Seismic horizon identification and method for tracing, system | |
| CN109903117A (en) | A kind of knowledge mapping processing method and processing device for commercial product recommending | |
| CN113870422B (en) | Point cloud reconstruction method, device, equipment and medium | |
| CN113095254B (en) | Method and system for positioning key points of human body part | |
| CN108830421A (en) | The gas distribution prediction method and device of tight sandstone reservoir | |
| CN114842351A (en) | Remote sensing image semantic change detection method based on twin transforms | |
| Tang et al. | Wnet: W-shaped hierarchical network for remote sensing image change detection | |
| Liu et al. | PISEP 2: pseudo-image sequence evolution-based 3D pose prediction | |
| CN112115744A (en) | Point cloud data processing method and device, computer storage medium and electronic equipment | |
| Tu et al. | Propagate and pair: A single-pass approach to critical point pairing in reeb graphs | |
| Barthakur et al. | Deep learning based semantic segmentation applied to satellite image | |
| CN112529057A (en) | Graph similarity calculation method and device based on graph convolution network | |
| Malekijoo et al. | Convolution-deconvolution architecture with the pyramid pooling module for semantic segmentation | |
| Zhang et al. | Multiscale depthwise separable convolution based network for high-resolution image segmentation | |
| Xin et al. | Digitalization system of ancient architecture decoration art based on neural network and image features | |
| Turay et al. | SSP Framework: A New Approach to Designing Lightweight Convolutional Neural Networks | |
| Sagar et al. | Morphing of grayscale DEMs via morphological interpolations | |
| Laban et al. | Sparse Pixel Training of Convolutional Neural Networks for Land Cover Classification | |
| Babaali et al. | A new approach for road extraction using data augmentation and semantic segmentation | |
| Kekre et al. | Discrete Sine Transform Sectorization for Feature Vector Generation in CBIR |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |