CN106780568B - A kind of video target tracking method based on the irregular piecemeal LBP of compression - Google Patents
A kind of video target tracking method based on the irregular piecemeal LBP of compression Download PDFInfo
- Publication number
- CN106780568B CN106780568B CN201611185481.3A CN201611185481A CN106780568B CN 106780568 B CN106780568 B CN 106780568B CN 201611185481 A CN201611185481 A CN 201611185481A CN 106780568 B CN106780568 B CN 106780568B
- Authority
- CN
- China
- Prior art keywords
- lbp
- target
- irb
- follows
- candidate target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F18/24155—Bayesian classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
A kind of video target tracking method based on the irregular piecemeal LBP of compression, this method uses the irregular piecemeal LBP feature vector after compression sampling to state tracked target or candidate target, candidate target is scanned for using particle filter frame, whether target following result differentiates the compressive features vector using Naive Bayes Classifier to candidate target.This method not only promotes the processing speed of video frequency object tracking, while can keep accurate tracking effect under severe lighting change and the Various Complexes scene such as postural change, visual angle rotation and movement suddenly, background clutter, similar purpose interference.
Description
Technical field
It is the present invention relates to motion target tracking field in sequence of frames of video, in particular to a kind of based on the irregular piecemeal of compression
The video target tracking method of LBP.
Background technique
Video frequency object tracking refer to analyzed from video sequence by target signature (such as color, texture, shape) it is specific
The kinematic parameter of target and track (such as position, size, shape, speed, acceleration), are the cores of computer vision system
One of task has a wide range of applications in various fields such as intelligent video monitoring, human-computer interaction, medical diagnosis, robot navigations
Prospect.However, illumination variation, shade, block, move mutation, background clutter etc. various complex scene factors to video object with
Track technology brings great challenge, and accurate and quick video target tracking method is concerned.
Accuracy: situations such as target signature selection is most important to tracking accuracy, and textural characteristics are to illumination variation, shade
With well adapting to property, especially LBP (Local Binary Pattern) feature has become the textural characteristics being widely used
One of.
Rapidity: compressive sensing theory (Compressive Sensing, CS) can be with far less than conventional Nyquist
The linear projection value of sampling realizes the accurate or approximate reconstruction of signal.It can be by traditional higher-dimension target based on compressive sensing theory
The lossless dimensionality reduction of feature vector is to improve the speed of video frequency object tracking.
1. basic LBP feature and piecemeal LBP feature
(1) basic LBP feature
Document " Ojala T, Pietikainen M, Maenpaa T.Multiresolution gray-scale and
rotation invariant texture classification with local binary patterns[J].IEEE
Transactions on pattern analysis and machine intelligence,2002,24(7):971-
987. " propose basic LBP feature.The calculating of basic LBP feature is related to the region 3*3, and each region is a pixel, calculates
Method is as follows.
(2) MB-LBP feature
Document " Zhang L, Chu R, Xiang S, et al.Face detection based on multi-block
lbp representation[C]//International Conference on Biometrics.Springer Berlin
Heidelberg, 2007:11-18. " propose MB-LBP feature in basic LBP feature base.The calculating of MB-LBP feature is also
It is related to the region 3*3, each region is extended, and can be the block (Block) of multiple pixels composition, calculation method is as follows.
2. the method compressed based on compressed sensing to Haar-like feature
Document " Zhang K, Zhang L, Yang M H.Fast compressive tracking [J] .IEEE
Transactions on Pattern Analysis and Machine Intelligence,2014,36(10):2002-
2015. " propose FCT tracking, and this method states target using compression Haar-like feature.
(1) calculation method of Haar-like feature is compressed
FCT uses sparse random gaussian matrix R to compress the higher-dimension Haar-like Feature Compression for tracking target for low-dimensional and surveys
The process for measuring vector is y=Rx, whereinBe tracked target or candidate target higher-dimension Haar-like feature to
Amount,It is the low-dimensional compression measurement vector after conversion,It is the sparse random height as calculation matrix
This matrix, m × n indicate the dimension of calculation matrix.
The element of the position R matrix (i, j) can indicate are as follows:Wherein s
Indicate the sparse degree of calculation matrix.S=n/ (a log10(n)) ≈ n/4~n/2.4, wherein assuming n=106~1010, a=
0.4, therefore, every row nonzero element number random value is 2~4, m=100 in R.Each element y in Y-direction amountiIt is considered as R
I-th of row vector and higher-dimension Haar-like feature vector x inner product, shown in schematic process Fig. 1.
(2) differentiate the classifier and its update method of feature
The problem of whether being tracking target for a compression Haar-like feature vector, FCT method is using simple pattra leaves
This classifier is differentiated that the classifier is as follows.
Wherein, p (z=1) and p (z=0) respectively indicates a candidate target and meets positive sample distribution and meet negative sample point
The probability of cloth, p (z=1)=p (z=0)=0.5, z ∈ { 0,1 }.H (y) value that one candidate target is calculated is bigger, the time
Select the confidence level of target higher.Assuming that the formula conditional distribution p (yi| z=1) and p (yi| z=0) meet four parametersGaussian Profile:
The available q target positive sample around current optimal candidate region, that is, tracking result, and then q can be formed
A compression measures vector.Assuming that it is μ that the ith measurement element of q measurement vector, which meets parameter, here1And δ1Gaussian Profile, μ1
And δ1Calculation method it is as follows:
λ > 0 is renewal rate,WithUpdate method it is as follows:
For negative sample,WithUpdate method withWithUpdate method it is similar.
(3) from thick to thin --- the search strategy of candidate target
FCT using from thick to thin by the way of (coarse-to-fine) around current tracking result position to candidate target
It scans for, as shown in Figure 2.
In the coarse search stage, search radius γ is setc=25, step-size in search Δc=4, the rough position of target is found, thin
Search radius γ is arranged in search phasef=10, step-size in search Δf=1, the accurate location of target is found, the entire search phase is candidate
Target is about 436.
However the prior art has the following problems:
(1) shortcoming of MB-LBP feature
MB-LBP is related to the region 3*3, and 9 regions are having the same wide high, i.e. a MB-LBP feature correspondence image region
The width and height of (outer boundary in the region 3*3) are all 3 multiple.Therefore, mainly there is following two in the shortcomings that MB-LBP feature:
1. a MB-LBP image-region can not be related to the image-region of all possible scales；2. it is all to calculate a tracking target area
It is very time-consuming that possible MB-LBP feature forms higher-dimension MB-LBP feature vector.
(2) shortcoming of Haar-like feature vector is compressed
It is very time saving to compress the calculating of Haar-like feature vector, FCT is even several by tens using the compressed sensing method of sampling
The computational short cut of the Haar-like feature vector of million dimensions is the linear operation within 400 times (100*2~100*4).However,
More documents are it has proven convenient that in the video frequency object tracking application under scenes such as illumination variation, dimensional variation, target rotation, 3.
The performance of Haar-like feature is much not as good as MB-LBP feature.FCT method assumes Haar-like feature vector dimension n=106
~1010, 4. it is unable to the dimension of precise expression tracked target provincial characteristics vector.The degree of rarefication of calculation matrix R is excessively high, i.e., every row
The number of nonzero element is very few on vector, and the information for 5. compressing Haar-like feature vector is caused to be lost.
(3) shortcoming of search strategy from thick to thin
FCT scans for candidate target using search strategy from thick to thin.Plan is searched for compared to common particle filter
Slightly, the candidate target that 6. search strategy generates on a large amount of impossible directions from thick to thin will cause unnecessary calculating to open
Pin.
Summary of the invention
To solve the above-mentioned problems of the prior art, the purpose of the present invention is to provide one kind based on irregular point of compression
The video target tracking method of block LBP will state target signature based on a kind of textural characteristics for being known as " irregular piecemeal LBP ",
And lossless dimensionality reduction is carried out to this feature vector using compressive sensing theory, and then achieve the purpose that accurate and quickly track.
In order to achieve the above objectives, the technical solution of the present invention is as follows:
A kind of video target tracking method based on the irregular piecemeal LBP of compression,
Steps are as follows:
Step 1 selectes tracing area
Using frame each in the sequence of frames of video upper left corner as coordinate origin (1,1), wide high respectively W and H；In selected first frame
Rectangular area (the x of target to be tracked0,y0, w, h), i.e., top left co-ordinate is (x0,y0), wide high respectively w, h；
Step 2 initializes particle assembly
The target to be tracked duplication selected according to first frame generates k particle, forms initialization particle assemblyThe initial weight of each particle is 1/k, i.e.,
Step 3 initialization survey matrix
Calculation matrix is compressed used in compression sampling processIt is a sparse random gaussian matrix,
The dimension of m × n expression calculation matrix；
The number of nonzero element is in every row in RThe number of all nonzero elements is in RRemaining member
Element is all zero；The iRB-LBP feature of the corresponding specific position of each nonzero element value in R；After matrix R is generated, entire
The nonzero element value of R and its corresponding iRB-LBP feature locations are no longer changed during tracking, i.e., each subsequent time
Selecting the compressive features vector of target will be calculated according to the R generated at this time；
The columns n of R is iRB-LBP feature vectorDimension；W' and h' is enabled to indicate an iRB-LBP characteristic value
The width and height of corresponding region, w' value range are 3~w, and the value range of h' is 3~h, and the calculation method of n is as follows:
N=(w-2) × (h-2) (1)
The line number m of R, is compressive featuresDimension；K=10 is enabled, the calculation method of c=1/logn, m are as follows:
The position (i, j) element calculation formula of matrix R are as follows:
Wherein, s=n/lnn indicates the sparse degree of calculation matrix；
Step 4 initialized target classifier
Using Naive Bayes Classifier H (y), to candidate target in subsequent frame sequence, whether tracked target differentiates,
H (y) is defined as follows:
Wherein, y is the CiRB-LBP feature vector of tracked target or candidate target region, and p (v=1) and p (v=0) divide
Not Biao Shi y meet positive sample distribution and meet negative sample distribution probability, p (v=1)=p (v=0)=0.5, v ∈ { 0,1 }；It is false
If condition distribution p (yi| v=1) and p (yi| v=0) meet four parametersGaussian Profile, i.e.,
Step 5 updates object classifiers
If it is first frame, using the tracked target rectangular area of current selected as reference area, otherwise, with currently most
The rectangular area of good candidate target, that is, tracking result is as reference area；Benchmark region generates positive sample and negative sample, into
And the parameters of object classifiers are updated；
Step 6 inputs next video frame
The particle assembly of step 7 predicting candidate target
Predict that the candidate particle list for generating present frame closes according to the particle assembly of previous frame
Step 8 calculates the CiRB-LBP feature vector of all candidate targets
It enablesIndicate the CiRB-LBP feature vector of rectangular area, m × 1 indicates the dimension of CiRB-LBP feature vector
Number,Indicate the iRB-LBP feature vector of tracked target or candidate target rectangular area, n × 1 indicates that iRB-LBP is special
The dimension of vector is levied, n is possible iRB-LBP Characteristic Number in rectangular area, each element x in xj, j=1~n corresponding one
A iRB-LBP characteristic value；The calculation method of candidate target specific for one, CiRB-LBP feature vector y is as follows:
Each element in CiRB-LBP feature vectorIt is calculatingProcess
In, work as rij=0, then rijxj=0, therefore, work as rij=0, then xjIt can be omitted calculating, directly enable xj=0；So, it is only necessary to it counts
Calculate rijCorresponding x in the case where ≠ 0 (i=1~m, j=1~n)j, that is, calculate each nonzero element in R and correspond to iRB-LBP spy
It levies position and extracts corresponding characteristic element value xj, need to calculate in totalA iRB-LBP characteristic value, can be obtained one
The corresponding CiRB-LBP feature vector y of candidate target；
It is first that an iRB-LBP feature is corresponding for the iRB-LBP feature of a specific position in candidate target region
Region division be 3*3 i.e. 9 block (Block), i-th piece is expressed as bi, i=0~8, i-th piece of wide height is expressed as
Calculation method it is as follows:
Since the width and height of iRB-LBP characteristic area are not 3 multiple in most cases, 9 piecemeals
Wide height is not exactly the same as irregular piecemeal；Work as the multiple that w ' is 3,9 pieces of same size, otherwise, intermediate column width
It is different from two column of left and right；When h' is 3 multiple, 9 pieces of height is identical, and otherwise, center row height is different from two rows up and down；Work as w '
It is all 3 multiple with h', 9 pieces of width is high all identical, and iRB-LBP characteristic value is identical as MB-LBP characteristic value；
Enable i-th piece of biThe mean value of middle all pixels gray value is expressed asIRB-LBP characteristic value calculation formula is as follows:
Step 9 differentiates tracking result
Firstly, calculating each candidate target CiRB-LBP feature vector y's according to the calculation formula of H (y) in step 4
Evaluation of estimate；Since the weight of particle must be a positive value, and H (y) can just be born, in order to correctly to candidate target weight into
Row updates, and finds the smallest candidate target of score valueAccording to formula Hj(y)=Hj(y)-Hminj(y)
All H (y) are converted positive value by+ε, wherein ε > 1.0；
Evaluation of estimate regularization to k candidate target, i.e.,According to commenting after regularization
Value is updated the weight of each candidate target, i.e. wtj=wtj×α+Hj(y) × (1- α), α=0~1；To updated
Weight carries out regularization, i.e.,Candidate target of the regularization with weight limit is the tracking knot of present frame
Fruit；
According to weight wtjCandidate target, that is, particle is sorted from large to small, is successively adopted again according to the biggish particle of weight
Sample gives up the lower particle of weight；
If step 10 present frame is last frame, tracking terminates；Otherwise, step 5 repetitive cycling is gone to.
Further, in the step 1, the rectangular area (x of target to be tracked in first frame is manually or automatically selected0,
y0,w,h)。
Further, in the step 2, particle assembly is initializedIn
Further, in the step 4, the value of four parameters is initialized are as follows:I=1
~m.
Further, in the step 5, benchmark region generates positive sample and negative sample, and then to object classifiers
Parameters be updated that the specific method is as follows:
For positive sample, and reference area central point distance α=4 range in institute identical as reference area size
Some rectangular areas constitute positive sample set to be selected；Q is randomly choosed from positive sample set to be selected1Make=45 rectangular areas
For target positive sample；It can be calculated separately to obtain q according to step 21The corresponding CiRB-LBP in=45 positive sample rectangular areas is special
Levy vector；Assuming that q here1It is μ that the ith measurement element of=45 feature vectors, which meets parameter,1And δ1Gaussian Profile, meter
Calculation method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method it is as follows:
For negative sample, and reference area central point distance identical as reference area size is in β=8~30 ranges
Interior all rectangular areas constitute negative sample set to be selected；Q is randomly choosed from negative sample set to be selected0=50 rectangle regions
Domain is as target negative sample；It can be calculated separately to obtain q according to step 20The corresponding CiRB- in=50 negative sample rectangular areas
LBP feature vector；Assuming that q here0It is μ that the ith measurement element of=50 feature vectors, which meets parameter,0And δ0Gaussian Profile,
Its calculation method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method formula it is as follows:
Compared with the existing technology, the invention has the benefit that
It improves to obtain " irregular piecemeal LBP feature " (irregular Block the present invention is based on MB-LBP feature
Local Binary Pattern, iRB-LBP), and it is real to use for reference the method that compressed sensing samples Haar-like Feature Compression
The compression sampling for having showed iRB-LBP feature has obtained compression iRB-LBP feature (Compressed iRB-LBP, CiRB-LBP),
And then target is stated using CiRB-LBP feature, realize quickly and accurately tracking effect.And solves the prior art
Middle following problems:
1. a MB-LBP image-region can not be related to the image-region of all possible scales；
The width and height of one MB-LBP image-region are necessary for 3 multiple.It is different from MB-LBP image-region, this hair
It is bright to propose a kind of iRB-LBP feature, allow the width range of its image-region to be 3~W, altitude range is 3~H, wherein W and H
The width in respectively tracked target region is high.
2. calculating all possible MB-LBP feature in a tracking target area forms higher-dimension MB-LBP feature vector very
It is time-consuming；
MB-LBP feature vector needs to calculate all possible MB-LBP characteristic element.The CiRB-LBP proposed in the present invention
Feature vector, the iRB-LBP characteristic element of corresponding calculation matrix neutral element position are not necessarily to calculate, in total calculative iRB-
LBP Characteristic Number isIt is a.For example, only needing to calculate 64 iRB-LBP characteristic elements in embodiment 1.
3. the performance of Haar-like feature is much not as good as MB-LBP feature in FCT method；
Since Haar-like principle itself determines that Haar-like feature can not adapt to the feelings such as dimensional variation, target rotation
Condition, and MB-LBP feature rotates well adapting to property for dimensional variation, target.IRB-LBP feature of the invention is from MB-
LBP feature improves, and maintains the advantages such as its dimensional variation, target rotation.
4. FCT method is unable to the dimension of precise expression tracked target provincial characteristics vector；
FCT method assumes Haar-like feature vector dimension n=106~1010.The present invention by formula n=(w-2) ×
(h-2) dimension that can accurately calculate iRB-LBP feature vector, for specifically tracking target with accurate adaptability.
5. the information loss for compressing Haar-like feature vector in FCT method is larger；
The degree of rarefication of calculation matrix R is excessively high in FCT method, i.e., the number of nonzero element is up to 4 in every row vector.This
Invent in the calculation matrix that uses in every row vector nonzero element number forIt is a, it is special preferably to remain original iRB-LBP
Levy the information of vector.
6. the candidate target that search strategy generates on a large amount of impossible directions from thick to thin in FCT method will cause not
Necessary computing cost.
FCT method can not produce unnecessary computing cost on direction using strategy from thick to thin, a large amount of.This
Invention predicts candidate target using particle filter frame, considerably reduces the computing cost on unnecessary direction.
Detailed description of the invention
Fig. 1 is the calculation method schematic diagram for compressing Haar-like feature.
Fig. 2 be FCT using from thick to thin by the way of (coarse-to-fine) around current tracking result position to time
Target is selected to scan for schematic diagram.
Fig. 3 is that the present invention is based on the video target tracking method flow diagrams for compressing irregular piecemeal LBP.
Fig. 4 is iRB-LBP characteristic value calculating process schematic diagram.
Fig. 5 is that video sequence first frame tracing area chooses situation schematic diagram.
Fig. 6 be the embodiment of the present invention in, be with the corresponding iRB-LBP feature of first nonzero element of calculation matrix the first row
Example, corresponding position is (44,12,4,8), piecemeal situation (irregular 9 pieces, central block b in candidate target region0It is wide high
For 2*2), each piece of mean value, characteristic value calculated case schematic diagram.
Fig. 7 is video frequency object tracking result screenshot of the present invention.
Specific embodiment
Technical solution of the present invention is described in further detail with reference to the accompanying drawings and detailed description:
A kind of video target tracking method based on the irregular piecemeal LBP of compression,
This method uses the irregular piecemeal LBP feature vector after compression sampling to carry out tracked target or candidate target
Statement, scans for candidate target using particle filter frame, the compression using Naive Bayes Classifier to candidate target
Whether target following result is differentiated feature vector.This method not only promotes the processing speed of video frequency object tracking, simultaneously
In severe lighting change and the Various Complexes field such as postural change, visual angle rotation and movement suddenly, background clutter, similar purpose interference
Accurate tracking effect can be kept under scape.Below by " irregular piecemeal LBP (irregular Block Local Binary
Pattern) " referred to as " iRB-LBP ", " irregular piecemeal LBP (Compressed irregular Block will be compressed
Local Binary Pattern) " it is referred to as " CiRB-LBP ".
As shown in figure 3, steps are as follows:
Step 1. selectes tracing area
Using frame each in the sequence of frames of video upper left corner as coordinate origin (1,1), wide high respectively W and H.Manually or automatically select
Determine the rectangular area (x of target to be tracked in first frame0,y0, w, h), i.e., top left co-ordinate is (x0,y0), wide high respectively w, h；
Step 2. initializes particle assembly
The target to be tracked duplication selected according to first frame generates k particle, forms initialization particle assemblyWhereinThe initial weight of each particle is 1/k,
I.e.
Step 3. initialization survey matrix
Calculation matrix is compressed used in compression sampling processIt is a sparse random gaussian matrix,
The dimension of m × n expression calculation matrix；
The columns n and iRB-LBP feature vector of RDimension；W' and h' is enabled to indicate an iRB-LBP feature
It is worth the width and height of corresponding region, w' value range is 3~w, and the value range of h' is 3~h, and the calculation method of n is as follows:
N=(w-2) × (h-2) (1)
The line number m and compressive features of RDimension.K=10 is enabled, the calculation method of c=1/logn, m are as follows:
The position (i, j) element calculation formula of matrix R are as follows:
Wherein, s=n/ln n indicates the sparse degree of calculation matrix；
The number of nonzero element is in every row in RThe number of all nonzero elements is in RRemaining member
Element is all zero；The iRB-LBP feature of the corresponding specific position of each nonzero element value in R；After matrix R is generated, entire
The nonzero element value of R and its corresponding iRB-LBP feature locations are no longer changed during tracking, i.e., each subsequent time
Selecting the compressive features vector of target will be calculated according to the R generated at this time；
Step 4. initialized target classifier
Using Naive Bayes Classifier H (y), to candidate target in subsequent frame sequence, whether tracked target differentiates,
H (y) is defined as follows:
Wherein, y is the CiRB-LBP feature vector of tracked target or candidate target region, and p (v=1) and p (v=0) divide
Not Biao Shi y meet positive sample distribution and meet negative sample distribution probability, p (v=1)=p (v=0)=0.5, v ∈ { 0,1 }；It is false
If condition distribution p (yi| v=1) and p (yi| v=0) meet four parametersGaussian Profile, i.e.,
Initialize the value of four parameters are as follows:
Step 5. updates object classifiers
If it is first frame, using the tracked target rectangular area of current selected as reference area, otherwise, with currently most
The rectangular area of good candidate target, that is, tracking result is as reference area；Benchmark region generates positive sample and negative sample, into
And the parameters of object classifiers are updated, the specific method is as follows:
For positive sample, and reference area central point distance α=4 range in institute identical as reference area size
Some rectangular areas constitute positive sample set to be selected；Q is randomly choosed from positive sample set to be selected1Make=45 rectangular areas
For target positive sample；It can be calculated separately to obtain q according to step 21The corresponding CiRB-LBP in=45 positive sample rectangular areas is special
Levy vector；Assuming that q here1It is μ that the ith measurement element of=45 feature vectors, which meets parameter,1And δ1Gaussian Profile, meter
Calculation method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method it is as follows:
For negative sample, and reference area central point distance identical as reference area size is in β=8~30 ranges
Interior all rectangular areas constitute negative sample set to be selected；Q is randomly choosed from negative sample set to be selected0=50 rectangle regions
Domain is as target negative sample；It can be calculated separately to obtain q according to step 20The corresponding CiRB- in=50 negative sample rectangular areas
LBP feature vector；Assuming that q here0It is μ that the ith measurement element of=50 feature vectors, which meets parameter,0And δ0Gaussian Profile,
Its calculation method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method formula it is as follows:
Step 6. inputs next video frame
The particle assembly of step 7. predicting candidate target
Predict that the candidate particle list for generating present frame closes according to the particle assembly of previous frame
Step 8. calculates the CiRB-LBP feature vector of all candidate targets
It enablesIndicate the CiRB-LBP feature vector of rectangular area, m × 1 indicates the dimension of CiRB-LBP feature vector
Number,Indicate the iRB-LBP feature vector of tracked target or candidate target rectangular area, n × 1 indicates that iRB-LBP is special
The dimension of vector is levied, n is possible iRB-LBP Characteristic Number in rectangular area, each element x in xj, j=1~n corresponding one
A iRB-LBP characteristic value；The calculation method of candidate target specific for one, CiRB-LBP feature vector y is as follows:
Each element in CiRB-LBP feature vectorIt is calculatingProcess
In, work as rij=0, then rijxj=0, therefore, work as rij=0, then xjIt can be omitted calculating, directly enable xj=0；So, it is only necessary to it counts
Calculate rijCorresponding x in the case where ≠ 0 (i=1~m, j=1~n)j, that is, calculate each nonzero element in R and correspond to iRB-LBP spy
It levies position and extracts corresponding characteristic element value xj, need to calculate in totalA time can be obtained in a iRB-LBP characteristic value
Select the corresponding CiRB-LBP feature vector y of target；
It is first that an iRB-LBP feature is corresponding for the iRB-LBP feature of a specific position in candidate target region
Region division be 3*3 i.e. 9 block (Block), i-th piece is expressed as bi, i=0~8, i-th piece of wide height is expressed as
Calculation method it is as follows:
Since the width and height of iRB-LBP characteristic area are not 3 multiple in most cases, 9 piecemeals
Wide height is not exactly the same as irregular piecemeal；Work as the multiple that w ' is 3,9 pieces of same size, otherwise, intermediate column width
It is different from two column of left and right；When h' is 3 multiple, 9 pieces of height is identical, and otherwise, center row height is different from two rows up and down；Work as w '
It is all 3 multiple with h', 9 pieces of width is high all identical, and iRB-LBP characteristic value is identical as MB-LBP characteristic value；
Enable i-th piece of biThe mean value of middle all pixels gray value is expressed asIRB-LBP characteristic value calculation formula is as follows:
IRB-LBP characteristic value calculating process it is as shown in Figure 4；
Step 9. differentiates tracking result
Firstly, calculating commenting for each candidate target CiRB-LBP feature vector y according to the calculation formula of H (y) in step 4
Value；Since the weight of particle must be a positive value, and H (y) can just be born, in order to correctly carry out to candidate target weight
It updates, finds the smallest candidate target of score valueAccording to formula Hj(y)=Hj(y)-Hminj(y)+ε
Positive value is converted by all H (y), wherein ε > 1.0；
Evaluation of estimate regularization to k candidate target, i.e.,According to commenting after regularization
Value is updated the weight of each candidate target, i.e. wtj=wtj×α+Hj(y) × (1- α), α=0~1；To updated
Weight carries out regularization, i.e.,Candidate target of the regularization with weight limit is the tracking knot of present frame
Fruit；
According to weight wtjCandidate target, that is, particle is sorted from large to small, is successively adopted again according to the biggish particle of weight
Sample gives up the lower particle of weight；
If step 10. present frame is last frame, tracking terminates；Otherwise, step 5 repetitive cycling is gone to.
Embodiment 1:
According to technical solution of the present invention, the target in one section of sequence of frames of video is tracked below, scene feature is mesh
Scale variation, illumination variation, apparent variation.
Step 1. selectes tracing area
The wide high respectively W=320 and H=240 of video frame.First frame target to be tracked rectangular area (121,58,51,
50), i.e., top left co-ordinate is (121,58), wide high by respectively 51,50, and it is as shown in Figure 5 to choose result.
Step 2. initializes particle assembly
The target to be tracked duplication selected according to first frame generates k=100 particle, forms initialization particle assemblyWhereinBefore following table lists
The case where 10 particles.
Step 3. initialization survey matrix
Compress calculation matrixLine number beIt is also compressive featuresDimension；Columns is n=(51-2) × (50-2)=2352 and iRB-LBP feature vectorDimension.
Every row nonzero element number is in matrix RIn R onlyA element is non-
Zero, nonzero element value isOrPositive and negative probability is identical, remaining element is all zero.
Following table lists each nonzero element value and the corresponding iRB-LBP of each nonzero element value in calculation matrix R
Position where feature, for example, the corresponding iRB-LBP characteristic area of first nonzero element of the first row 17.41 is in corresponding R
(a) information of the first row first row in~(d): apart from upper left comer horizontal and vertical offset being respectively 44 in candidate target region
With 12, characteristic area field width high score is not 4 and 8；And so on that the corresponding iRB-LBP of each nonzero element in R can be obtained is special
Levy region.
(e) each nonzero element value in R
It after matrix R is generated, is no longer changed during entire tracking, i.e., the compression of each subsequent candidate target is special
Sign vector will be calculated according to the R generated at this time.
Step 4. initialized target classifier
Using Naive Bayes Classifier H (y), to candidate target in subsequent frame sequence, whether tracked target differentiates,
Initialize the value of H (y) four parameters are as follows:
Step 5. updates object classifiers
If it is first frame, using the tracked target rectangular area of current selected as reference area, otherwise, with currently most
The rectangular area of good candidate target, that is, tracking result is as reference area.Benchmark region generates positive sample and negative sample, into
And the parameters of object classifiers are updated, the updated classifier parameters of first frame are as shown in the table:
Step 6. inputs new video frame
The particle assembly of step 7. predicting candidate target
Predict that the candidate particle list for generating present frame closes according to the particle assembly of previous framePrediction preceding 10
The case where a particle, is as shown in the table:
Step 8. calculates the CiRB-LBP feature vector of all candidate targets
The calculation method of CiRB-LBP feature vector is y=R × x, the calculation method of each element are as follows:Work as rijCorresponding x when=0jWithout calculating, therefore it need to only calculate non-zero rijThe x of corresponding positionj, that is, walk
The corresponding iRB-LBP characteristic area of each nonzero element extracts corresponding characteristic element value x in rapid 3 calculation matrixj, Zong Gongxu
64 characteristic element values are calculated, final CiRB-LBP feature vector y can be obtained.
By taking the corresponding iRB-LBP feature of first nonzero element of calculation matrix the first row as an example, in candidate target region
Middle corresponding position is (44,12,4,8), piecemeal situation (irregular 9 pieces, central block b0The a height of 2*2 of width), each piece of mean value, feature
Value calculates as shown in fig. 6, its characteristic value is 135.Method can calculate the corresponding iRB- of 8 nonzero elements of the first row according to this
LBP characteristic value, it is possible thereby to calculate first element -20260.8 of the CiRB-LBP feature vector of first candidate target.
Same method can calculate remaining element of the CiRB-LBP feature vector of first candidate target, and then form one
The corresponding CiRB-LBP feature vector of candidate target.
The corresponding CiRB-LBP feature vector of preceding 10 candidate targets is as shown in the table, and first row indicates first candidate mesh
Corresponding CiRB-LBP feature vector is marked, and so on, the CiRB-LBP feature vector of the corresponding candidate target of each column.
| -20260.8 | -18032.8 | -9921.52 | -17928.4 | -14586.4 | -12497.6 | -12497.6 | -17928.4 | -17928.4 | -17928.4 |
| 11992.85 | 11261.79 | 13454.97 | 15891.83 | 11435.85 | 15282.62 | 15282.62 | 15891.83 | 15891.83 | 15891.83 |
| 6509.908 | 6492.502 | 1966.897 | 3690.108 | -191.468 | 3637.89 | 3637.89 | 3690.108 | 3690.108 | 3690.108 |
| 7275.78 | 8529.024 | 1984.304 | 4490.792 | 4769.291 | 4386.355 | 4386.355 | 4490.792 | 4490.792 | 4490.792 |
| 10965.89 | 5761.443 | 13454.97 | 11540.29 | 10965.89 | 13280.91 | 13280.91 | 11540.29 | 11540.29 | 11540.29 |
| -9660.43 | -8685.68 | -12880.6 | -9521.18 | -8146.09 | -13785.7 | -13785.7 | -9521.18 | -9521.18 | -9521.18 |
| 3429.016 | 4717.073 | 487.3728 | 1618.774 | 1758.023 | 1862.46 | 1862.46 | 1618.774 | 1618.774 | 1618.774 |
| 3307.173 | -2262.8 | -4612.64 | 365.5296 | -765.872 | -17.4062 | -17.4062 | 365.5296 | 365.5296 | 365.5296 |
Step 9. differentiates tracking result
According to formulaIt calculates every
The H (y) of one candidate target, the evaluation of estimate of preceding 10 candidate targets are as follows:
| 8.83 | 2.73 | 16.82 | 18.21 | 7.70 | 18.40 | 18.40 | 18.21 | 18.21 | 18.21 |
In the evaluation of estimate H (y) of i.e. 100 candidate targets of 100 particles, min j=49 i.e. the 49th candidate target is commented
Value is minimum.All evaluation value correction methods are as follows:
Hj(y)=Hj(y)-Hminj(y)+ε=Hj(y)-H49(y)+ε=Hj(y) -1.06+5=Hj(y)+3.94
The evaluation of estimate of revised preceding 10 candidate targets are as follows:
| 12.77 | 6.67 | 20.76 | 22.15 | 11.64 | 22.34 | 22.34 | 22.15 | 22.15 | 22.15 |
To 100 evaluation of estimate regularizations, i.e.,Preceding 10 candidate targets after regularization
Evaluation of estimate are as follows:
| 0.007 | 0.004 | 0.0109 | 0.0116 | 0.006 | 0.012 | 0.0117 | 0.0116 | 0.0116 | 0.0116 |
The weight for updating 100 candidate targets enables α=0.85, i.e. wtj=wtj×0.85+Hj(y) × 0.15, before after update
The evaluation of estimate of 10 candidate targets are as follows:
| 0.0095 | 0.0090 | 0.0101 | 0.0102 | 0.0094 | 0.0103 | 0.0103 | 0.0102 | 0.0095 | 0.0105 |
Regularization is carried out to the weight of 100 candidate targets, i.e.,32nd candidate target after regularization
With maximum weight 0.0106, therefore, the 32nd candidate target is the tracking result of present frame, corresponding rectangular area
For (122,59,51,50).
According to weight wtjCandidate target, that is, particle is ranked up, according to the successively resampling of the biggish particle of weight, is given up
The lower particle of weight.
If step 10. present frame is last frame, tracking terminates；Otherwise, step 5 is gone to.
The above tracking process operation is in CPU:Intel Core i7-4790 CPU 3.6GHz, and memory: the hardware of 16GB is flat
On platform, the processing speed of 135 frame each second can achieve.Part tracking result screenshot as shown in fig. 7, it can be seen from the figure that
In the case of tracked target meets with the Various Complexes such as lighting change, postural change, visual angle rotation, unexpected movement, using this hair
Bright method is able to maintain accurate tracking effect always.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
The change or replacement expected without creative work, should be covered by the protection scope of the present invention.Therefore, of the invention
Protection scope should be determined by the scope of protection defined in the claims.
Claims (6)
1. a kind of video target tracking method based on the irregular piecemeal LBP of compression,
It is characterized in that, steps are as follows:
Step 1 selectes tracing area
Using frame each in the sequence of frames of video upper left corner as coordinate origin (1,1), the wide high respectively W and H of video frame；Selected first frame
In target to be tracked rectangular area (x0,y0, w, h), i.e., top left co-ordinate is (x0,y0), the wide high score of target area to be tracked
It Wei not w, h；
Step 2 initializes particle assembly
The target to be tracked duplication selected according to first frame generates k particle, forms initialization particle assemblyThe initial weight of each particle is 1/k, i.e.,
Step 3 initialization survey matrix
Calculation matrix is compressed used in compression sampling processIt is a sparse random gaussian matrix, m × n
Indicate the dimension of calculation matrix；
The number of nonzero element is in every row in RThe number of all nonzero elements is in RRemaining element is all
It is zero；The irregular piecemeal LBP feature i.e. iRB-LBP feature of the corresponding specific position of each nonzero element value in R；
After matrix R is generated, the nonzero element value of R and its corresponding iRB-LBP feature locations no longer occur during entire tracking
Variation, i.e., the compressive features vector of each subsequent candidate target will be calculated according to the R generated at this time；
The columns n of R is iRB-LBP feature vectorDimension；W' and h' is enabled to indicate that an iRB-LBP characteristic value is corresponding
The width and height in region, w' value range are 3~w, and the value range of h' is 3~h, and the calculation method of n is as follows:
N=(w-2) × (h-2) (1)
The line number m of R, is compressive featuresDimension；K=10 is enabled, the calculation method of c=1/logn, m are as follows:
The position (i, j) element calculation formula of matrix R are as follows:
Wherein, s=n/lnn indicates the sparse degree of calculation matrix；
Step 4 initialized target classifier
Use whether Naive Bayes Classifier H (y) differentiates for tracked target candidate target in subsequent frame sequence, H
(y) it is defined as follows:
Wherein, y be tracked target or candidate target region compression iRB-LBP feature vector i.e. CiRB-LBP feature to
Amount, p (v=1) and p (v=0) respectively indicate y and meet positive sample distribution and meet the probability of negative sample distribution, p (v=1)=p (v
=0)=0.5, v ∈ { 0,1 }；Assumed condition distribution p (yi| v=1) and p (yi| v=0) meet four parameters
Gaussian Profile, i.e.,
Step 5 updates object classifiers
If it is first frame, using the tracked target rectangular area of current selected as reference area, otherwise, waited with currently best
Select the target i.e. rectangular area of tracking result as reference area；Benchmark region generates positive sample and negative sample, and then right
The parameters of object classifiers are updated；
Step 6 inputs next video frame
The particle assembly of step 7 predicting candidate target
Predict that the candidate particle list for generating present frame closes according to the particle assembly of previous frame
Step 8 calculates the CiRB-LBP feature vector of all candidate targets
It enablesIndicating the CiRB-LBP feature vector of rectangular area, m × 1 indicates the dimension of CiRB-LBP feature vector,Indicate the iRB-LBP feature vector of tracked target or candidate target rectangular area, n × 1 indicates iRB-LBP feature
The dimension of vector, n are possible iRB-LBP Characteristic Number in rectangular area, each element x in xj, j=1~n correspondence one
IRB-LBP characteristic value；The calculation method of candidate target specific for one, CiRB-LBP feature vector y is as follows:
Step 9 differentiates tracking result
Firstly, calculating the evaluation of each candidate target CiRB-LBP feature vector y according to the calculation formula of H (y) in step 4
Value；Since the weight of particle must be a positive value, and H (y) can just be born, in order to correctly carry out more to candidate target weight
Newly, the smallest candidate target of score value is foundAccording to formula Hj(y)=Hj(y)-Hminj(y)+ε will
All H (y) are converted into positive value, wherein ε > 1.0；
Evaluation of estimate regularization to k candidate target, i.e.,According to the evaluation of estimate after regularization
The weight of each candidate target is updated, i.e.,
wtj=wtj×φ+Hj(y) × (1- φ), φ=0~1；Regularization is carried out to updated weight, i.e.,Candidate target of the regularization with weight limit is the tracking result of present frame；
According to weight wtjCandidate target, that is, particle is sorted from large to small, the foundation biggish particle of weight successively resampling,
Give up the lower particle of weight；
If step 10 present frame is last frame, tracking terminates；Otherwise, step 5 repetitive cycling is gone to.
2. according to claim 1 a kind of based on the video target tracking method for compressing irregular piecemeal LBP, feature exists
In manually or automatically selecting the rectangular area (x of target to be tracked in first frame in the step 10,y0,w,h)。
3. according to claim 1 a kind of based on the video target tracking method for compressing irregular piecemeal LBP, feature exists
In, in the step 2, initialization particle assemblyIn
4. according to claim 1 a kind of based on the video target tracking method for compressing irregular piecemeal LBP, feature exists
In initializing the value of four parameters in the step 4 are as follows:I=1~m
5. according to claim 1 a kind of based on the video target tracking method for compressing irregular piecemeal LBP, feature exists
In in the step 5, benchmark region generates positive sample and negative sample, and then carries out to the parameters of object classifiers
The specific method is as follows for update:
It is identical as reference area size, all in the range of α=4 with reference area central point distance for positive sample
Rectangular area constitutes positive sample set to be selected；Q is randomly choosed from positive sample set to be selected1=45 rectangular areas are as mesh
Mark positive sample；It can be calculated separately to obtain q according to step 81The corresponding CiRB-LBP feature in=45 positive sample rectangular areas to
Amount；Assuming that q here1It is μ that the ith measurement element of=45 feature vectors, which meets parameter,1And δ1Gaussian Profile, calculating side
Method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method it is as follows:
For negative sample, and reference area central point distance β=8~30 ranges in institute identical as reference area size
Some rectangular areas constitute negative sample set to be selected；Q is randomly choosed from negative sample set to be selected0Make=50 rectangular areas
For target negative sample；It can be calculated separately to obtain q according to step 80The corresponding CiRB-LBP in=50 negative sample rectangular areas is special
Levy vector；Assuming that q here0It is μ that the ith measurement element of=50 feature vectors, which meets parameter,0And δ0Gaussian Profile, meter
Calculation method is as follows；
λ > 0 is renewal rate, takes λ=0.85,WithUpdate method formula it is as follows:
6. according to claim 1 a kind of based on the video target tracking method for compressing irregular piecemeal LBP, feature exists
In, in the step 8, each element in CiRB-LBP feature vectorIt is calculating
During, work as rij=0, then rijxj=0, therefore, work as rij=0, then xjIt can be omitted calculating, directly enable xj=0；So only
It needs to calculate rijCorresponding x in the case where ≠ 0 (i=1~m, j=1~n)j, that is, it is corresponding to calculate each nonzero element in R
IRB-LBP feature locations extract corresponding characteristic element value xj, need to calculate in totalA iRB-LBP characteristic value
Obtain the corresponding CiRB-LBP feature vector y of a candidate target；
For the iRB-LBP feature of a specific position in candidate target region, first by the corresponding area of an iRB-LBP feature
Domain is divided into 3*3 i.e. 9 block, and i-th piece is expressed as bi, i=0~8, i-th piece of wide height is expressed as
Calculation method it is as follows:
Since the width and height of iRB-LBP characteristic area are not 3 multiple in most cases, the width of 9 piecemeals is high
It is not exactly the same as irregular piecemeal；Work as the multiple that w ' is 3,9 pieces of same size, otherwise, intermediate column width difference
In two column of left and right；When h' is 3 multiple, 9 pieces of height is identical, and otherwise, center row height is different from two rows up and down；Work as w ' and h'
It is all 3 multiple, 9 pieces of width is high all identical, and iRB-LBP characteristic value is identical as MB-LBP characteristic value；
Enable i-th piece of biThe mean value of middle all pixels gray value is expressed asIRB-LBP characteristic value calculation formula is as follows:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611185481.3A CN106780568B (en) | 2016-12-20 | 2016-12-20 | A kind of video target tracking method based on the irregular piecemeal LBP of compression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611185481.3A CN106780568B (en) | 2016-12-20 | 2016-12-20 | A kind of video target tracking method based on the irregular piecemeal LBP of compression |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106780568A CN106780568A (en) | 2017-05-31 |
| CN106780568B true CN106780568B (en) | 2019-08-13 |
Family
ID=58894164
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611185481.3A Expired - Fee Related CN106780568B (en) | 2016-12-20 | 2016-12-20 | A kind of video target tracking method based on the irregular piecemeal LBP of compression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106780568B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107527356B (en) * | 2017-07-21 | 2020-12-11 | 华南农业大学 | Video tracking method based on lazy interaction mode |
| CN107977982B (en) * | 2017-11-30 | 2021-11-02 | 云南大学 | Video target tracking method based on compressed regularization block difference |
| CN108076158B (en) * | 2018-01-08 | 2020-07-03 | 苏州大学 | Minimum load route selection method and system based on naive Bayes classifier |
| CN108596954B (en) * | 2018-04-25 | 2020-06-30 | 山东师范大学 | Video vehicle tracking method based on compressed sensing |
| CN111741332B (en) * | 2020-06-22 | 2022-08-26 | 稿定（厦门）科技有限公司 | Particle swarm rotating video production method and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915544A (en) * | 2012-09-20 | 2013-02-06 | 武汉大学 | Video image motion target extracting method based on pattern detection and color segmentation |
| CN103632382A (en) * | 2013-12-19 | 2014-03-12 | 中国矿业大学(北京) | Compressive sensing-based real-time multi-scale target tracking method |
| CN104299247A (en) * | 2014-10-15 | 2015-01-21 | 云南大学 | Video object tracking method based on self-adaptive measurement matrix |
| CN104850865A (en) * | 2015-06-05 | 2015-08-19 | 三峡大学 | Real-time compression tracking method of multi-characteristic transfer learning |
-
2016
- 2016-12-20 CN CN201611185481.3A patent/CN106780568B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102915544A (en) * | 2012-09-20 | 2013-02-06 | 武汉大学 | Video image motion target extracting method based on pattern detection and color segmentation |
| CN103632382A (en) * | 2013-12-19 | 2014-03-12 | 中国矿业大学(北京) | Compressive sensing-based real-time multi-scale target tracking method |
| CN104299247A (en) * | 2014-10-15 | 2015-01-21 | 云南大学 | Video object tracking method based on self-adaptive measurement matrix |
| CN104850865A (en) * | 2015-06-05 | 2015-08-19 | 三峡大学 | Real-time compression tracking method of multi-characteristic transfer learning |
Non-Patent Citations (2)
| Title |
|---|
| "A Fast and Accurate Unconstrained Face Detector";Shengcai Liao 等;《IEEE》;20160229;第38卷(第2期);第211-223页 |
| "粒子群优化的压缩跟踪算法";李杰 等;《中国图象图形学报》;20160831;第21卷(第8期);第1068-1077页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106780568A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106780568B (en) | A kind of video target tracking method based on the irregular piecemeal LBP of compression | |
| CN107292234B (en) | Indoor scene layout estimation method based on information edge and multi-modal features | |
| CN105225226B (en) | A kind of cascade deformable part model object detection method based on image segmentation | |
| Ma et al. | Crowd density analysis using co-occurrence texture features | |
| CN103632382B (en) | A kind of real-time multiscale target tracking based on compressed sensing | |
| KR100519781B1 (en) | Object tracking method and apparatus | |
| EP2680228A1 (en) | Improvements in or relating to three dimensional close interactions. | |
| Fan et al. | Iterative particle filter for visual tracking | |
| KR101409810B1 (en) | Real-time object tracking method in moving camera by using particle filter | |
| CN111445497A (en) | Target tracking and following method based on scale context regression | |
| Peker | Comparison of tensorflow object detection networks for licence plate localization | |
| US11816857B2 (en) | Methods and apparatus for generating point cloud histograms | |
| Zheng et al. | Shadow removal for pedestrian detection and tracking in indoor environments | |
| Yin et al. | Research on real-time object tracking by improved CamShift | |
| Suheryadi et al. | Spatio-temporal analysis for moving object detection under complex environment | |
| Dang et al. | Fast object hypotheses generation using 3D position and 3D motion | |
| CN113963270A (en) | High resolution remote sensing image building detection method | |
| Liu et al. | Eigenshape kernel based mean shift for human tracking | |
| Rahmaniar et al. | A novel object detection method based on Fuzzy sets theory and SURF | |
| Sun et al. | Fast and Robust RGB-D Multiple Human Tracking Based on Part Model for Mobile Robots | |
| Xie et al. | Research on a method of airport runway detection algorithm | |
| Yao et al. | RETRACTED: Moving object surveillance using object proposals and background prior prediction | |
| Chen et al. | Robust pedestrian detection and tracking with shadow removal in indoor environments | |
| Zhang et al. | Efficient defect detection with sign information of Walsh Hadamard transform | |
| Ioannidis et al. | Occupant-aware indoor monitoring for enhanced building analysis. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190813 Termination date: 20191220 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |