CN106920255B - Moving object extraction method and device for image sequence - Google Patents
Moving object extraction method and device for image sequence Download PDFInfo
- Publication number
- CN106920255B CN106920255B CN201510983350.9A CN201510983350A CN106920255B CN 106920255 B CN106920255 B CN 106920255B CN 201510983350 A CN201510983350 A CN 201510983350A CN 106920255 B CN106920255 B CN 106920255B
- Authority
- CN
- China
- Prior art keywords
- frame
- bounding box
- moving
- attribute information
- probability
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention relates to the technical field of computer vision, and discloses a moving target extraction method and a device for an image sequence, wherein the moving target extraction method comprises the following steps: establishing hidden Markov models of various moving targets; detecting an image sequence of a moving target to be extracted by adopting a hidden Markov model, and outputting attribute information sets of all bounding boxes, which are obtained by detecting the hidden Markov model corresponding to various moving targets in all frame images; and searching out a set of moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm. The hidden Markov model is adopted to model the moving target, the appearance of the target under different postures is accurately represented, and the target with complex change can be processed. And (3) searching an optimal subsequence in a bounding box set obtained by detection in the image sequence by combining a forward probability propagation method and a backward backtracking method, so that the moving target can be quickly and accurately extracted.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a moving target extraction method and device for an image sequence.
Background
In the field of computer vision, two applications of similar target extraction and target tracking are carried out on a specific moving target in an image sequence, and the difference is that the target extraction is to extract the position and the size of the target in each frame from the existing image sequence; and the target tracking is to find the position and the size of a known target in a new image according to an existing image sequence when a new frame of image arrives, such as missile guidance.
In the technology, the target extraction process generally assumes that each frame of image is known and can be used for searching and matching, and a common searching method has a dynamic programming algorithm, but the algorithm efficiency is low; generally, target tracking models known targets in latest multi-frame images, predicts position and size changes of the targets, finds the known targets in a new frame image as much as possible, updates a model after finding the targets, and makes new predictions.
In addition, for target extraction and target tracking, there are also methods for extracting and tracking multiple types of moving targets in the prior art, for example, a component analysis method is used to perform multiple types of target matching, determine the attributes of the moving targets, recognize the moving targets, and perform temporal association to realize tracking of the moving targets. For another example, motion trajectory information of each motion target is extracted from video data, a motion mode of multi-motion target motion behaviors is modeled on each level, and recognition of the multi-motion target motion behaviors in the video is achieved by using a classifier. However, these methods have a drawback of low efficiency when classifying and extracting a specific moving object in an image sequence.
Disclosure of Invention
The invention aims to provide a moving object extraction method and a moving object extraction device for an image sequence, which are used for solving the problem of low efficiency of a classification extraction scheme of a specific moving object in the existing image sequence.
In order to achieve the above object, the present invention provides a moving object extraction method for an image sequence, the moving object extraction method comprising: establishing hidden Markov models of various moving targets; detecting an image sequence of a moving target to be extracted by adopting a hidden Markov model, and outputting attribute information sets of all bounding boxes, which are obtained by detecting the hidden Markov model corresponding to various moving targets in all frame images; and searching out a set of moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm.
Preferably, the outputting the attribute information set of each bounding box detected by the hidden markov model corresponding to each moving target in each frame image includes: and for any bounding box of each frame of image, if the probability that the image features of the bounding box are any states of any kinds of moving objects is greater than a first set threshold value, outputting the attribute information set of the bounding box.
Preferably, the attribute information includes an abscissa, an ordinate, a width, a height of the bounding box, and a probability that the bounding box is an arbitrary state of an arbitrary kind of moving object.
Preferably, the searching out a set of moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a backward backtracking algorithm includes: for each moving target, combining the attribute information set, and calculating the optimal probability of taking the image sequence at the end of each surrounding frame of each frame of image as the moving target by adopting a forward probability propagation algorithm; and searching out a set of moving objects to be extracted from the attribute information set by adopting a reverse backtracking algorithm in combination with the optimal probability.
Preferably, the calculating the optimal probability of the moving object by using the forward probability propagation algorithm with the image sequence at the end of each bounding box of each frame image comprises: v for t frame imagetA bounding box, a previous bounding box v of the bounding box is calculatedt-1And the v thtThe surrounding frames are the same moving object, and the v-th frame of the imagetProbability E (v) of each bounding box being state it-1I); from all the probabilities calculated, E (v)t-1I) selecting the maximum value E', and corresponding v to Et-1And i are set to v ″, respectivelyt-1And i'; if (W (k, t, v)t)+E＇)/(L(k,t,vt)+1)>z', wherein W (k, t, v)t) Indicating the v-th frame for the k-th moving objecttThe probability that the image sequence at the end of the bounding box is the k-th moving object, L (k, t, v)t) Indicating a vth in a tth frametThe number of frames of the image sequence at the end of each bounding box, z', is a second set threshold, then the recursive operation is performed according to the following formula: q (k, t, v)t)＝v＇t-1，L(k,t,vt)＝L(k,t,v＇t-1)+1，W(k,t,vt)＝W(k,t,vt) + E'; wherein, Q (k, t, v)t) V-th frame representing the frame of ttThe bounding box serial number of the t-1 th frame connected with the bounding box; and finally calculated W (k, t, v)t) As for the k-th moving object, the v-th frame of the t-th frametThe image sequence at the end of each bounding box is the optimal probability for this kth moving object.
Preferably, the probability E (v) is calculated using the following formulat-1,i)：E(vt-1,i)＝Pk(a(t,vt)|i)+αT(S(k,t,vt-1),i)+βC(t,vt,t-1,vt-1) (ii) a Wherein i represents the ith state, and i is 1, …, n, n is the total number of state types; c (t, v)t,t-1,vt-1) Is the v-th frame of the t-th frametA v th frame surrounding the frame and the t-1 th framet-1The size of the overlapping area of the bounding box accounts for the vth of the tth frametThe ratio of the sizes of the bounding boxes passing through t and v in the attribute information settCalculating corresponding attribute information; t (S (k, T, v)t-1) I) is the S (k, t, v)t-1) Probability of state transition of individual state to i-th state, S (k, t, v)t-1) V-th representing a t-th framet-1The sequence numbers of the states corresponding to the surrounding frames, α and β are the weight values of the state transition probability and the proportion of the overlapping area respectively.
Preferably, the searching out the set of moving objects to be extracted from the attribute information set by using a reverse backtracking algorithm includes: obtaining the v th frame of the t th frame according to the calculation result of the forward probability propagation algorithmtThe image sequence at the end of each bounding box is the maximum probability W ' of the kth moving object, and k corresponding to W ' is set as k '; let r be the v th frame to be searchedtAdding the attribute information corresponding to k' in the attribute information set into r for the image sequence at the end of each bounding box; find the v th frame of the t th frametThe bounding box number of the t-1 th frame of the bounding box connection is marked as v～ t-1K 'and v' in the attribute information set～ t-1Adding corresponding attribute information into r, and repeating the operation until the v th frame of the t th frametThe serial number of the bounding box of the t-1 th frame connected with the bounding box is 0; and combining the finally obtained r and k ' into a set { r, k ' } of the moving target to be extracted, wherein r represents the position and the size of the moving target, and k ' represents the type of the moving target.
The technical scheme of the invention also provides a moving object extraction device for the image sequence, which comprises the following steps: the model training module is used for establishing hidden Markov models of various moving targets; the target detection module is used for detecting the image sequence of the moving target to be extracted by adopting a hidden Markov model and outputting attribute information sets of all the surrounding frames, which are obtained by detecting the hidden Markov model corresponding to various moving targets in all the frames of images; and the target extraction module is used for searching a set of the moving targets to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm.
Preferably, the object detection module comprises: the sequence detection submodule is used for detecting the image sequence of the moving target to be extracted by adopting a hidden Markov model; and the selection output sub-module is used for outputting the attribute information set of the bounding box for any bounding box of each frame of image if the image characteristics of the bounding box are that the probability of any state of any kind of moving object is greater than a first set threshold value.
Preferably, the object extraction module comprises: the forward calculation submodule is used for calculating the optimal probability of taking the image sequence at the end of each enclosing frame of each frame of image as the moving target by combining the attribute information set for each moving target and adopting a forward probability propagation algorithm; and the reverse calculation submodule is used for searching out a set of the moving objects to be extracted from the attribute information set by adopting a reverse backtracking algorithm in combination with the optimal probability.
Through the technical scheme, the invention has the beneficial effects that: the hidden Markov model is adopted to model the moving target, the appearance of the target under different postures is accurately represented, and the target with complex change can be processed. And (3) searching an optimal subsequence in a bounding box set obtained by detection in the image sequence by combining a forward probability propagation method and a backward backtracking method, so that the moving target can be quickly and accurately extracted. After possible moving targets are rapidly detected through the appearance features and the moving features, sequence features are blended, and the sequence features are gradually refined from coarse to fine, so that the target extraction efficiency is improved.
Additional features and advantages of the invention will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
fig. 1 is a flowchart illustrating a moving object extraction method according to a first embodiment of the present invention.
Fig. 2 is a schematic structural diagram of a moving object extraction device in a second embodiment of the present invention.
Description of the reference numerals
1 model training module 2 target detection module
3 target extraction module 21 sequence detection submodule
22 selection output submodule 31 forward calculation submodule
32 inverse calculation submodule
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.
The invention provides a scheme for extracting a target by using a Hidden Markov Model (HMM) aiming at the problem of low efficiency of a classification extraction scheme of a specific moving target in the existing image sequence. The classical usage of HMMs is for recognition, such as shooting, after modeling a series of movements of the framed object, where modeling requires framing the position of the shooting hand in each frame of training data in advance, and recognition requires knowing the position of the shooting hand in each frame of test data in advance, and then the HMM can give the probability that this is a shot sequence. Compared with the classical usage, the method mainly emphasizes the extraction of the target sequence, disassembles the HMM, and utilizes the probability and backtracking of each frame of the HMM in each frame, and the details are described below.
The moving object extraction to be realized by the invention is to select an image sequence of the moving object from a plurality of sections of specific moving objects and each frame of image by using a rectangular surrounding frame. In view of this, a first embodiment of the present invention provides a moving object extraction method, as shown in fig. 1, including:
step S1, establishing hidden Markov models of various moving targets;
step S2, detecting the image sequence of the moving target to be extracted by adopting a hidden Markov model, and outputting the attribute information set of each bounding box detected in each frame image by the hidden Markov model corresponding to each moving target; and
and step S3, searching a set of the moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm.
The above steps S1-S3 implement model building, target detection and target extraction, respectively, and the detailed implementation processes of these three parts are described in detail below.
First, model establishment
First, the image features in the bounding box in each frame image are extracted, and the embodiment preferably extracts the HoG + HOF combined features commonly used in computer vision applications. HoG + HOF is the direct connection of the HoG characteristic vector and the HOF characteristic vector, and HoG (histograms of gradients) is a gradient histogram and represents the gradient distribution of a local area in an image; HOF (histograms of Optical flow) is an Optical flow histogram, and represents the Optical flow distribution of a local area in an image.
Then, for each moving object, using the image features of the image sequences corresponding to the moving object in the training data (hereinafter also referred to as image sequences simply as sequences), training an ns(k) HMM (Hidden MarkovModel) of a state obtains a state transition probability and an emission probability. Recording the state transition probability corresponding to the kth moving target as TkAnd the transmission probability is denoted as Pk. Let the number of kinds of moving objects be naThen k is 1, …, na. Number of states ns(k) Each state corresponds to a pose of the moving object, depending on the complexity of the moving object change. Probability of state transition Tk(i, j) represents the probability of the kth moving object jumping from the ith state to the jth state, and the emission probability Pk(q | i) represents the probability that the ith state of the kth moving object exhibits an image feature of q.
Second, target detection
First, a sliding window method is used to generate images in each frameAnd (4) surrounding frames with different positions and sizes, and extracting image features in each surrounding frame. Setting the image characteristic as a for any one of the surrounding frames, and calculating the probability P of the image characteristic relative to each state of each motion through the emission probability in the hidden Markov model training modulek(a|i)，k＝1,…,na，i＝1…ns(k) In that respect If the attribute information set B includes attribute information such as the position, size, and probability of each bounding box detected by the hidden markov model corresponding to each moving target in each frame image, in this embodiment, after the image sequence of the moving target to be extracted is detected by the hidden markov model, it is preferable that the probability P of a certain state i is obtained by taking the image feature a as a certain motion kk(ai) the attributes of the bounding box that are greater than a first set threshold z are saved in a set B, where the vth of the tth frametThe attributes of each bounding box are noted as:
B(k,t,vt)＝(x(k,t,vt),y(k,t,vt),w(k,t,vt),h(k,t,vt),P1(a(t,vt)|1),…,Pna(a(t,vt)|ns(na)) (1)
wherein t is 1, …, nf，v＝1,…,nb(t)，a(t,vt) Is the tth frame vtThe image features in the surrounding frame, x, y, w, h, P and other items on the right side of the formula (1) respectively represent the v < th > frame detected by the hidden Markov model of the k < th > moving targettThe abscissa, ordinate, width, height of each bounding box, the probability … … that the bounding box is the 1 st state of the 1 st motion, and the bounding box is the nth stateaN th of movements(na) Probability of individual state, nb(t) represents the number of bounding boxes in the tth frame in which the probability of a certain state of a certain motion is greater than the first set threshold value z. Wherein v istThe order of z can be arbitrarily specified, z is the minimum value of the probability that a single bounding box can be considered as a moving object, and the value of z is an empirical value, and can be preferably the average value of a plurality of collected P values.
Third, target extraction
For the kth moving object, let W (k, t, v)t) Watch (A)Denoted as the vth of the t frametThe sequence at the end of each bounding box is the probability, L (k, t, v), of the kth moving objectt) Indicating a vth in a tth frametNumber of sequence frames at the end of the bounding box, Q (k, t, v)t) V-th frame representing the frame of ttBounding box number of t-1 frame, S (k, t, v) of a bounding box connectiont) V-th representing a t-th frametThe second set threshold value z' represents the minimum value of the average probability of each frame that a bounding box sequence can be regarded as a moving object, preferably 1.5 times of the first set threshold value z, and R represents the finally extracted moving object set.
And step 1, for each moving target, combining the attribute information set, and calculating the optimal probability of taking the image sequence at the end of each enclosing frame of each frame image as the moving target by adopting a forward probability propagation algorithm.
For t 1 … nf，vt＝1…nb(t)，k＝1…naPerforming steps 1.1-1.4
Step 1.1, if t is equal to 1, skipping steps 1.1-1.3, and executing step 1.4; otherwise, for vt-1＝1…nb(t-1) calculating the bounding box and the tth frametThe surrounding frames are the same moving object, and the nth frame of the tth frame is calculated by adopting the following formulatProbability E (v) of each bounding box being state it-1,i)。
E(vt-1,i)＝Pk(a(t,vt)|i)+αT(S(k,t,vt-1),i)+βC(t,vt,t-1,vt-1) (2)
Wherein i represents the ith state, and i is 1, …, ns(k)，ns(k) The state number corresponding to the kth motion is obtained; c (t, v)t,t-1,vt-1) Is the v-th frame of the t-th frametA v th frame surrounding the frame and the t-1 th framet-1The size of the overlapping area of the bounding box accounts for the vth of the tth frametThe ratio of the sizes of the bounding boxes passing through t and v in the attribute information settCalculating corresponding attribute information; t (S (k, T, v)t-1) I) is the S (k, t, v)t-1) Probability of state transition of individual state to i-th state, S (k, t, v)t-1) V-th representing a t-th framet-1The sequence numbers of the states corresponding to the surrounding frames, α and β are the weight values of the state transition probability and the proportion of the overlapping area respectively.
Step 1.2, assign E ═ maxvt-1,iE(vt-1I), i.e. from all the probabilities E (v) calculatedt-1I) selecting the maximum E' and the maximum vt-1And i are each vt-1'and i'.
Step 1.3, if (W (k, t, v)t)+E＇)/(L(k,t,vt)+1)>z', then assigning values according to the following rules:
Q(k,t,vt)＝v＇t-1(3)
L(k,t,vt)＝L(k,t,vt-1＇)+1 (4)
W(k,t,vt)＝W(k,t,vt)+E＇(5)
and after the values are assigned according to the formula (3) to the formula (5), skipping the step 1.4;
if (W (k, t, v)t)+E＇)/(L(k,t,vt)+1)>z' does not hold, then step 1.4 is continued.
Step 1.4, carrying out recursive operation according to the following formula:
Q(k,t,vt)＝0， (6)
L(k,t,vt)＝1 (7)
E＇＝maxiPk(a(t,vt)|i) (8)
W(k,t,vt)＝W(k,t,vt)+E＇ (9)
performing an optimal search through the above steps 1.1-1.4, W (k, t, v) will be finally calculatedt) As for the k-th moving object, the v-th frame of the t-th frametThe image sequence at the end of each bounding box is the optimal probability for this kth moving object.
And 2, searching a set of moving targets to be extracted from the attribute information set by adopting a reverse backtracking algorithm in combination with the optimal probability.
For t ═ nf…1，vt＝nb(t) … 1, performing steps 2.1-2.2,
step 2.1, find out the v th frame with t th frametThe sequence at the end of a bounding box is the maximum probability of a moving object, i.e. of
W＇＝maxkW(k,t,vt) (10)
Let k be k' which is the maximum value in the formula (10).
Step 2.2, let r be the v th frame to be searchedtThe sequence at the end of each bounding box is initially empty. Collecting attribute information B (k', t, v)t) { x (k', t, v) } int),y(k＇,t,vt),w(k＇,t,vt),h(k＇,t,vt) Add sequence r. With Q (k', t, v)t) Find the v th frame of the t th frametThe bounding box number of the t-1 th frame of the bounding box connection is marked as v～ t-1B (k', t-1, v)～ t-1) { x (k', t-1, v) } of (C)～ t-1),y(k＇,t-1,v～ t-1),w(k＇,t-1,v～ t-1),h(k＇,t-1,v～ t-1) Add sequence r.
Thus cycling L (k', t, v)t) 1 times, i.e. up to Q (k ', t-L (k', t, v)t)-1,vt-L(k,t,vt)-1) Until the value of (2) is 0. In this case, the content in r should be
{{x(k＇,t,vt),y(k＇,t,vt),w(k＇,t,vt),h(k＇,t,vt)},
{x(k＇,t-1,v～ t-1),y(k＇,t-1,v～ t-1),w(k＇,t-1,v～ t-1),h(k＇,t-1,v～ t-1),
……}。 (11)
Adding the group of R, k 'into the set R, wherein R represents the position and the size of the moving target, k' represents the type of the moving target,
in order to avoid repetition of bounding boxes among a plurality of acquired sequences, after each bounding box is acquired, whether the overlapping degree of the existing bounding box of the frame and the existing bounding box of the frame is greater than a certain threshold is checked in the found sequence set R, the threshold is preferably 50%, if so, the bounding box before the frame is considered to be other moving targets, the search is not continued, and the frame is used as the starting point of the current sequence.
Based on the same inventive idea as the moving object extracting method in the first embodiment described above, a second embodiment of the present invention proposes a moving object extracting device, as shown in fig. 2, including: the model training module 1 is used for establishing hidden Markov models of various moving targets; the target detection module 2 is configured to detect an image sequence of a moving target to be extracted by using a hidden markov model, and output attribute information sets of each bounding box detected in each frame image by the hidden markov model corresponding to each moving target; and the target extraction module 3 is used for searching a set of the moving targets to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm.
Further, the object detection module 2 includes: the sequence detection submodule 21 is configured to detect an image sequence of a moving target to be extracted by using a hidden markov model; and a selection output sub-module 22, configured to, for any bounding box of each frame of image, output the attribute information set of the bounding box if the probability that the image feature of the bounding box is any state of any type of moving object is greater than a first set threshold.
Further, the target extraction module 3 includes: the forward calculation submodule 31 is configured to calculate, for each moving object, the optimal probability of taking the image sequence at the end of each bounding box of each frame image as the moving object by using a forward probability propagation algorithm in combination with the attribute information set; and a reverse calculation submodule 32, configured to search out a set of moving objects to be extracted from the attribute information set by using a reverse backtracking algorithm in combination with the optimal probability.
It should be noted that each functional module of the moving object extracting apparatus of this embodiment corresponds to the relevant steps of the moving object extracting method of the first embodiment, and therefore the specific implementation process and the working principle of each functional module are the same as or similar to those of the first embodiment, and are not described herein again.
In summary, the moving object extraction method and apparatus in the embodiments of the present invention have the following advantages:
1. the hidden Markov model is adopted to model the moving target, the appearance of the target under different postures is accurately represented, and the target with complex change can be processed.
2. And (3) by combining the methods of a forward probability propagation algorithm and a reverse backtracking algorithm, searching the optimal subsequence in the bounding box set obtained by detection in the image sequence, and realizing the rapid and accurate extraction of the moving target.
3. After possible moving targets are rapidly detected through the appearance features and the moving features, sequence features are blended, and the sequence features are gradually refined from coarse to fine, so that the target extraction efficiency is improved.
It will be understood, however, that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, however, the present invention is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present invention within the technical idea of the present invention, and these simple modifications are within the protective scope of the present invention.
It should be noted that the various features described in the above embodiments may be combined in any suitable manner without departing from the scope of the invention. The invention is not described in detail in order to avoid unnecessary repetition.
In addition, any combination of the various embodiments of the present invention is also possible, and the same should be considered as the disclosure of the present invention as long as it does not depart from the spirit of the present invention.
Claims (6)
1. A moving object extraction method for an image sequence, the moving object extraction method comprising:
establishing hidden Markov models of various moving targets;
detecting an image sequence of a moving target to be extracted by adopting a hidden Markov model, and outputting attribute information sets of all bounding boxes, which are obtained by detecting the hidden Markov model corresponding to various moving targets in all frame images; and
searching out a set of moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a backward backtracking algorithm,
the outputting of the attribute information sets of the bounding boxes detected by the hidden markov models corresponding to the various moving targets in the images of the frames includes: for any bounding box of each frame of image, if the image feature of the bounding box is that the probability of any state of any kind of moving object is greater than a first set threshold, outputting the attribute information set of the bounding box;
the searching out a set of moving objects to be extracted from the attribute information set by combining a forward probability propagation algorithm and a backward backtracking algorithm comprises the following steps:
for each moving target, combining the attribute information set, and calculating the optimal probability of taking the image sequence at the end of each surrounding frame of each frame of image as the moving target by adopting a forward probability propagation algorithm; and
and searching a set of the moving objects to be extracted from the attribute information set by adopting a reverse backtracking algorithm in combination with the optimal probability.
2. The moving object extraction method according to claim 1, wherein the attribute information includes an abscissa, an ordinate, a width, a height of the bounding box, and a probability that the bounding box is an arbitrary state of an arbitrary kind of moving object.
3. The method for extracting a moving object according to claim 1, wherein the calculating the optimal probability of the moving object by using the forward probability propagation algorithm with the image sequence at the end of each bounding box of each frame image comprises:
v for t frame imagetA bounding box, a previous bounding box v of the bounding box is calculatedt-1And the v thtThe surrounding frames are the same moving object, and the v-th frame of the imagetProbability E (v) of each bounding box being state it-1,i)；
From all the probabilities calculated, E (v)t-1I) selecting the maximum value E', and corresponding v to Et-1And i are set to v ″, respectivelyt-1And i';
if (W (k, t, v)t)+E＇)/(L(k,t,vt)+1)>z', wherein W (k, t, v)t) Indicating the v-th frame for the k-th moving objecttThe probability that the image sequence at the end of the bounding box is the k-th moving object, L (k, t, v)t) Indicating a vth in a tth frametThe number of frames of the image sequence at the end of each bounding box, z', is a second set threshold, then the recursive operation is performed according to the following formula:
Q(k,t,vt)＝v＇t-1，
L(k,t,vt)＝L(k,t,v＇t-1)+1，
W(k,t,vt)＝W(k,t,vt-1)+E＇；
wherein, Q (k, t, v)t) V-th frame representing the frame of ttThe bounding box serial number of the t-1 th frame connected with the bounding box; and
w (k, t, v) to be finally calculatedt) As for the k-th moving object, the v-th frame of the t-th frametThe image sequence at the end of each bounding box is the optimal probability for this kth moving object.
4. A moving object extraction method according to claim 3, characterized in that the probability E (v) is calculated using the following formulat-1,i)：
E(vt-1,i)＝Pk(a(t,vt)|i)+αT(S(k,t,vt-1),i)+βC(t,vt,t-1,vt-1)
Wherein i represents the ith state, and i is 1, …, n, n is the total number of state types; c (t, v)t,t-1,vt-1) Is the v-th frame of the t-th frametA v th frame surrounding the frame and the t-1 th framet-1The size of the overlapping area of the bounding box accounts for the vth of the tth frametThe ratio of the sizes of the bounding boxes passing through t and v in the attribute information settCalculating corresponding attribute information; t (S (k, T, v)t-1) I) is the S (k, t, v)t-1) Probability of state transition of individual state to i-th state, S (k, t, v)t-1) V-th representing a t-th framet-1The sequence numbers of the states corresponding to the surrounding frames, α and β are the weight values of the state transition probability and the proportion of the overlapping area respectively.
5. The method for extracting moving objects according to claim 1, wherein the searching out the set of moving objects to be extracted from the attribute information set by using a backward backtracking algorithm comprises:
obtaining the v th frame of the t th frame according to the calculation result of the forward probability propagation algorithmtThe maximum probability W' that the image sequence at the end of the bounding box is of the kth moving objectAnd setting the k corresponding to W 'as k';
let r be the v th frame to be searchedtAdding the attribute information corresponding to k' in the attribute information set into r for the image sequence at the end of each bounding box;
find the v th frame of the t th frametThe bounding box number of the t-1 th frame of the bounding box connection is marked as v～ t-1K 'and v' in the attribute information set～ t-1Adding corresponding attribute information into r, and repeating the operation until the v th frame of the t th frametThe serial number of the bounding box of the t-1 th frame connected with the bounding box is 0; and
and (3) combining the finally obtained r and k ' into a set { r, k ') of the moving target to be extracted, wherein r represents the position and size of the moving target, and k ' represents the type of the moving target.
6. A moving object extraction apparatus for an image sequence, characterized by comprising:
the model training module is used for establishing hidden Markov models of various moving targets;
the target detection module is used for detecting the image sequence of the moving target to be extracted by adopting a hidden Markov model and outputting attribute information sets of all the surrounding frames, which are obtained by detecting the hidden Markov model corresponding to various moving targets in all the frames of images; and
a target extraction module, which is used for searching a set of the moving targets to be extracted from the attribute information set by combining a forward probability propagation algorithm and a reverse backtracking algorithm,
wherein the target detection module comprises:
the sequence detection submodule is used for detecting the image sequence of the moving target to be extracted by adopting a hidden Markov model; and
the selection output sub-module is used for outputting an attribute information set of each bounding box for any bounding box of each frame of image, if the image characteristics of the bounding box are that the probability of any state of any kind of moving target is greater than a first set threshold value;
the target extraction module includes:
the forward calculation submodule is used for calculating the optimal probability of taking the image sequence at the end of each enclosing frame of each frame of image as the moving target by combining the attribute information set for each moving target and adopting a forward probability propagation algorithm; and
and the reverse calculation submodule is used for searching out a set of the moving targets to be extracted from the attribute information set by adopting a reverse backtracking algorithm in combination with the optimal probability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510983350.9A CN106920255B (en) | 2015-12-24 | 2015-12-24 | Moving object extraction method and device for image sequence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510983350.9A CN106920255B (en) | 2015-12-24 | 2015-12-24 | Moving object extraction method and device for image sequence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106920255A CN106920255A (en) | 2017-07-04 |
| CN106920255B true CN106920255B (en) | 2020-06-05 |
Family
ID=59456967
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510983350.9A Active CN106920255B (en) | 2015-12-24 | 2015-12-24 | Moving object extraction method and device for image sequence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106920255B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107507190B (en) * | 2017-07-12 | 2020-02-14 | 西北工业大学 | Low-altitude moving target detection method based on visible light sequence image |
| CN108829248B (en) * | 2018-06-01 | 2020-11-20 | 中国科学院软件研究所 | Moving target selection method and system based on user performance model correction |
| CN112465859A (en) * | 2019-09-06 | 2021-03-09 | 顺丰科技有限公司 | Method, device, equipment and storage medium for detecting fast moving object |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102656886A (en) * | 2009-12-14 | 2012-09-05 | 汤姆逊许可证公司 | Object-aware video encoding strategies |
| US8391548B1 (en) * | 2008-05-21 | 2013-03-05 | University Of Southern California | Tracking multiple moving targets in digital video |
| CN103069414A (en) * | 2010-08-11 | 2013-04-24 | 索尼公司 | Information processing device, information processing method, and program |
| CN103530995A (en) * | 2013-10-12 | 2014-01-22 | 重庆邮电大学 | Video monitoring intelligent early-warning system and method on basis of target space relation constraint |
| CN104937607A (en) * | 2012-11-22 | 2015-09-23 | 弗兰霍菲尔运输应用研究公司 | Device, method, and computer program for reconstructing a motion of an object |
-
2015
- 2015-12-24 CN CN201510983350.9A patent/CN106920255B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8391548B1 (en) * | 2008-05-21 | 2013-03-05 | University Of Southern California | Tracking multiple moving targets in digital video |
| CN102656886A (en) * | 2009-12-14 | 2012-09-05 | 汤姆逊许可证公司 | Object-aware video encoding strategies |
| CN103069414A (en) * | 2010-08-11 | 2013-04-24 | 索尼公司 | Information processing device, information processing method, and program |
| CN104937607A (en) * | 2012-11-22 | 2015-09-23 | 弗兰霍菲尔运输应用研究公司 | Device, method, and computer program for reconstructing a motion of an object |
| CN103530995A (en) * | 2013-10-12 | 2014-01-22 | 重庆邮电大学 | Video monitoring intelligent early-warning system and method on basis of target space relation constraint |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106920255A (en) | 2017-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107481270B (en) | Table tennis target tracking and trajectory prediction method, device, storage medium and computer equipment | |
| Parisotto et al. | Global pose estimation with an attention-based recurrent network | |
| Johnander et al. | A generative appearance model for end-to-end video object segmentation | |
| Ullah et al. | Hog based real-time multi-target tracking in bayesian framework | |
| CN109871821B (en) | Pedestrian re-identification method, device, equipment and storage medium of self-adaptive network | |
| US10657625B2 (en) | Image processing device, an image processing method, and computer-readable recording medium | |
| CN109472274B (en) | Training device and method for deep learning classification model | |
| CN109940614B (en) | Mechanical arm multi-scene rapid motion planning method integrating memory mechanism | |
| CN106920255B (en) | Moving object extraction method and device for image sequence | |
| Ammirato et al. | A mask-rcnn baseline for probabilistic object detection | |
| Capellen et al. | ConvPoseCNN: Dense convolutional 6D object pose estimation | |
| CN113129335B (en) | Visual tracking algorithm and multi-template updating strategy based on twin network | |
| CN109063549B (en) | High-resolution aerial video moving target detection method based on deep neural network | |
| CN111027555B (en) | License plate recognition method and device and electronic equipment | |
| JPWO2018235219A1 (en) | Self-location estimation method, self-location estimation device, and self-location estimation program | |
| KR101903684B1 (en) | Image characteristic estimation method and device | |
| Bashar et al. | Multiple object tracking in recent times: A literature review | |
| KR101705584B1 (en) | System of Facial Feature Point Descriptor for Face Alignment and Method thereof | |
| CN112200862B (en) | Training method of target detection model, target detection method and device | |
| CN109101858B (en) | Action recognition method and device | |
| CN106934339B (en) | Target tracking and tracking target identification feature extraction method and device | |
| Tian et al. | High-speed tiny tennis ball detection based on deep convolutional neural networks | |
| CN113033523A (en) | Method and system for constructing falling judgment model and falling judgment method and system | |
| Nikpour et al. | Deep reinforcement learning in human activity recognition: A survey | |
| CN110288633B (en) | Target tracking method and device, readable storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |