CN102682454A

CN102682454A - Method and device for tracking region of interest in video

Info

Publication number: CN102682454A
Application number: CN201210132913XA
Authority: CN
Inventors: 刘震; 张冬; 李厚强
Original assignee: University of Science and Technology of China USTC
Current assignee: University of Science and Technology of China USTC
Priority date: 2012-04-28
Filing date: 2012-04-28
Publication date: 2012-09-19
Anticipated expiration: 2032-04-28
Also published as: CN102682454B

Abstract

The invention discloses a method and a device for tracking a region of interest in a video. The method for tracking the region of interest in the video comprises the steps: first, obtaining motion vectors of pixels or macro blocks in an existing frame, confirming distributed parameters of the movement speed of the region of interest (ROI) according to the motion vectors, confirming ROI scaling parameters according to the ROI state in a reference frame, utilizing the distributed parameters of the movement speed of the ROI and the scaling parameters to perform state transition to particles obtained through sampling in the existing frame, and confirming the ROI position and the ROI size of the existing frame according to the particles after the state transition. The method and a device for tracking the region of interest in the video can utilize motion vector information which exists in compressed code flows or is generated in encoding to guide the state transition process of the particles, accordingly can decrease required particle number in the tracking process under the condition that the tracking effect is ensured, further reduce the complexity level of tracking processing, and can have good tracking effect.

Description

Method and device for tracking region of interest in video

Technical Field

The invention relates to the technical field of video processing, in particular to a method and a device for tracking an interested area in a video processing process.

Background

With the popularization and development of communication technology, corresponding video services such as mobile televisions, video conferences, video monitoring and the like are rapidly developed. In the process that a user accesses and develops corresponding video services through various terminals and different access modes, the diversity of the user terminals and the complexity of the network environment make how to effectively transmit video contents become a great challenge in designing a video service system.

Currently, SVC (scalable video coding) technology is capable of efficiently transmitting corresponding video content to some extent. In the SVC technology, subcode streams with various code rates, resolutions and frame rates are coded in a section of code stream at the same time, and a corresponding adaptive code stream can be generated by performing simple extraction operation at a transmission node according to the network condition and the requirements of users or user equipment. Compared with a single code stream technology, the SVC technology can provide a code stream with scalable space, time and quality, namely, some sub-code streams can be extracted from the code stream. The corresponding sub-code stream can meet the requirements of network transmission rate and terminal users on the aspects of space, time, signal to noise ratio and the like of the video, so that the SVC technology enables the video stream to be better suitable for various network environments and user terminals.

In SVC technology, the lowest quality coding layer that can be provided is called BL (base layer) and the coding layer that can enhance spatial resolution, temporal resolution or signal-to-noise strength is called EL (enhancement layer). Among them, the spatial scalability uses a Layered Coding (Layered Coding) method, using motion, texture, and residual information between layers; temporal scalability employs a Hierarchical bi-directional predictive frame (Hierarchical B) coding technique; for the scalability of the signal-to-noise ratio, methods of CGS (coarse grain quality scalability) and MGS (medium grain quality scalability) may be employed.

The corresponding SVC technique also provides support for ROI (region of interest) coding. An ROI generally refers to a region of a video frame that contains an object with a clear high level of semantic meaning to the viewer, such as a person, an object, etc. In the process of browsing the video by the user, if the display size of the device is small or the available bandwidth of the device is reduced, the definition of the region of interest can be maintained as much as possible, so that the viewing experience of the user on the video is not influenced. For example, when the access bandwidth is insufficient, a part of the non-interesting region may be deleted to adapt to the influence of the bandwidth requirement on the subjective quality of the video, that is, when the bandwidth is insufficient to transmit the encoded code streams of the base layer and the enhancement layer, the encoded code streams of the base layer and the ROI may be transmitted to fully utilize the bandwidth, so that the video quality may be maintained to a certain extent, and the subjective experience of the user may be ensured.

In order to implement the use of ROI coding techniques to accommodate a variety of different applications, the location and size of the ROI in each frame of the video needs to be determined, and video tracking techniques can be generally employed to determine the size and location of the ROI in each frame of the video.

The following describes the implementation of the presently adopted video tracking technique for determining ROIs in frames of video.

One currently used ROI tracking method is implemented based on a particle filter algorithm, specifically, a tracked region is represented as a particle (rectangular or elliptical, etc.), and the particle may include various states, such as a motion speed, a motion direction, and a region size. During tracking, a certain number of particles are generated in the current frame through importance sampling, and the stable state of the particles in the current frame is obtained through weighting according to the correlation between the certain number of particles and the particles in the region to be tracked in the reference frame, so that the region to be tracked in the current frame is obtained.

Correspondingly, a specific processing mode for obtaining the region to be tracked in the current frame based on the particle filter algorithm includes:

(1) initialization: take k as 0 (i.e., initial time) and press p (x)₀) Extracting N sample points

1, …, N, wherein p (x)₀) Refers to the initialized probability distribution of the target, and specifically, may be set to a gaussian distribution with the position size of the target as the mean in the first frame.

(2) Importance sampling:

order to

Wherein i is 1, …, N, wherein,

indicating the state of the ith particle at time k,

represents the state of the particle from time 0 (initial time) to time k, z_1:kRepresents an observed value (generally referred to as a color histogram of a tracking target) of a target from time 1 (initial tracking time) to time k,

it means that,

it is an estimation of the state distribution of the particles in the k-th frame, i.e. the importance function, under the condition that the observed values of the 1-k frames and the states of the i-th particle to the k-1 frame exist.

(3) Calculating the weight If one-step transfer posterior state distribution is employed, the equation can be simplified as:

wherein,

refers to observation models, i.e. representing particlesIs the probability of the object being tracked,

it is referred to as a state transition model, i.e. a probability distribution model of the target moving from k-1 frame to k frame.

(4) Normalizing the weight:

(5) resampling: according to respective normalized weight

Size of (2) copy or discard samples

Obtaining N approximate obeys

Distributed sampleOrder to

i＝1，…，N。

(6) And outputting a result: the output of the algorithm is a set of particlesBy which the posterior probability and the function x can be approximated_0:kWherein:

a posterioriRate:

function x_0:kThe expectation of (2):

(7) let k be k +1, the above-described processes (2) to (6) are repeated.

In the above implementation, if a stable tracking effect is to be obtained, a larger number of particles is required, and the larger the number of particles is, the larger the amount of calculation required for tracking is, resulting in a great increase in processing complexity.

Disclosure of Invention

The invention aims to provide a method and a device for tracking a region of interest in a video, which aim to reduce the processing complexity in the tracking process on the premise of ensuring the tracking effect.

The purpose of the invention is realized by the following technical scheme:

a method of region of interest tracking in a video, comprising:

acquiring a motion vector of a pixel or a macro block in a current frame, determining a moving speed distribution parameter of an ROI (region of interest) according to the motion vector, and determining an ROI scaling parameter according to state information of the ROI in a reference frame;

and performing state transition processing on the particles obtained by sampling in the current frame by using the moving speed distribution parameter and the scaling parameter of the ROI, and determining the position and the size of the ROI of the current frame according to the particles after the state transition.

Optionally, the step of determining the moving speed distribution parameter of the ROI includes:

determining the position of a reference pixel or a macro block corresponding to the pixel or the macro block in the current frame in the reference frame according to the motion vector of the pixel or the macro block in the current frame, and selecting the motion vector of the pixel or the macro block in the current frame, wherein the position of the reference pixel or the macro block is positioned in the ROI in the reference frame, in the motion vector of the pixel or the macro block in the current frame; determining the moving speed distribution parameter of the ROI according to the selected motion vector of the pixel or the macro block in the current frame;

optionally, the step of determining ROI scaling parameters comprises:

and determining the corresponding pixel or macro block in the current frame according to the selected motion vector of the pixel or macro block in the current frame, and determining the ROI scaling parameter according to the pixel or macro block in the current frame and the pixel or macro block in the corresponding reference frame.

Optionally, the moving speed distribution parameters of the ROI include:

wherein δ is a dirac function, G is a motion vector set of pixels or macroblocks in the current frame corresponding to the reference pixel or macroblock whose position is located in the ROI in the reference frame, and MV is_iRefers to the ith motion vector, MV, in G^ROIMoving speed of ROI from previous frame to current frame; the ROI scaling parameters include:

and a is, b is an affine parameter, the affine parameter is solved by combining a least square method and a four-parameter transformation model, and a corresponding four-parameter transformation module is established according to the pixel or the macro block in the current frame and the pixel or the macro block in the corresponding reference frame.

Optionally, the step of performing state transition processing on the particles sampled and obtained in the current frame includes:

determining the velocity v of the particle state transition according to the moving velocity distribution parameter of the ROI₁、v₂Comprises the following steps of; generating a random number mu in a uniform distribution of 0-1, and if mu is less than alpha, making

Sign variable

Otherwise, with distribution P (MV)^ROI) Selecting two components (MV ') corresponding to one element in the motion vector set G counted in the previous module'_x，MV′_y) Respectively as v₁、v₂Value of, i.e. order v₁＝MV′_x、v₂＝MV′_yTo and from

α is a state transition parameter, an initial value of which is preset, and a mode of updating the value in a subsequent particle updating process includes:

is v in particle n in the k-1 frame_xThe components of the first and second images are,

is v in particle n in the k-1 frame_yA component;

determining a particle scaling parameter η from the ROI scaling parameter, including; generating a random number gamma in a uniform distribution of 0-1, if gamma is less than beta, making eta equal to rho, otherwise making eta equal to 1, wherein beta is a target size change degree parameter, the value of beta is preset, and rho is the scaling parameter;

performing particle state transition processing according to the particle state transition speed and the particle scaling parameter, and obtaining a result after particle state transition of an nth particle in a kth frame includes: the position of the particle (

) The moving speeds of the particles in the x and y directions are respectively as follows:

and

the major semi-axis of the ellipse corresponding to the particle at the position is

Short semi-axis is

Wherein epsilon_x、ε_yIs a random variable obeying a distribution of N (0, 0.2),is a random variable, ε, obeying N (0, 0.25)_a、ε_bIs a random variable obeying N (0, 0.1).

Optionally, the step of determining the ROI of the current frame according to the state-transferred particles includes:

performing the statistical processing of the color histogram of each particle area on the particles subjected to the state transition processing, and performing particle updating processing according to the statistical processing result of the color histogram of each particle area;

the ROI position size is calculated based on the result obtained by the particle update process.

An apparatus for region of interest tracking in a video, comprising:

a moving speed distribution parameter and scaling parameter determining module, for obtaining the motion vector of the pixel or macro block in the current frame, and determining the moving speed distribution parameter of the ROI according to the motion vector, and also for determining the scaling parameter according to the state information of the ROI in the reference frame;

the particle state transition module is used for carrying out state transition processing on the particles sampled and obtained in the current frame by utilizing the moving speed distribution parameters and the scaling parameters of the ROI determined by the moving speed distribution parameter and scaling parameter determining module;

and the ROI determining module is used for determining the position and the size of the ROI of the current frame according to the particles subjected to state transfer by the particle state transfer module.

Optionally, the module for determining the moving speed distribution parameter and the scaling parameter specifically includes:

the motion vector acquisition module is used for determining the position of a reference pixel or a macro block corresponding to the pixel or the macro block in the current frame in the reference frame according to the motion vector of the pixel or the macro block in the current frame, and selecting the motion vector of the pixel or the macro block in the current frame corresponding to the reference pixel or the macro block of which the position is positioned in the ROI in the reference frame from the motion vector of the pixel or the macro block in the current frame;

the parameter determining module is used for determining the moving speed distribution parameter and the zooming parameter of the ROI according to the motion vector of the pixel or the macro block in the current frame selected and obtained by the motion vector obtaining module; and determining the corresponding pixel or macro block in the current frame according to the selected motion vector of the pixel or macro block in the current frame, and determining the ROI scaling parameter according to the pixel or macro block in the current frame and the pixel or macro block in the corresponding reference frame.

Optionally, the parameter determination module includes:

a moving speed distribution parameter estimation module of the ROI, configured to estimate a moving speed distribution parameter of the ROI, and the estimating of the obtained moving speed distribution parameter of the ROI includes:

wherein δ is a dirac function, G is a motion vector set of pixels or macroblocks in the current frame corresponding to the reference pixel or macroblock whose position is located in the ROI in the reference frame, and MV isⁱRefers to the ith motion vector, MV, in G^ROIMoving speed of ROI from previous frame to current frame;

an ROI scaling parameter estimation module, configured to estimate an ROI scaling parameter, and estimating the obtained ROI scaling parameter includes:

wherein, a is, b is affine parameter, the affine parameter is solved by adopting least square method combined with four-parameter transformation model, and corresponding four-parameter transformation moduleIs established according to the pixel or the macro block in the current frame and the pixel or the macro block in the corresponding reference frame.

Optionally, the particle state transition module includes:

a particle position and velocity transfer module for determining the velocity v of particle state transfer according to the ROI moving velocity distribution parameter obtained by the ROI moving velocity distribution parameter estimation module₁、v₂Comprises the following steps of; generating a random number mu in a uniform distribution of 0-1, and if mu is less than alpha, making

Sign variable

is v in particle n in the k-1 frame_yA component;

the particle size transfer module is used for determining a particle scaling parameter eta according to the ROI scaling parameter obtained by the ROI scaling parameter estimation module, and comprises the following steps of; generating a random number gamma in a uniform distribution of 0-1, if gamma is less than beta, making eta equal to rho, otherwise making eta equal to 1, wherein beta is a target size change degree parameter, the value of beta is preset, and rho is the ROI scaling parameter;

a particle state transition result determining module, configured to perform particle state transition processing according to the particle state transition speed determined by the particle position and speed transferring module and the particle scaling parameter determined by the particle size transferring module, where obtaining a result after particle state transition of an nth particle in a kth frame includes: the position of the particle (

and

Short semi-axis is

Optionally, the ROI determination module comprises:

each particle area color histogram statistical processing module is used for executing each particle area color histogram statistical processing on the particles after the state transition processing;

the particle updating module is used for performing particle updating processing according to the statistical processing result of the color histogram of each particle area;

and the ROI position size calculation module is used for calculating the ROI position size according to the result obtained by the particle updating processing.

It can be seen from the above technical solutions that, the ROI tracking technology provided in the embodiments of the present invention can use motion vector information existing in a compressed code stream or generated during encoding to guide a particle state transition process, so as to reduce the number of particles required in a tracking process under a condition that a tracking effect is ensured, thereby reducing a complexity of tracking processing, and obtaining a better tracking effect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without any inventive work.

FIG. 1 is a schematic process diagram of a method according to an embodiment of the present invention;

FIG. 2 is a first schematic structural diagram of an apparatus according to an embodiment of the present invention;

FIG. 3 is a second schematic structural diagram of an apparatus according to an embodiment of the present invention;

FIG. 4 is a third schematic structural diagram of an apparatus according to an embodiment of the present invention;

FIG. 5 is a fourth schematic structural diagram of an apparatus according to an embodiment of the present invention;

FIG. 6 is a MV distribution histogram provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of a particle state transition provided by an embodiment of the present invention;

FIG. 8 is a fifth schematic structural diagram of an apparatus according to an embodiment of the present invention;

FIG. 9 is a particle region color histogram provided by an embodiment of the present invention;

FIG. 10 is a schematic diagram of a particle update process according to an embodiment of the present invention;

FIG. 11 is a first diagram illustrating an application effect of the embodiment of the present invention;

FIG. 12 is a second schematic view illustrating an application effect of the embodiment of the present invention;

FIG. 13 is a first exemplary embodiment of an application environment;

fig. 14 is a schematic diagram of an application environment of the embodiment of the invention.

Detailed Description

The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.

Embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

The embodiment of the invention provides a method for tracking a region of interest in a video, and the specific implementation manner of the method is shown in fig. 1, and the method can comprise the following steps:

step 11, obtaining the motion vector of the pixel (or macro block) in the current frame, and determining the moving speed distribution parameter of the ROI according to the corresponding motion vector, and determining the ROI scaling parameter according to the state information of the ROI in the reference frame;

the current frame of the current frame on the coding structure is referred to as a reference frame for short, and the ROI position in the reference frame is known;

specifically, in this step, the process of determining the moving speed distribution parameter of the ROI may specifically include, but is not limited to:

firstly, determining the position of a reference pixel or a macro block corresponding to a pixel or a macro block in a current frame in the reference frame according to a motion vector of the pixel or the macro block in the current frame; then, selecting a reference pixel or a macro block with a corresponding position in the ROI of the reference frame from the motion vectors of the pixels or the macro blocks in the current frame, and acquiring the motion vector of the pixels or the macro blocks in the current frame corresponding to the reference pixel or the macro block; then, the corresponding moving speed distribution parameter of the ROI can be determined according to the selected motion vector of the pixel or the macro block in the current frame;

the step of determining ROI scaling parameters from the state information of the ROI in the reference frame may include, but is not limited to:

firstly, determining the corresponding pixels or macro blocks in the current frame according to the selected motion vectors of the pixels or macro blocks in the current frame; the ROI scaling parameters may then be determined from the pixels or macroblocks in the current frame and the corresponding pixels or macroblocks in the reference frame.

The calculation method of the moving speed distribution parameter of the ROI and the ROI scaling parameter may include, but is not limited to:

moving speed distribution parameter of ROI

Wherein δ is a dirac function, G is a motion vector set of pixels or macroblocks in the current frame corresponding to the reference pixel or macroblock whose position is located in the ROI in the reference frame, and MV isⁱRefers to the ith motion vector, MV, in G^ROIThe moving speed of the ROI from the previous frame (i.e. the reference frame) to the current frame;

ROI scaling parameters

Step 12, performing state transition processing on the particles sampled and obtained in the current frame by using the determined moving speed distribution parameter of the ROI and the ROI scaling parameter;

the specific implementation manner of the step may include:

first, according toThe moving speed distribution parameter of the ROI determines the speed v of the state transition of the particles₁、v₂Comprises the following steps of; generating a random number mu in a uniform distribution of 0-1, and if mu is less than alpha, making

Sign variable

is v in particle n in the k-1 frame_yComponent, and the current frame is the kth frame, and the reference frame is the (k-1) th frame;

then, determining a particle scaling parameter eta according to the ROI scaling parameter, including; generating a random number gamma in a uniform distribution of 0-1, if gamma is less than beta, making eta equal to rho, otherwise making eta equal to 1, wherein beta is a target size change degree parameter, the value of beta is preset, and rho is the scaling parameter;

finally, the particle state transition processing is performed according to the particle state transition speed and the particle scaling parameter, and the result obtained after the particle state transition of the nth particle in the kth frame includes: the position of the particle (

and

Short semi-axis is

Wherein epsilon_x、ε_yIs a random variable obeying a distribution of N (0, 0.2),

is a random variable, ε, obeying N (0, 0.25)_a、ε_bIs a random variable obeying N (0, 0.1).

Step 13, determining the position and size of the ROI of the current frame according to the particles after state transfer, and realizing tracking processing aiming at the ROI in the video;

specifically, the specific implementation manner of this step may include:

The embodiment of the invention mainly utilizes the additional information generated in the code stream or during the coding to guide the state transition of the particles, thereby ensuring the tracking effect aiming at the ROI and reducing the processing complexity of the tracking process. Specifically, the invention utilizes MV information existing in compressed code stream or generated during coding to guide the particle state transfer process in the particle filter algorithm, thereby reducing the number of required particles under the condition of ensuring the tracking effect and further reducing the complexity of the tracking processing process; alternatively, the present invention can obtain a more stable tracking effect in the case of performing tracking processing using the same number of particles. In addition, the invention also utilizes the stability of the particle filter theory on noise to obtain the robustness of the tracking algorithm, and further ensures the tracking effect of the tracking technical scheme provided by the embodiment of the invention.

An embodiment of the present invention further provides a device for tracking a region of interest in a video, and a specific implementation structure of the device is shown in fig. 2, where the device may include:

(1) a moving speed distribution parameter and scaling parameter determining module 21, configured to obtain a motion vector of a pixel or a macroblock in a current frame, determine a moving speed distribution parameter of a region of interest ROI according to the motion vector, and determine a scaling parameter of the ROI according to state information of the ROI in a reference frame;

further, as shown in fig. 3, the moving speed distribution parameter and scaling parameter determining module 21 may specifically include:

a motion vector obtaining module 211, configured to determine, according to a motion vector of a pixel or a macroblock in a current frame, a position of a reference pixel or a macroblock in the reference frame, where the pixel or the macroblock in the current frame corresponds to the reference pixel or the macroblock, select, from motion vectors of the pixel or the macroblock in the current frame, a reference pixel or a macroblock, where the position is located in an ROI in the reference frame, and then obtain a motion vector of the pixel or the macroblock in the current frame, where the reference pixel or the macroblock corresponds to the reference pixel or the macroblock;

a parameter determining module 212, configured to determine a moving speed distribution parameter of the corresponding ROI according to the motion vector of the pixel or the macroblock in the current frame selected and obtained by the motion vector obtaining module 211; determining a corresponding pixel or macro block in the current frame according to the selected motion vector of the pixel or macro block in the current frame, and determining a corresponding ROI (region of interest) scaling parameter according to the pixel or macro block in the current frame and the pixel or macro block in the corresponding reference frame;

the parameter determination module 212 may specifically include a ROI movement velocity distribution parameter estimation module 2121 and a ROI scaling parameter estimation module 2122, wherein:

a moving speed distribution parameter estimation module 2121 of the ROI, configured to estimate a moving speed distribution parameter of the ROI, and the estimating the obtained moving speed distribution parameter of the ROI includes:

wherein δ is the Dirac function, and G is the current corresponding to the reference pixel or macroblock whose position is within the ROI in the reference frameSet of motion vectors, MV, for pixels or macroblocks in a frameⁱRefers to the ith motion vector, MV, in G^ROIThe moving speed of the ROI from the previous frame (i.e. the reference frame) to the current frame;

an ROI scaling parameter estimation module 2122, configured to estimate an ROI scaling parameter, where estimating the obtained ROI scaling parameter includes:

(2) A particle state transition module 22, configured to perform state transition processing on the particles sampled and obtained in the current frame by using the moving speed distribution parameter and the scaling parameter of the ROI determined by the moving speed distribution parameter and scaling parameter determining module 21;

further, referring to fig. 8, the corresponding particle state transition module may include:

a particle position and velocity transition module 221, configured to determine the velocity v of particle state transition according to the moving velocity distribution parameter of the ROI obtained by the moving velocity distribution parameter estimation module 2121 of the ROI₁、v₂Comprises the following steps of; generating a random number mu in a uniform distribution of 0-1, and if mu is less than alpha, making

Sign variable

Otherwise, with distribution P (MV)^ROI) Selecting two components (MV ') corresponding to one element in the motion vector set G counted in the previous module'_x，MV′_y) Respectively as v₁、v₂Value of, i.e. order v₁＝MV′_x、v₂＝MV′_yTo and fromα is a state transition parameter, an initial value of which is preset, and a mode of updating the value in a subsequent particle updating process includes:

is v in particle n in the k-1 frame_yA component;

a particle size transfer module 222, configured to determine a particle scaling parameter η according to the ROI scaling parameter obtained by the ROI scaling parameter estimation module 2122, which may specifically include; generating a random number gamma in a uniform distribution of 0-1, if gamma is less than beta, making eta equal to rho, otherwise making eta equal to 1, wherein beta is a target size change degree parameter, the value of beta is preset, and rho is the ROI scaling parameter;

a particle state transition result determining module 223, configured to perform particle state transition processing according to the particle state transition speed determined by the corresponding particle position and speed transferring module 221 and the particle scaling parameter determined by the particle size transferring module 222, where the result obtained after the particle state transition of the nth particle in the kth frame includes: the position of the particle (

and

Short semi-axis is

(3) An ROI determining module 23, configured to determine a position and a size of an ROI of the current frame according to the particle subjected to the state transition by the particle state transition module 22;

specifically, as shown in fig. 4, the ROI determining module may specifically include:

a particle region color histogram statistical processing module 231, configured to perform particle region color histogram statistical processing on the particles subjected to the state transition processing;

a particle update module 232, configured to perform particle update processing according to a result obtained by performing the color histogram statistical processing on each particle region by the particle region color histogram statistical processing module 231;

the ROI position size calculation module 233 is configured to calculate the ROI position size according to a result obtained by the particle update module 232 performing the particle update process.

In the device, the MV information existing in the compressed code stream or generated during coding is used for guiding the particle state transition process, so that the required particle number can be reduced under the condition of ensuring the tracking effect, or the more stable tracking effect can be obtained under the condition of adopting the same particle number for tracking processing.

That is to say, in order to improve the accuracy of ROI tracking and reduce the tracking complexity, the embodiment of the present invention employs an implementation manner that MV information generated in a corresponding encoding process or already existing in a code stream is used to guide state transition of particles in a particle filter algorithm, so as to obtain a more superior tracking effect.

For a better understanding of the embodiments of the present invention, reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings and described in the following description.

In the embodiment of the invention, the ROI is tracked by utilizing MV information generated in the encoding process or already existing in a code stream according to the particle filter theory. The ROI is obtained frame by frame, and the data of the position, the size and the like of the ROI in the current frame are obtained according to the information of the ROI in the reference frame. The corresponding tracking procedure may mainly include:

firstly, obtaining a reference frame of a current frame from a buffer memory of the reference frame, and estimating the speed distribution and the scaling parameters of the ROI moving to the current frame by an ROI moving speed distribution parameter and scaling parameter estimation module according to the state information of the ROI in the reference frame and the MV of pixels (or macro blocks);

secondly, estimating the position and size information distribution of the ROI in the current frame by a particle state transfer module;

thirdly, the color histogram of each particle area obtained by statistics of the color histogram statistical module of each particle coverage area is utilized, the information distribution of the ROI obtained by the particle state transition module is further corrected by the particle updating module, the position size information of the ROI in the current frame is calculated and output, and the position size information and the current frame are stored into a reference frame for caching so as to estimate the ROI information in the subsequent frame.

Specifically, as shown in fig. 4, the apparatus provided in the embodiment of the present invention may specifically include:

(1) reference frame acquisition module

This module is used to obtain the reference frame of the current frame, and is the same as the method for obtaining the reference frame in the conventional encoding method, which is not shown in the figure.

(2) R0I moving speed distribution parameter and scaling parameter estimation module

The module is a moving speed distribution parameter and scaling parameter determining module, which is used to obtain moving speed distribution parameter estimation and scaling parameter estimation of the ROI region, as shown in fig. 5, the module may further include an MV obtaining module, an ROI moving speed distribution parameter estimation module and an ROI scaling parameter estimation module, and the processing functions performed by each module will be described below:

(21) an MV obtaining module, i.e. a motion vector obtaining module, for counting MVs of pixels of a reference pixel (or a macro block, wherein the pixels appearing in the following description can be replaced by macro blocks) located in the ROI region of the reference frame, wherein the corresponding MV is a two-dimensional vector (MV)_x，MV_y) I.e. the components of the MV of a pixel in the x, y direction;

specifically, as is known from the h.264/SVC coding standard, the position of the reference pixel in the reference frame can be obtained from the MV of the pixel in the current frame, so that if the reference pixel of a certain pixel in the current frame is located in the ROI region in the reference frame, the MV of the pixel in the current frame is obtained, a plurality of pixels in the current frame whose reference pixels are located in the ROI region in the reference frame are recorded as a set G, the number of elements in the set G is | G |, and the ith element is G_i，g_iMV of (1) is recorded as

i ═ 0, 1, …, | G | -1; further, a set of each element MV in G is recorded as M;

for example, as shown in fig. 6, the reference pixels associated with the corresponding MVs 1, 2, and 3 are located in the ROI region in the reference frame, so that the pixels in the current frame corresponding to the MVs 1, MV2, and MV3 are in statistical columns, i.e., the corresponding pixels need to be recorded in the set G; the reference pixel associated with MV0 is located outside the ROI in the reference frame, so the pixel in the current frame corresponding to MV0 is not in the statistics.

(22) A moving speed distribution estimation module for counting the (-MV) distribution histogram of the pixel in the current frame with the reference pixel located in the ROI area in the reference frame, wherein-MV is obtained by taking the negative sign of two components of MV respectively, i.e. the-MV distribution histogram of the pixel in the set G obtained by the MV acquisition module is normalized by the L1 norm, and is used as the estimation of the moving speed probability distribution of the ROI area from the reference frame to the current frame, and specifically can be recorded as a moving speed distribution parameter P (MV)^ROI) Namely:

wherein, δ is the Dirac function, and G is the current frame corresponding to the reference pixel whose position is located in the ROI in the reference frameMotion vector set, MV, of medium pixels or macroblocksⁱRefers to the ith motion vector, MV, in G^ROIThe moving speed of the ROI from the previous frame (i.e., the reference frame) to the current frame.

Only the statistics P (MV) are given in FIG. 6^ROI) Wherein '+' represents a pixel, a pixel reference pixel corresponding to the end of the dotted arrow is not located in the ROI area in the reference frame, and its motion vector MV⁰Is not in the statistical column; other motion vectors, e.g. MV¹，MV²These are all in the statistical column.

(23) A module for estimating a scaling parameter of the ROI region, configured to estimate a scaling parameter for obtaining the ROI region, where the scaling parameter is used to indicate a scaling relationship between the ROI region in the reference frame and the ROI region in the current frame, and specifically, the scaling parameter may be obtained based on an affine transformation, and the scaling parameter may be:

wherein a and b are affine transformation parameters;

the manner of specifically obtaining the ROI scaling parameter may include:

let the ith element G in G_iHas the coordinates of (x)_i，y_i) Then the coordinate of the corresponding reference pixel is

A special case of affine transformation, namely a four-parameter transformation model, is used as a target motion transformation model, namely:

wherein a, b, c, d are affine transformation parameters;

using the least squares criterion to estimate a uniform four-parameter transformation model of the ROI for all pixels in the set G, the following matrix A, B, C can be constructed, namely:

A = [\begin{matrix} a \\ b \\ c \\ d \end{matrix}],

B = [\begin{matrix} . \\ . \\ . \\ u_{i} & ({- v}_{i}) & 1 & 0 \\ v_{i} & u_{i} & 0 & 1 \\ . \\ . \\ . \end{matrix}],

C = [\begin{matrix} . \\ . \\ . \\ x_{i} \\ y_{i} \\ . \\ . \\ . \end{matrix}];

thus, the moving transformation model of the corresponding target can be converted into C ═ B × A, and A ═ B (B) can be obtained by the least square criterion^TB)^-1B^TC, wherein B^TThe matrix B and the matrix C are known quantities, so that the values of the affine transformation parameters a and B can be obtained, and further the value ρ of the corresponding scaling parameter can be obtained through calculation.

(3) Particle transfer module

The module, i.e. the particle state transition module, may be configured to estimate the probability distribution of the moving speed of the ROI region from the reference frame to the current frame based on the ROI moving speed distribution parameters and the estimation of the scaling parameters by the estimation moduleMeter P (MV)^ROI) And scaling the estimated rho of the parameter to guide the state transition of the particle so as to achieve a better tracking result.

Specifically, the purpose of the module is to perform preliminary estimation on ROI distribution of a current frame, as shown in fig. 7, each circle represents a particle (including information such as position and size of the ROI), a particle set in a reference frame represents state information distribution of the ROI in the reference frame, the particle set in the reference frame is subjected to particle state transition to obtain a particle set in the current frame, and the particle set in the current frame represents the state information distribution of the ROI in the current frame;

in the process of particle state transition, the information that the module needs to utilize includes: the particle states in the reference frame representing the ROI distribution, the state transition parameter α, and P (MV)^ROI) And ρ, where the initial value of α may be, but is not limited to, set to 0.5, and the value of the state transition parameter α is also automatically updated during subsequent processing.

Specifically, the same processing needs to be performed for each particle, and the description is given only by taking the state transition processing for the nth particle as an example, and as shown in fig. 8, the state transition processing procedure for the nth particle may include:

first, determining a speed of the particle state transition may specifically include: generating a random number mu in a uniform distribution of 0-1, and if mu is less than alpha, making

(i.e., v in particle n in k-1 frame_xComponent, the following expressions are similar)

Otherwise, the estimation result P (MV) of the probability distribution of the moving speed from the reference frame to the current frame according to the ROI area is used^ROI) Set of motion vectors M obtained beforeTwo components (MV ') corresponding to one element are selected'_x，MV′_y) Respectively as v₁、v₂Value of, i.e. order v₁＝MV′_x、v₂＝Mv′_yTo and from

Wherein v is₁、v₂A temporary variable;

secondly, determining a particle scaling parameter may specifically include: generating a random number gamma in a uniform distribution of 0-1, if gamma is less than beta, making eta equal to rho, otherwise making eta equal to 1, wherein eta is a temporary variable, beta is a target size change degree parameter, which is a constant parameter and is used for controlling the change degree of the target size, if the size change speed of the target is relatively high, the value of beta is relatively large, otherwise, the value of beta is relatively small, and the target size change in two adjacent frames is generally not very severe, so that the parameter beta can be set to be 0.2 without limitation;

third, particle state transitions, based on a previously determined temporary variable v₁、v₂And eta, performing particle state transition processing, specifically, calculating the state transition result of the corresponding particle of the nth particle in the kth frameComprises the following steps: the position of the particle in the x-axis directionThe position of the particle in the y-axis direction

The moving speed of the particles in the x-axis direction

The moving speed of the particles in the y-axis direction

The particles correspond to the major semiaxis of the ellipse at the corresponding positionThe minor semi-axis of the ellipse corresponding to the particle at the corresponding position

is a random variable, ε, obeying N (0, 0.25)_a、ε_bIs a random variable obeying N (0, 0.1). Wherein,

represents the nth particle in the kth frame, and means: ROI in k-th frame with probability

Appearing at (x, y) with the major and minor axes a, b, respectively, and in (v)_x，v_y) Moves to the next frame;

(4) color histogram statistical module for each particle region

This module is used to calculate an observation model of the particle filter theory, and the same operation needs to be performed for each particle, and the following description will take the observation model of the particle filter theory calculated for the particle n as an example.

Particle n in the current frame, the state is

It can be mapped to an ellipse with (x, y) as the center and a, b as the semi-major axis on the image of the current frame, and is marked as

As shown in FIG. 9, this module is used for the color histogram of the elliptical area in the histogram and normalized by the L1 norm, which can be recorded as

The calculation method is as follows:

wherein, suppose the video input is YUV color space, N_Y、N_U、N_VIs the number of bins divided by each color channel (obtained by quantizing YUV space, if each dimension of YUV space is equally divided into 10 parts, then N_Y、N_U、N_VAll equal to 10);

is falling on

The bin number of the bin into which the t-th pixel falls on each color channel (the bin number of the corresponding bin of each color channel is incremented from 0 up to 9);

representing an ellipse

The number of pixels in (1).

In particular, the implementation of the observation model for computing the particle filter theory in this module is the same as in the prior art, except that the input to the module is the state transition result of the particles obtained according to the method of the invention

s_{k}^{n} = {x, v_{x}, y, v_{y}, a, b} .

(5) Particle update module

The module is used for performing corresponding particle update processing, and specifically is used for calculating the weight of particles (namely the output of the particle transfer module) which are obtained through state transfer and represent the ROI state distribution in the current frame, obtaining the distribution estimation of the ROI in the current frame through resampling, and updating the state transfer parameter alpha.

Specifically, as shown in fig. 10, the corresponding particle update process may include:

first, the weight of each particle is updated

Wherein,

represents the weight of the nth particle in the kth frame (the probability that the region represented by the particle is the ROI region), where N represents the number of bins of the color histogram used, and N is equal to N_YN_UN_V；

A value indicating the jth bin of the color histogram of the particle n counted in the previous block (block for counting the color histograms of the respective particle regions); p (sb)₀) Refers to the color histogram of the initialized ROI area; parameter σ is a constant, which may be set to 0.25, but is not limited to; normalizing each particle weight

Secondly, the resampling operation may specifically include: first, the effective particle number is calculated

Then judge

If number of significant particles

Performing corresponding resampling operation, otherwise, not performing resampling operation, and directly performing the next third step to update the state transition parameter alpha, wherein N is the number of particles, and lambda is a constant parameter which can be preset; specifically, the corresponding resampling operation includes: set of particles for current frame

According to the weight

Sampling and putting in a new particle set

Of medium, i.e. particles

Is provided with

Is added into a new particle set S ', and the weight of each particle in S' is updated after resampling

And making S equal to S' to complete the corresponding resampling operation process.

Third, the state transition parameter α is updated:

(6) module for calculating ROI position size

After the calculation of the above modules, the particle set in the current frame is obtained

Now the position size of the ROI in the current frame, the position of the ROI center can be calculated

Length of side of ROI

Then, it is also necessary to assemble the particles representing the ROI state in the current frame

And storing the reference frame into a buffer of the reference frame so as to be conveniently applied in the subsequent tracking processing process.

The implementation of the embodiment of the invention can obtain better tracking effect with lower complexity in the process of tracking the ROI.

Specifically, the ROI tracking scheme provided in the embodiment of the present invention and the tracking scheme in the prior art are compared with each other using coastguard and stephen sequences as input, and the following results are found in the following experiment: in the coastguard sequence, the tracking scheme in the prior art loses the tracking result from the 49 th frame to the 56 th frame, but the tracking scheme provided by the embodiment of the invention shows that the tracking result is good; in the stephen sequence, the tracking scheme in the prior art loses the tracking result from the 33 rd frame to completely lose the tracking result from the 78 th frame, but the tracking scheme proposed by the embodiment of the present invention also shows that the tracking result is good.

In particle filter theory, one criterion used to measure the quality of a tracking scheme is the variation of the number of effective particles with tracking time. As shown in fig. 11 and fig. 12, which respectively show the change of the effective particle number with the tracking time in the tracking schemes of the prior art and the tracking schemes proposed by the embodiments of the present invention adopted in the coastguard and stephen sequences, the upper line in the two figures corresponds to the tracking scheme proposed by the embodiments of the present invention, and the lower line corresponds to the tracking scheme of the prior art, it can be seen that the effective particle number of the tracking scheme proposed by the embodiments of the present invention decreases more slowly with the tracking time, i.e. the tracking scheme proposed by the embodiments of the present invention is significantly superior to the tracking scheme in the prior art.

An environment in which the method and apparatus for tracking a region of interest in a video according to an embodiment of the present invention can be applied will be described below.

Application embodiment 1

The embodiment of the invention provides a method and a device for tracking a region of interest in a Video, which can be applied to Scalable Video Coding (SVC) Coding containing ROI scalability.

Specifically, the structure of an SVC encoder with ROI scalability is shown in fig. 13, and the corresponding SVC provides support for region of interest coding. The ROI is often a region of the video frame that contains an object with a clear high level of semantic meaning to the viewer, such as a person, an object, etc. In the process of browsing a video by a user, if the display size of the device is small or the available bandwidth is reduced, in order not to affect the viewing experience of the video, it is necessary to maintain the definition of the region of interest as much as possible. As shown in fig. 13, the present invention provides a method and apparatus for tracking a region of interest in a video, which can be used in the process of acquiring a ROI region in an SVC encoder with ROI scalability, i.e. can be used to implement the module for acquiring an ROI region in an enhancement layer shown in fig. 13. The input of the module is the coded base layer MV and video data information of the current frame, and after tracking processing, the size and position information of ROI in the current frame is output, so as to facilitate the ROI area coding of the enhancement layer.

Application example two

The method and the device for tracking the region of interest in the video provided by the embodiment of the invention can be applied to scalable video coding technology and can also be applied to transcoding of the region of interest.

Due to the limitations of the size of a display screen of a video terminal, network bandwidth and the like, an existing compressed video code stream is often required to be transcoded into a target code stream required by a client, and in order to ensure visual quality, an original compressed code stream can be transcoded into an ROI code stream, that is, visual information which is not critical to the client is discarded, and only the high quality of an ROI area is reserved. The structure of the corresponding cascaded transcoder is shown in fig. 14, and the ROI tracking module of the cascaded transcoder shown in fig. 14 can be used to obtain the position and size information of the ROI, so as to encode the ROI into a code stream with high quality.

Of course, the method and the device for tracking the ROI in the video according to the embodiments of the present invention can also be applied to other similar application environments requiring tracking the ROI. This is not exemplified.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for tracking regions of interest in a video, comprising:

2. The method of claim 1,

the step of determining the moving speed distribution parameter of the ROI includes:

the step of determining ROI scaling parameters comprises:

3. The method of claim 2, wherein the moving speed distribution parameters of the ROI comprise:wherein δ is a dirac function, G is a motion vector set of pixels or macroblocks in the current frame corresponding to the reference pixel or macroblock whose position is located in the ROI in the reference frame, and MV isⁱRefers to the ith motion vector, MV, in G^ROIMoving speed of ROI from previous frame to current frame; the ROI scaling parameters include:

4. The method of claim 3, wherein the step of performing state transition processing on the sampled particles in the current frame comprises:

Sign variable

is v in particle n in the k-1 frame_xThe components of the first and second images are,is v in particle n in the k-1 frame_yA component;

and

Short semi-axis is

is a random variable, ε, obeying N (0, 0.25)_a、ε_bIs a random variable obeying N (0, 0.1)。

5. The method according to any of claims 1-4, wherein the step of determining the ROI of the current frame from the state-transferred particles comprises:

6. An apparatus for tracking regions of interest in a video, comprising:

7. The apparatus of claim 6, wherein the moving speed distribution parameter and scaling parameter determining module specifically comprises:

8. The apparatus of claim 7, wherein the parameter determination module comprises:

and a is, b is an affine parameter, the affine parameter is solved by combining a least square method with a four-parameter transformation model, and a corresponding four-parameter transformation module is established according to the pixel or the macro block in the current frame and the pixel or the macro block in the corresponding reference frame.

9. The apparatus of claim 8, wherein the particle state transition module comprises:

Sign variable

is v in particle n in the k-1 frame_yA component;

and

Short semi-axis is

10. The apparatus of any one of claims 6-9, wherein the ROI determination module comprises: