CN102542593A - Interactive video stylized rendering method based on video interpretation - Google Patents

Abstract

The invention relates to an interactive video stylized rendering method based on video interpretation, wherein an interactive video semantic segmentation module and a video stylization module are utilized. A segmentation method of the interactive video semantic segmentation module comprises the following steps of: (1) interactive segmentation and automatic identification of key frame images; (2) matching of dense characteristic points among key frames; and (3) area competition segmentation. A stylization method of the video stylization module comprises the following steps of: (4) un-reality sense drawing of the key frames based on semantic analysis; (5) a brushwork propagating method of a sequence frame; and (6) a damping pen brush system for preventing shaking. The interactive video stylized rendering method based on the video interpretation, disclosed by the invention, has the advantages of short manufacturing period, low cost and favorability on manufacturing in batches.

Description

A kind of interactive video stylized rendering method interpreted based on video

Technical field

The present invention is a kind of interactive video stylized rendering method interpreted based on video, belongs to the renovation technique of the interactive video stylized rendering method interpreted based on video. 

Background technology

As computer, digital camera and DV are popularized on a large scale, people are for the making demand more and more higher in terms of audio-visual amusement.What is triggered therewith is flourishing for family digital entertainment field.Increasing people begins attempt to, when having played amateurish " director ", be keen to make and edit various common realistic videos.Recent years, various stylized videos are progressively received by people, and as fashion elements, especially in terms of animated video and network game make.The oil painting short-movie of such as manual drawing《Old man and sea》With ink and wash video《Small tadpole looks for mother》Deng all causing being absorbed in extensively for people, the former also obtains a series of awards such as Oscar short-movie.Video stylized rendering not only needs professional technique, and also needs to a large amount of manpower financial capacity's supports, and the stylized technology of traditional video is to realize stylized rendering by method for drafting frame by frame.Although visual effect of the works completed with this making pattern per two field picture can be with hand-guided, but continuous play then result in video pictures and there is larger jitter phenomenon due to lacking the uniformity of interframe, and these method fabrication cycle length, high costs, it is unfavorable for batch making.Such as, it is mentioned above《Old man and sea》Oil painting short-movie, although duration only has 22 minutes, but fabrication cycle but be up to nearly 3 years as long as. 

The content of the invention

There is provided that a kind of fabrication cycle is short, low cost it is an object of the invention to consider above mentioned problem, beneficial to the interactive video stylized rendering method interpreted based on video of batch making. 

The technical scheme is that：The interactive video stylized rendering method that the present invention is interpreted based on video, includes the stylized module of interactive video semantic segmentation module, video, and the dividing method of interactive video semantic segmentation module comprises the following steps： 

1) Interactive Segmentation and automatic identification of key frame images； 

2) between key frame dense characteristic point matching； 

3) region-competitive partitioning algorithm； 

The stylizing method of video stylization module comprises the following steps： 

4) the key frame non-photorealistic rendering based on semanteme parsing； 

5) the style of writing transmission method of sequence frame； 

6) it is used for the damping brush system of stabilization. 

Stylization to video will use the two modules successively.It is i.e. first that semantic segmentation is carried out to video using interactive semantic segmentation module.Reuse video stylized module and stylized rendering is carried out to the video after segmentation.Above-mentioned steps 1) key frame images Interactive Segmentation and automatic identifying method it is as follows： 

They be divide into 12 classes by the semantic region of segmentation according to its different material properties, including sky/cloud, mountain range/land, rock/building, leaf/grove, hair/hair, flower/fruit, skin/hide, trunk/branch, abstract background, wood/plastics, water, clothes； 

In practical operation, employ texture, distribution of color, positional information these three principal characters and be trained and recognize, give an area image X, the conditional probability for defining its classification c is： 

LogP (x | X, θ)=∑_iΨ_i(c_i, X；θ_Ψ)+π(c_i, X；θ_π)+λ(c_i, X；θ_λ)-logZ (θ, X) (*)

Four are texture potential-energy function, color potential-energy function, position potential-energy function and normalization item respectively after in formula. 

Texture potential-energy function is defined as Ψ_i(c_i, X；θ_Ψ)=logP (c_i| X, i), P (c_i| X, i) it is a normalized distribution function being given by Boost graders； 

Color potential-energy function is defined as π (c_i, X；θ_π)=log ∑s_kθ_n(c_i, k) P (k | x_i), with gauss hybrid models (the Gaussian Mixture Models in CIELab color spaces：GMMs color model) is represented, to a pixel color x in given image, its conditional probability is： 

Wherein μ_kRepresent the average and variance of k-th of color cluster respectively with ∑ k； 

Position potential-energy function is defined as λ (c_i, X；θ_λ)=log θ_λ(c_i, i), for above two potential-energy functions, position potential-energy function is relatively weak, and in the definition of this function, the category label of image pixel is only related to absolute position in the picture； 

It is trained using the method for 12 class materials, all pixels in the probability for each classification for giving each pixel in an image-region, last statistical regions is then calculated using above formula *, the classification in each region is determined by the way of ballot；During stylized rendering, the material that the selection of paintbrush is identified by object area is determined, to realize that automatic render lays the foundation. 

Above-mentioned steps 2) key frame between dense characteristic point matching process it is as follows： 

After semantic information on key frame is obtained, comprehensive line drawing feature, texture and blend of colors image template feature provide abundant characteristic set and expression for image matching problems； 

11) line drawing feature is shown as by Gabor base tables： 

F^sk(I_i)=| |<I_i, G_{Cos, x, θ}>||²+||<I_i, G_{Sin, x, θ}>||², G_{Sin, x, θ}And G_{Cos, x, θ}It is illustrated respectively in position

X prescriptions to for θ sine and cosine Gabor bases.The distribution of its characteristic probability is expressed as： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{sk}}\right)}\exp \left\{{\mathrm{\&lambda;}}_i^{\mathrm{sk}}{h}^{\mathrm{sk}}\right[F^{\mathrm{sk}}\left(I_i\right)\left]\right\}$

Represent parameter θ_i, h^skIt is a sigmoid function,

It is normalization constraints. 

So model will encourage have stronger corresponding edge than background distributions； 

12) textural characteristics are modeled with a simplified gradient orientation histogram (HOG) to textural characteristics, and 6 characteristic dimensions represent different gradient directions respectively；HOG j-th of direction is represented, and

Represent ith feature I_iCorresponding description； 

It is F^txt(I_i) average in all positive samples.The probabilistic model of feature is expressed as by the present invention： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{txt}}\right)}\exp \left\{{\mathrm{\&lambda;}}_i^{\mathrm{txt}}{\mathrm{\&Sigma;}}_j{h}_j^{\mathrm{txt}}\right[F^{\mathrm{txt}}\left(I_i\right)\left]\right\}$

It is parameter θ_i.It can be seen that model encourages to respond the set for the characteristic image block relatively concentrated； 

13) color characteristic be using simple pixel intensity as description,

It is the wave filter on the x of position.Quantizing pixel brightness value of the present invention is interval to each statistics, then model can be reduced to： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{fl}}\right)}\exp \left\{{\mathrm{\&Sigma;}}_j{\mathrm{\&lambda;}}_i^{\mathrm{fl}}{h}_{\mathrm{xj}}^{\mathrm{fl}}\right[F_{\mathrm{xj}}^{\mathrm{fl}}\left(I_i\right)\left]\right\}$

The small feature of similar image by combining, the local combinations of features with strong judgement index can be just received, over-segmentation is carried out to image first, the tiny image block of some in image is obtained, line drawing can be described by being extracted from small image block, texture, the statistical nature of color, in order to effectively obtain combinations of features, increased and model-learning algorithm using the region of iteration, by constantly updating characteristic model, iteration growth feature combination zone finally gives the local combinations of features with strong judgement index； 

On the basis of expressing herein, matching problem of the moving target in time domain and spatial domain is modeled as the hierarchical diagram the matching frame on figure is represented, the mixed image template characteristic of extraction is used as node of graph, graph structure is built between frames, side annexation between node of graph can be defined with the affiliated object type of the similarity between feature based, locus, and feature； 

With Is, It represents artwork and target figure, and U, V represent hybrid template characteristic set in Is, It respectively, to each characteristic point u ∈ U ', there is two marks：Level mark I (u) ∈ { 1,2 ..., K } and matching candidate mark

With the higher Candidate Set C of each Feature Points Matching degree in artwork, the vertex set of graph structure is set up, with E=E⁺∪E^-Build line set.Represent that the candidate of connection is mutually exclusive with negative side, and define its " repulsive force " and be： 

With the adjacent and not mutually exclusive candidate feature point of positive side connection space, order

The tightness degree cooperated between them is represented,Represent v_i, v_jBetween space length； 

By original image and the graph structure G of target image^s、G^T, K+1 layers are divided into, wherein K represents the object number in artwork, with G^sExemplified by, division is expressed as ∏={ g₀, g₁..., g_k}.Wherein, g_kIt is G^sA subgraph, its vertex set is with U_kRepresent.Similar, G^TVertex set with V_kRepresent.Then G^sAnd G^TBetween matching relationship be expressed asAssuming that the matching between subgraph is separate, then： 

Estimated with geometric transformation, outward appearance and define Matching sub-image to (g_k, g_k') between similarity measure, use

Represent；In summary, the solution of graph structure matching problem is configurable to： 

W=(K, ∏={ g₀, g₁..., g_k, Ψ={ Φ_k, Φ={ Φ_k}) 

Under bayesian theory framework, graph structure matching problem is described to maximize posterior probability： 

W^*=argmaxp (W | G^s, G^T)=argmaxp (W) p (G^s, G^T|W) 

Above formula is solved by markov chain Monte-Carlo (MCMC) method, while in order to efficiently calculate, by efficiently redirecting in solution space, globally optimal solution is rapidly converged to, to reach the matching of interframe characteristic point. 

Above-mentioned steps 3) region-competitive dividing method it is as follows： 

On the basis of the matching relationship of interframe stabilization is obtained, pass through the advantage in Video segmentation of excavation regions competition mechanism, utilize the image matching algorithm for being layered graph structure, the matching relationship between former frame and present frame feature can be determined, the semantic information of so former frame is just traveled in present frame, then region-competitive partitioning algorithm is utilized according to the characteristic information of each matching area, present frame is divided into multiple semantic regions； 

Given image I, corresponding image segmentation solution is defined as follows： 

W={ (R₁, R₂... R_N), (θ₁, θ₂..., θ_N), (I₁, I₂, ..., I_N)} 

Wherein, R_iThe region with same characteristic being partitioned into is represented,

θ_iRepresent region R_iThe parameter of corresponding characteristic probability distributed model, I_iRepresent region R_iCorresponding mark； 

According to the matching relationship of feature in front and rear frame, it may be determined that cut zone number N.If the feature zonule set S={ S corresponding to each region₁, S₂..., S_N, for each region R_i, the zonule S occupied according to feature_iEstimate the initial parameter θ of model_i, obtain initial posterior probability P (θ_i| I (x, y)).According to MDL principles, posterior probability is converted into solution energy function minimum problem, obtained： 

$\log P\left(W\right|I)=E[\mathrm{\&Gamma;},\{{\mathrm{\&theta;}}_i\left\}\right]=\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}-{\&Integral;\&Integral;}_{R_1}\log P\left({\mathrm{\&theta;}}_i\right|I(x,y))\mathrm{dxdy}$

Wherein

Represent region R_iBoundary profile.The present invention estimates parameter { θ stage by stage using iterative manner_iAnd Γ, alternating iteration two benches, constantly reduce energy function in the various stages, so that constantly Reasoning With Learning goes out the final segmentation result of entire image； 

During region-competitive, its characteristic probability distributed model is constantly updated in each region, while declining the ownership that principle fights for pixel according to steepest, updates respective boundary profile so that scope is constantly expanded in each region, finally gives the image segmentation result of present frame； 

Its specific iterative step is：First stage, fixed Γ, according to current region segmentation state estimation { θ_i, try to achieve parameter θ under current state_iMaximal possibility estimation be used as its optimal solution

To minimize the cost for describing each region, therefore energy function is converted into： 

${\mathrm{\&theta;}}_i^{*}={\arg \max }_{{\mathrm{\&theta;}}_i}\left\{{\&Integral;\&Integral;}_R\log P\left({\mathrm{\&theta;}}_i\right|I(x,y))\mathrm{dxdy}\right\},\&ForAll;i\&Element;[1,N]$

Second stage, { θ_i, it is known that carrying out steepest decline to Γ, in order to quickly try to achieve the minimal solution of energy function, the present invention solves the equation of motion that steepest declines to the border Γ in all regions.For any point on boundary profile Γ

Have

$\frac{d\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}}{\mathrm{dt}}=-\frac{\mathrm{\&delta;E}(\mathrm{\&Gamma;},\{{\mathrm{\&theta;}}_i\left\}\right)}{\mathrm{\&delta;}\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}}={\mathrm{\&Sigma;}}_{k\&Element;Q\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right)}\log P\left({\mathrm{\&theta;}}_k\right|I\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right))\&CenterDot;{\stackrel{\&RightArrow;}{n}}_{k\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right)}$

Wherein,

It is τ k in pointDirection vector, point

Which region belonged to, depending on point

It is adapted to the degree described by provincial characteristics probability Distribution Model； 

To determine each pixel and interregional subordinate relation, the image segmentation algorithm process description based on competition mechanism is as follows： 

In initial phase, the initial parameter of each class model is estimated according to the characteristic image block matched, the boundary point of all characteristic image blocks is added in queue undetermined, and calculate the posterior probability that all boundary points belong to all kinds of； 

In the loop iteration stage, the boundary point i that present energy steepest declines is selected from queue undetermined, and then update all borders where boundary point i；Then under current cutting state, the model parameter in each region is recalculated using maximal possibility estimation；Using each provincial characteristics distributed model newly obtained, the posterior probability that all boundary points belong to all kinds of is recalculated； 

So, the boundary point for constantly selecting present energy steepest to decline from queue undetermined updates corresponding border, update the feature distribution probabilistic model in each region according to current region segmentation state in good time simultaneously, multiple regions are mutually restricted, the ownership of image-region is competed simultaneously, until energy function convergence, so as to divide the image into as multiple regions. 

The stylizing method step 4 of the stylized module of above-mentioned video) video stylization is based on interactive video semanteme segmentation module, and the selection of paintbrush is only determined by the material corresponding to the object area that identifies； 

Above-mentioned paintbrush is all based on professional artist and a large amount of typical styles of writing is drawn on paper, then it is scanned and parameterizes, finally set up style of writing storehouse, drawn for each image-region, bottoming is carried out using big brush first, then brush size and opacity are gradually decreased to carry out meticulous depiction to the detail section of object, during drafting, using drafting strategy internal behind first edge：The drafting present invention of each tomographic image is drawn, and brush is alignd according to flow field first first since edge along the edge of line drawing； 

In Video Rendering, in order to ensure stability of the paintbrush in time domain, the propagation of style of writing is carried out using thin-plate spline interpolation technology, in addition, style of writing is in communication process, also by calculating the area in style of writing region, style of writing is devised and deletes and increase mechanism；And using damping spring system is simulated, reduce " shake " effect of rendering result. 

The stylizing method step 5 of the stylized module of above-mentioned video) the key frame non-photorealistic rendering method based on semanteme parsing it is as follows： 

How to design different artistic style style of writing models is one of stylized focus of attention of video, the works of different artistic expressions, differ from one another in style of writing expression, basic drafting strategy in video stylization is that the suitable style of writing of image content-based selection is drawn, style of writing storehouse is that a large amount of typical styles of writing are drawn on paper based on professional artist, then it is scanned and parameterizes, finally complete foundation, for the paintbrush B that will be drawn_nInclude following information：The classification information I of brush_n, placement area scope Λ_n, color mapping C_n, the α of transparency_n, height field H_nAnd control point { P_ni, that is, have： 

B_n={ I_n, Λ_n, C_n, α_n, H_n, { P_ni}} 

When designing style of writing model, the low level informations such as style of writing shape, texture are not only allowed for, while also combine its high-layer semantic information, so that in render process, each interpretation region of image/video has " pen " can be according to；When choosing style of writing, to interpret area classification as keyword, a collection of style of writing with identical category is selected from style of writing storehouse with simple and fast.And then a therefrom optional style of writing in a random basis； 

" alignment " principle in being drawn for simulation oil painting, uses for reference original simple model theory, in each region R_iIt is interior, calculate its original brief figure SK_iExpression.Brief figure is made up of the conspicuousness primitive of a group echo body surface feature, such as the spot on clothes, lines, gauffer；In render process, different paintbrush will be covered on these primitives to produce desired artistic effect；Interpret region R_i, R_i∈Λ_iIt is divided into the line drawing part for describing line drawing

And for describing the non-line drawing part with identical structural region

R_iField of direction θ x are defined as： 

${\mathrm{\&Theta;}}_i=\left\{\mathrm{\&theta;}(x,y)\right|\mathrm{\&theta;}(x,y)\&Element;[0,\mathrm{\&pi;}),\&ForAll;(x,y)\&Element;{\mathrm{\&Lambda;}}_i\}$

Wherein field of direction θ_iInitial value is line drawingGradient direction.Then direction is traveled to non-line drawing region using diffusion equation

Render process to key frame is the continuous process chosen style of writing and put style of writing；To interpret region R_iExemplified by, its non-line drawing part is rendered first

Then line drawing part is rendered

This is in order to ensure when the region rendered overlaps, the style of writing of line drawing part can be in upper strata；In non-line drawing part, optional one pixel region that is not rendered, using the center in the region as originating point, spreads along the field of direction to both sides, generates a flow pattern regions；The line on the basis of the axis in the region, the paintbrush chosen is transformed in the flow pattern regions, style of writing axis is alignd with region axis；It is similar to rendering for region line drawing part. 

The stylizing method step 5 of the stylized module (2) of above-mentioned video) sequence frame style of writing transmission method it is as follows： 

Rendering for non-key frame is obtained by the rendering result " propagation " of key frame, the foundation of propagation is the space-time corresponding relation for interpreting region, in communication process, as the change in interpretation region is increasing, style of writing may gradually be leaked to region exterior, and the space with occurring in time domain to be rendered, so, in style of writing figure is propagated, it is necessary to while considering the addition of style of writing and deleting mechanism, otherwise, the jitter phenomenon that rendering result occurs；The propagation of style of writing, add and to delete mechanism as follows： 

(a) style of writing is propagated：C is made to represent some interpretation region, R of video t key frame_i(t+1) R is represented_i(t) in t+1 moment corresponding region.Their image-region is respectively with Λ_i(t)、Λ_i(t+1) represent.With P_ij(t)、P_ij(t+1) Λ is represented_i(t), dense matching points (video interpretation during calculate) of the Λ x (t+1) in time domain.Assuming that R_i(t+1) table can pass through R_i(t) non-rigid transformation of table is obtained.When style of writing is propagated, the present invention wishes Λ_i(t) the match point P on_ij(t) image-region Λ new in t+1 frames can be mapped to_i(t+1) match point P_ij(t+1).Based on considerations above, the present invention is from thin-plate spline interpolation model (Thin-plate Spline, TPS).It can be Λ_i(t) key point P in_ij(t) it is mapped to Λ_i(t+1) match point P_ij(t+1), and for Λ_i(t) pixel of the non-key point of remaining in, TPS makes Λ by minimizing energy function_i(t) pixel grid occurs (non-rigid) deformation of elasticity and distorted. 

(b) style of writing is deleted：After being propagated in video due to paintbrush or when there occurs hiding relation or too many style of writing propagation frame number, region corresponding to some paintbrush can become less and less, therefore, the present invention will reject these paintbrush when their corresponding region areas are less than some given threshold value.Equally, also to be deleted when the paintbrush of propagation falls outside corresponding zone boundary. 

(c) style of writing increases.When there is new semantic region or already present semantic region becomes increasing (expansion of such as clothes), the present invention must increase new paintbrush to cover these emerging regions, and in order to fill the space between paintbrush, the present invention only need to simply change size and the position of adjacent paintbrush.If the region not covered by paintbrush becomes bigger and bigger and exceeded some given threshold value, system can be automatically created new paintbrush to cover it.Nevertheless, the present invention is still unlikely to draw one to it at once when space occurs for the first time.Then, comparatively the present invention is provided with compares high threshold value, and postpones to render emerging region when they rise to sufficiently large.Then, the present invention puts algorithm to be filled up to the sufficiently large space of threshold value using general paintbrush, finally oppositely propagates and converts again these new paintbrush and go to fill void area that is previously having occurred but not rendering.The process of filling paintbrush can avoid continually converting paintbrush backward, while less some scrappy paintbrushes can be linked as larger paintbrush again, so as to reduce scintillation effect and other undesirable visual effects artificially caused.Similarly, since the present invention is that new paintbrush is added in the bottom, so they are drawn in below the paintbrush existed, This further reduces visual scintillation effect. 

The stylizing method step 6 of the stylized module of above-mentioned video) in be used for stabilization damping brush system it is as follows： 

It is stabilization operation to the final step that video carries out stylized rendering, paintbrush adjacent in time domain and spatial domain is attached with spring, to simulate damping system；By the energy for minimizing the system, it is possible to reach the effect for removing shake； 

For i-th of paintbrush of t, present invention A_{I, t}=(x_{I, t}, y_{I, t}, s_{I, t}) its centre coordinate and the geometric attribute of size are represented, and its initial value is designated as

The energy function of damping brush system is defined as follows： 

E=E_data+λ₁E_smooth1+λ₂E_smooth2

λ₁And λ₂For weight, λ₁=2.8, λ₂=1.1； 

The first item constraint paintbrush position will can not be partially too far away with initial position in formula： 

$E_{\mathrm{data}}=\underset{i,t}{\mathrm{\&Sigma;}}{(A_{i,t}-A_{i,t}^0)}^2$

Section 2 is the smoothness constraint carried out to paintbrush i in time domain in formula： 

$E_{\mathrm{smooth}1}=\underset{i,t}{\mathrm{\&Sigma;}}{(A_{i,t+1}-{2A}_{i,t}+A_{i,t-1})}^2$

Section 3 all carries out smoothness constraint to adjacent paintbrush in time domain and spatial domain in formula；Note

For the adjacent paintbrush of i-th of paintbrush of t, for arbitrary neighborhood paintbrush

By the relative distance difference and difference in size between them, Δ A is designated as_{I, j}, t=A_{I, t}-A_{J, t}, and smooth item is defined as below： 

$E_{\mathrm{smooth}1}=\underset{i,t}{\mathrm{\&Sigma;}}{({\mathrm{\&Delta;A}}_{i,j,t+1}-{2\mathrm{\&Delta;A}}_{i,j,t}+\mathrm{\&Delta;}{A}_{i,j,t-1})}^2$

Pass through Levenbergy-Marquard Algorithm for Solving energy minimization problems. 

Above-mentioned λ₁=2.8, λ₂=1.1. 

Segmentation, identification and the foundation of space-time corresponding relation of the invention by studying video, inquires into the video stylized rendering technology of semantics-driven, reaches the expression effect of art needs.The present invention is by since the semantic analysis research of input video, using the interactive mode based on key frame, while farthest reduction burden for users, sufficient prior information is provided for Video segmentation, then by setting up the feature point correspondence between frame and frame, the interactive information on key frame is propagated to subsequent frame using region-competitive algorithm so that user semantic information can sufficiently instruct accurate Video segmentation.And for different-style, create different style of writing storehouses.When rendering, key frame is rendered according to semantic information first, then using the time-space relationship of semantic region as constraint, the style of writing of key frame is traveled in sequence frame by spatial alternation, so as to effectively suppress " shake " effect of rendering result.In addition, present invention further propose that being easy to the system schema that user mutual is created, so as to improve the applicability of this project.The various industries such as advertisement, education, amusement are present invention can be extensively applied to, with important application background. 

Embodiment

Embodiment： 

The interactive video stylized rendering method that the present invention is interpreted based on video, includes the stylized module of interactive video semantic segmentation module, video, and the dividing method of interactive video semantic segmentation module comprises the following steps： 

1) Interactive Segmentation and automatic identification of key frame images； 

2) between key frame dense characteristic point matching； 

3) region-competitive partitioning algorithm； 

The stylizing method of video stylization module comprises the following steps： 

4) the key frame non-photorealistic rendering based on semanteme parsing； 

5) the style of writing transmission method of sequence frame； 

6) it is used for the damping brush system of stabilization. 

Stylization to video will use the two modules successively.It is i.e. first that semantic segmentation is carried out to video using interactive semantic segmentation module.Reuse video stylized module and stylized rendering is carried out to the video after segmentation.Step 1 in above-mentioned interactive video semantic segmentation module 1) key frame images Interactive Segmentation and automatic identifying method it is as follows： 

In the present invention, the identification technology TextonBoost and interactive segmentation method GraphCut of comprehensive more maturation interact semantic segmentation and the identification of formula to key frame images, so as to obtain the object area in image and mutually be layered and hiding relation.They divide into 12 classes, including sky by the semantic region of segmentation by present system according to its different material properties, water, land, rock, hair, skin, clothes etc., as shown in table 1. 

Table 1：12 kinds of material classifications of semantic region

Mountain range	Water	Rock/building	Leaf/grove
				Skin/hide	Hair/hair	Flower/fruit	Sky/cloud
Clothes	Trunk/branch	Abstract background	Wood/plastics

In practical operation, it is trained and recognizes present invention employs texture, distribution of color, positional information these three principal characters.An area image X is given, the conditional probability for defining its classification c is： 

$\log P\left(x\right|X,\mathrm{\&theta;})=\underset{i}{\mathrm{\&Sigma;}}{\mathrm{\&Psi;}}_i(c_i,X;{\mathrm{\&theta;}}_{\mathrm{\&Psi;}})+\mathrm{\&pi;}(c_i,X;{\mathrm{\&theta;}}_{\mathrm{\&pi;}})+\mathrm{\&lambda;}(c_i,X;{\mathrm{\&theta;}}_{\mathrm{\&lambda;}})-\log Z(\mathrm{\&theta;},X)$

Four are texture potential-energy function, color potential-energy function, position potential-energy function and normalization item respectively after in formula. 

Texture potential-energy function is defined as Ψ_i(c_i, X；θ_Ψ)=logP (c_i| X,_i), P (c_i| X,_i) it is the normalized distribution function given by Boost graders. 

Color potential-energy function is defined as π (c_i, X；θ_π)=log ∑s_kθ_π(c_i, k) P (k | x_i), gauss hybrid models (Gaussian Mixture Models of the present invention in CIELab color spaces here：GMMs color model) is represented, to a pixel color x in given image, its conditional probability is： Wherein μ_kRepresent the average and variance of k-th of color cluster respectively with ∑ k. 

Position potential-energy function is defined as λ (c_i, X；θ_λ)=log θ_λ(c_i, i), for above two potential-energy functions, position potential-energy function is relatively weak, and in the definition of this function, the category label of image pixel is only related to absolute position in the picture. 

It is trained using the method for 12 class materials, all pixels in the probability for each classification for giving each pixel in an image-region, last statistical regions is then calculated using above formula, the classification in each region is determined by the way of ballot.During stylized rendering, the material that the selection of paintbrush is identified by object area is determined, to realize that automatic render lays the foundation. 

2) between key frame dense characteristic point matching

After semantic information on key frame is obtained, semantic information is effectively traveled to sequence frame up by the matching algorithm that the present invention needs to explore between a kind of frame. 

The present invention proposes comprehensive line drawing, texture and blend of colors image template feature first, and the characteristic set enriched and expression are provided for image matching problems. 

(a) line drawing feature is shown as by Gabor base tables： F^sk(I_i)=| |<I_i, G_{Cos, x, θ}>||²+||<I_i, G_{Sin, x, θ}>||², G_{Sin, x, θ}And G_{Cos, x, θ}Position x prescriptions are illustrated respectively in the sine and cosine Gabor bases for θ.The distribution of its characteristic probability is expressed as： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{sk}}\right)}\exp \left\{{\mathrm{\&lambda;}}_i^{\mathrm{sk}}{h}^{\mathrm{sk}}\right[F^{\mathrm{sk}}\left(I_i\right)\left]\right\}$

Represent parameter θ_i, h^skIt is a sigmoid function,

It is normalization constraints. 

So model will encourage have stronger corresponding edge than background distributions. 

(b) textural characteristics are modeled with a simplified gradient orientation histogram (HOG) to textural characteristics, and 6 characteristic dimensions represent different gradient directions respectively. HOG j-th of direction is represented, and

Represent corresponding description of ith feature Ii. 

It is F^txt(I_i) average in all positive samples.The probabilistic model of feature is expressed as by the present invention： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{txt}}\right)}\exp \left\{{\mathrm{\&lambda;}}_i^{\mathrm{txt}}{\mathrm{\&Sigma;}}_j{h}_j^{\mathrm{txt}}\right[F^{\mathrm{txt}}\left(I_i\right)\left]\right\}$

It is parameter θ_i.It can be seen that model encourages to respond the set for the characteristic image block relatively concentrated. 

(c) color characteristic is to be used as description using simple pixel intensity. 

It is the wave filter on the x of position.Quantizing pixel brightness value of the present invention is interval to each statistics, then model can be reduced to： 

$\frac{P\left(I_i\right|B_i;{\mathrm{\&theta;}}_i)}{q\left(I_i\right)}=\frac{1}{Z\left({\mathrm{\&lambda;}}_i^{\mathrm{fl}}\right)}\exp \left\{{\mathrm{\&Sigma;}}_j{\mathrm{\&lambda;}}_i^{\mathrm{fl}}{h}_{\mathrm{xj}}^{\mathrm{fl}}\right[F_{\mathrm{xj}}^{\mathrm{fl}}\left(I_i\right)\left]\right\}$

The present invention is by combining the small feature of similar image, it is possible to receive the local combinations of features with strong judgement index.Over-segmentation is carried out to image first, the tiny image block of some in image is obtained.Line drawing can be described by being extracted from small image block, texture, the statistical nature of color.In order to effectively obtain combinations of features, increased and model-learning algorithm using the region of iteration, by constantly updating characteristic model, iteration growth feature combination zone finally gives the local combinations of features with strong judgement index. 

On the basis of expressing herein, the present invention is modeled as matching problem of the moving target in time domain and spatial domain the hierarchical diagram the matching frame on figure is represented.The mixed image template characteristic of extraction is as node of graph, and the side annexation between graph structure, node of graph is built between frames to be defined with the affiliated object type of the similarity between feature based, locus, and feature. 

With Is, It represents artwork and target figure, and U, V represent hybrid template characteristic set in Is, It respectively.To each characteristic point u ∈ U ', there are two marks：Level mark I (u) ∈ { 1,2 ..., K } and matching candidate mark

With the higher Candidate Set C of each Feature Points Matching degree in artwork, the vertex set of graph structure is set up.With E=E⁺∪E^-Build line set.Represent that the candidate of connection is mutually exclusive with negative side, and define its " repulsive force " and be： 

With the adjacent and not mutually exclusive candidate feature point of positive side connection space, order

The tightness degree cooperated between them is represented,Represent v_i, v_jBetween space length. 

By original image and the graph structure G of target image^s、G^T, K+1 layers are divided into, wherein K represents the object number in artwork.With G^sExemplified by, division is expressed as ∏={ g₀, g₁..., g_k}.Wherein, g_kIt is G^sA subgraph, its vertex set is with U_kRepresent.Similar, G^TVertex set with V_kRepresent.Then G^sAnd G^TBetween matching relationship be expressed as

Assuming that the matching between subgraph is separate, then： 

In the present invention, estimated with geometric transformation, outward appearance and define Matching sub-image to (g_k, g_k') between similarity measure, useRepresent.In summary, the solution of graph structure matching problem is configurable to： 

W=(K, ∏={ g₀, g₁..., g_k, Ψ={ Φ_k, Φ={ Φ_k}) 

Under bayesian theory framework, the present invention describes graph structure matching problem to maximize posterior probability： 

W^*=argmaxp (W | G^s, G^T)=argmaxp (W) p (G^s, G^T|W) 

The present invention can be solved by markov chain Monte-Carlo (MCMC) method to above formula.Simultaneously in order to efficiently calculate, the present invention explores cluster sampling policy, by efficiently redirecting in solution space, globally optimal solution is rapidly converged to, to reach the matching of interframe characteristic point. 

(1) region-competitive partitioning algorithm

On the basis of the matching relationship of interframe stabilization is obtained, by the advantage in Video segmentation of excavation regions competition mechanism, the present invention proposes the region-competitive propagation algorithm based on frame matching.Utilize the image matching algorithm for being layered graph structure, present invention may determine that the matching relationship between former frame and present frame feature, the semantic information of former frame is traveled in present frame, then region-competitive partitioning algorithm is utilized according to the characteristic information of each matching area, present frame is divided into multiple semantic regions. 

Given image I, corresponding image segmentation solution is defined as follows： 

W={ (R₁, R₂... R_N), (θ₁, θ₂..., θ_N), (I₁, I₂..., I_N)} 

Wherein, R_iThe region with same characteristic being partitioned into is represented,

θ_iRepresent region R_iThe parameter of corresponding characteristic probability distributed model, I_iRepresent region R_iCorresponding mark. 

According to the matching relationship of feature in front and rear frame, it may be determined that cut zone number N.If the feature zonule set S={ S corresponding to each region₁, S₂..., S_N, for each region R_i, the zonule S occupied according to feature_iEstimate the initial parameter θ of model_i, obtain initial posterior probability P (θ_i| I (x, y)).According to MDL principles, posterior probability is converted into solution energy function minimum problem, obtained： 

$\log P\left(W\right|I)=E[\mathrm{\&Gamma;},\{{\mathrm{\&theta;}}_i\left\}\right]=\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}-{\&Integral;\&Integral;}_{R_1}\log P\left({\mathrm{\&theta;}}_i\right|I(x,y))\mathrm{dxdy}$

Wherein

Represent region R_iBoundary profile.The present invention estimates parameter { θ stage by stage using iterative manner_iAnd Γ, alternating iteration two benches, constantly reduce energy function in the various stages, so that constantly Reasoning With Learning goes out the final segmentation result of entire image. 

During region-competitive, its characteristic probability distributed model is constantly updated in each region, while declining the ownership that principle fights for pixel according to steepest, updates respective boundary profile so that scope is constantly expanded in each region, finally gives the image segmentation result of present frame. 

Its specific iterative step is：First stage, fixed Γ, according to current region segmentation state estimation { θ_i, try to achieve parameter θ under current state_iMaximal possibility estimation be used as its optimal solution

To minimize the cost for describing each region, therefore energy function is converted into： 

${\mathrm{\&theta;}}_i^{*}={\arg \max }_{{\mathrm{\&theta;}}_i}\left\{{\&Integral;\&Integral;}_R\log P\left({\mathrm{\&theta;}}_i\right|I(x,y))\mathrm{dxdy}\right\},\&ForAll;i\&Element;[1,N]$

Second stage, { θ_i, it is known that carrying out steepest decline to Γ, in order to quickly try to achieve the minimal solution of energy function, the present invention solves the equation of motion that steepest declines to the border Γ in all regions.For any point on boundary profile Γ

Have

$\frac{d\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}}{\mathrm{dt}}=-\frac{\mathrm{\&delta;E}(\mathrm{\&Gamma;},\{{\mathrm{\&theta;}}_i\left\}\right)}{\mathrm{\&delta;}\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}}={\mathrm{\&Sigma;}}_{k\&Element;Q\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right)}\log P\left({\mathrm{\&theta;}}_k\right|I\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right))\&CenterDot;{\stackrel{\&RightArrow;}{n}}_{k\left(\stackrel{\&RightArrow;}{\mathrm{\&upsi;}}\right)}$

Wherein,

It is τ k in point

Direction vector.Point

Which region belonged to, depending on point

It is adapted to the degree described by provincial characteristics probability Distribution Model. 

To determine each pixel and interregional subordinate relation, the present invention proposes the image segmentation algorithm based on competition mechanism to be rapidly completed image segmentation.The specific image segmentation algorithm process description based on competition mechanism is as follows： 

In initial phase, the initial parameter of each class model is estimated according to the characteristic image block matched, the boundary point of all characteristic image blocks is added in queue undetermined, and calculate the posterior probability that all boundary points belong to all kinds of. 

In the loop iteration stage, the boundary point i that present energy steepest declines is selected from queue undetermined, and then update all borders where boundary point i；Then under current cutting state, the model parameter in each region is recalculated using maximal possibility estimation；Using each provincial characteristics distributed model newly obtained, the posterior probability that all boundary points belong to all kinds of is recalculated. 

So, the boundary point for constantly selecting present energy steepest to decline from queue undetermined updates corresponding border, update the feature distribution probabilistic model in each region according to current region segmentation state in good time simultaneously, multiple regions are mutually restricted, the ownership of image-region is competed simultaneously, until energy function convergence, so as to divide the image into as multiple regions. 

1. video stylization module

Video stylization is based on interactive video semanteme segmentation module.The selection of paintbrush is only determined by the material corresponding to the object area that identifies.The paintbrush of present system is all based on professional artist and a large amount of typical styles of writing is drawn on paper, is then scanned and parameterizes, finally sets up style of writing storehouse.Drawn for each image-region, bottoming is carried out using big brush first, then gradually decrease brush size and opacity to carry out meticulous depiction to the detail section of object.During drafting, using drafting strategy internal behind first edge：The drafting present invention of each tomographic image is drawn, and brush is alignd according to flow field first first since edge along the edge of line drawing.In Video Rendering, in order to ensure stability of the paintbrush in time domain, the present invention carries out the propagation of style of writing using thin-plate spline interpolation technology.In addition, style of writing is in communication process, also by calculating the area in style of writing region, devises style of writing and delete and increase mechanism.And using damping spring system is simulated, reduce " shake " effect of rendering result. 

(1) the key frame non-photorealistic rendering technology based on semanteme parsing

How to design different artistic style style of writing models is one of stylized focus of attention of video.The works of different artistic expressions, differ from one another in style of writing expression.Basic drafting strategy of the invention is that the suitable style of writing of image content-based selection is drawn in video stylization, and style of writing storehouse is that a large amount of typical styles of writing are drawn on paper based on professional artist, is then scanned and parameterizes, finally completes foundation.For the paintbrush B that will be drawn_nInclude following information：The classification information l of brush_n, placement area scope Λ_n, color mapping C_n, the α of transparency_n, height field H_nAnd control point { P_ni, that is, have： 

B_n={ I_n, Λ_n, C_n, α_n, H_n, { P_ni}} 

When designing style of writing model, the present invention not only allows for the low level informations such as style of writing shape, texture, while also combining its high-layer semantic information.So as in render process, each interpretation region of image/video has " pen " can be according to.This is one of Rendering algorithms of the present invention key different from the past based on pen-contact type Rendering algorithms.Thus when choosing style of writing, to interpret area classification as keyword, a collection of style of writing with identical category can be selected from style of writing storehouse with simple and fast.And then a therefrom optional style of writing in a random basis. 

" alignment " principle in being drawn for simulation oil painting, the present invention uses for reference original simple model theory, in each region R_iInterior, the present invention calculates its original brief figure SK_iExpression.Brief figure is made up of the conspicuousness primitive of a group echo body surface feature, such as the spot on clothes, lines, gauffer etc..In render process, different paintbrush will be covered on these primitives to produce desired artistic effect.Interpret region R_i, R_i∈Λ_iIt is divided into the line drawing part for describing line drawing

And for describing the non-line drawing part with identical structural region

R_iField of direction θ_iIt is defined as： 

${\mathrm{\&Theta;}}_i=\left\{\mathrm{\&theta;}(x,y)\right|\mathrm{\&theta;}(x,y)\&Element;[0,\mathrm{\&pi;}),\&ForAll;(x,y)\&Element;{\mathrm{\&Lambda;}}_i\}$

Wherein field of direction θ_iInitial value is line drawing

Gradient direction.Then direction is traveled to non-line drawing region using diffusion equation

Render process to key frame is the continuous process chosen style of writing and put style of writing.To interpret region R_iExemplified by, the present invention renders its non-line drawing part first

Then line drawing part is rendered

This is in order to ensure when the region rendered overlaps, the style of writing of line drawing part can be in upper strata.In non-line drawing part, optional one pixel region that is not rendered, using the center in the region as originating point, spreads along the field of direction to both sides, generates a flow pattern regions.The line on the basis of the axis in the region, the paintbrush chosen is transformed in the flow pattern regions, style of writing axis is alignd with region axis.It is similar to rendering for region line drawing part. 

(2) the style of writing propagation algorithm of sequence frame

In the present invention, rendering for non-key frame is obtained by the rendering result " propagation " of key frame.The foundation of propagation is the space-time corresponding relation for interpreting region.In communication process, as the change in interpretation region is increasing, style of writing may gradually be leaked to region exterior, and the space with occurring in time domain to be rendered.So, in style of writing figure is propagated, it is necessary to while considering the addition of style of writing and deleting mechanism.Otherwise, the jitter phenomenon that rendering result occurs.The present invention describes the propagation of style of writing, adds and delete mechanism respectively below. 

(d) style of writing is propagated：C is made to represent some interpretation region, R of video t key frame_i(t+1) R is represented_i(t) in t+1 moment corresponding region.Their image-region is respectively with Λ_i(t)、Λ_i(t+1) represent.With P_ij(t)、P_ij(t+1) Λ is represented_i(t)、Λ_i(t+1) dense matching point (being calculated during video interpretation) in time domain.Assuming that R_i(t+1) table can pass through R_i(t) non-rigid transformation of table is obtained.When style of writing is propagated, the present invention wishes Λ_i(t) the match point P on_ij(t) image-region Λ new in t+1 frames can be mapped to_i(t+1) match point P_ij(t+1).Based on considerations above, the present invention is from thin-plate spline interpolation model (Thin-plate Spline, TPS).It can be Λ_i(t) key point P in_ij(t) it is mapped to Λ_i(t+1) match point P_ij(t+1), and for Λ_i(t) pixel of the non-key point of remaining in, TPS makes Λ by minimizing energy function_i(t) pixel grid occurs (non-rigid) deformation of elasticity and distorted. 

(e) style of writing is deleted：After being propagated in video due to paintbrush or when there occurs hiding relation or too many style of writing propagation frame number, region corresponding to some paintbrush can become less and less, therefore, the present invention will reject these paintbrush when their corresponding region areas are less than some given threshold value.Equally, also to be deleted when the paintbrush of propagation falls outside corresponding zone boundary. 

(f) style of writing increases.When there is new semantic region or already present semantic region becomes increasing (expansion of such as clothes), the present invention must increase new paintbrush to cover these emerging regions, and in order to fill the space between paintbrush, the present invention only need to simply change size and the position of adjacent paintbrush.If the region not covered by paintbrush becomes bigger and bigger and exceeded some given threshold value, system can be automatically created new paintbrush to cover it.Nevertheless, the present invention is still unlikely to draw one to it at once when space occurs for the first time.Then, comparatively the present invention is provided with compares high threshold value, and postpones to render emerging region when they rise to sufficiently large.Then, the present invention puts algorithm to be filled up to the sufficiently large space of threshold value using general paintbrush, finally oppositely propagates and converts again these new paintbrush and go to fill void area that is previously having occurred but not rendering.The process of filling paintbrush can avoid continually converting paintbrush backward, while less some scrappy paintbrushes can be linked as larger paintbrush again, so as to reduce scintillation effect and other undesirable visual effects artificially caused.Similarly, since the present invention is that new paintbrush is added in the bottom, so they are drawn in below the paintbrush existed, This further reduces visual scintillation effect. 

(3) it is used for the damping brush system of stabilization

It is stabilization operation to the final step that video carries out stylized rendering.The present invention is attached to paintbrush adjacent in time domain and spatial domain with spring, to simulate damping system.By the energy for minimizing the system, it is possible to reach the effect for removing shake. 

For i-th of paintbrush of t, present invention A_{I, t}=(x_{I, t}, y_{I, t}, s_{I, t}) its centre coordinate and the geometric attribute of size are represented, and its initial value is designated as

The energy function of damping brush system is defined as follows： 

E=E_data+λ₁E_smooth1+λ₂E_smooth2

λ₁And λ₂For weight, in an experiment, the present invention is set to λ₁=2.8, λ₂=1.1. 

The first item constraint paintbrush position will can not be partially too far away with initial position in formula： 

$E_{\mathrm{data}}=\underset{i,t}{\mathrm{\&Sigma;}}{(A_{i,t}-A_{i,t}^0)}^2$

Section 2 is the smoothness constraint carried out to paintbrush i in time domain in formula： 

$E_{\mathrm{smooth}1}=\underset{i,t}{\mathrm{\&Sigma;}}{(A_{i,t+1}-{2A}_{i,t}+A_{i,t-1})}^2$

Section 3 all carries out smoothness constraint to adjacent paintbrush in time domain and spatial domain in formula.Note

For the adjacent paintbrush of i-th of paintbrush of t, for arbitrary neighborhood paintbrush

By the relative distance difference and difference in size between them, Δ A is designated as_{I, j, t}=A_{I, t}-A_{J, t}, and smooth item is defined as below： 

$E_{\mathrm{smooth}1}=\underset{i,t}{\mathrm{\&Sigma;}}{({\mathrm{\&Delta;A}}_{i,j,t+1}-{2\mathrm{\&Delta;A}}_{i,j,t}+\mathrm{\&Delta;}{A}_{i,j,t-1})}^2$

Pass through Levenbergy-Marquard Algorithm for Solving energy minimization problems. 

Claims (9)

1. a kind of interactive video stylized rendering method interpreted based on video, it is characterised in that include the stylized module of interactive video semantic segmentation module, video.

The dividing method of interactive video semantic segmentation module comprises the following steps：

1) Interactive Segmentation and automatic identification of key frame images；

2) between key frame dense characteristic point matching；

3) region-competitive is split；

The stylizing method of video stylization module comprises the following steps：

1) the key frame non-photorealistic rendering based on semanteme parsing；

2) style of writing of sequence frame is propagated；

3) handled with the damping brush system of stabilization.

Stylization to video will be successively using interactive video semantic segmentation module and the two modules of the stylized module of video, semantic segmentation first is carried out to video using interactive video semantic segmentation module, video stylized module is reused and stylized rendering is carried out to the video after segmentation.

2. the interactive video stylized rendering method according to claim 1 interpreted based on video, it is characterised in that the Interactive Segmentation and automatic identifying method of the key frame images of above-mentioned steps are as follows：

They be divide into 12 classes by the semantic region of segmentation according to its different material properties, including sky/cloud, mountain range/land, rock/building, leaf/grove, hair/hair, flower/fruit, skin/hide, trunk/branch, abstract background, wood/plastics, water, clothes；

In practical operation, employ texture, distribution of color, positional information these three principal characters and be trained and recognize, give an area image X, the conditional probability for defining its classification c is： 

(formula 1)

Four are texture potential-energy function, color potential-energy function, position potential-energy function and normalization item respectively after in formula.

Texture potential-energy function is defined as Ψ_i(c_i, X；θ_Ψ)=logP (c_i| X, i), P (c_i| X, is the normalized distribution function given by Boost graders i)；

Color potential-energy function is defined as π (c_i, X；θ_n)=log ∑s_kθ_n(c_i, k) P (k | x_i), with gauss hybrid models (the Gaussian Mixture Models in CIELab color spaces：GMMs color model) is represented, to a pixel color x in given image, its conditional probability is： 

Wherein μ_kRepresent the average and variance of k-th of color cluster respectively with ∑ k；

Position potential-energy function is defined as λ (c_i, X；θ_λ)=log θ₂(c_i, i), for above two potential-energy functions, position potential-energy function is relatively weak, and in the definition of this function, the category label of image pixel is only related to absolute position in the picture；

It is trained using the method for 12 class materials, all pixels in the probability for each classification of each pixel in an image-region, last statistical regions is then given using formula 1, the classification in each region is determined by the way of ballot；During stylized rendering, the material that the selection of paintbrush is identified by object area is determined, to realize that automatic render lays the foundation.

3. it is according to claim 1 based on video interpret interactive video stylized rendering method, it is characterised in that above-mentioned steps 2) key frame between dense characteristic point matching process it is as follows：

After semantic information on key frame is obtained, comprehensive line drawing feature, texture and blend of colors image template feature provide abundant characteristic set and expression for image matching problems；

11) line drawing feature is shown as by Gabor base tables： 

F^sk(I_i)=| |<I_i, G_{Cos, x, θ}>||²+||<I_i, G_{Sin, x, θ}, G_iAnd G_cBe illustrated respectively in position x prescriptions to for sine and cosine Gabor bases.The distribution of its characteristic probability is expressed as：

Represent parameter θ_i, h^skIt is a sigmoid function,

It is normalization constraints.

So model will encourage have stronger corresponding edge than background distributions；

12) textural characteristics are modeled with a simplified gradient orientation histogram (HOG) to textural characteristics, and 6 characteristic dimensions represent different gradient directions respectively；HOG j-th of direction is represented, and

Represent corresponding description of ith feature； 

It is F^tAverage in all positive samples.The probabilistic model of feature is expressed as by the present invention：

It is parameter θ_i.It can be seen that model encourages to respond the set for the characteristic image block relatively concentrated；

13) color characteristic be using simple pixel intensity as description,

Quantizing pixel brightness value of the present invention is interval to each statistics, then model can be reduced to：

The small feature of similar image by combining, the local combinations of features with strong judgement index can be just received, over-segmentation is carried out to image first, the tiny image block of some in image is obtained, line drawing can be described by being extracted from small image block, texture, the statistical nature of color, in order to effectively obtain combinations of features, increased and model-learning algorithm using the region of iteration, by constantly updating characteristic model, iteration growth feature combination zone finally gives the local combinations of features with strong judgement index；

On the basis of expressing herein, matching problem of the moving target in time domain and spatial domain is modeled as the hierarchical diagram the matching frame on figure is represented, the mixed image template characteristic of extraction is used as node of graph, graph structure is built between frames, side annexation between node of graph can be defined with the affiliated object type of the similarity between feature based, locus, and feature；

With Is, It represents artwork and target figure, and U, V represent hybrid template characteristic set in Is, It respectively, has two marks to each characteristic point u ∈ U '：Level mark I (u) ∈ { 1,2 ..., K } and matching candidate mark

With the higher Candidate Set C of each Feature Points Matching degree in artwork, the vertex set of graph structure is set up, with E=E⁺∪E^-Build line set.Represent that the candidate of connection is mutually exclusive with negative side, and define its " repulsive force " and be：

With the adjacent and not mutually exclusive candidate feature point of positive side connection space, order

The tightness degree cooperated between them is represented,

Represent v_i, v_iBetween space length；

By original image and the graph structure G of target image^s、G^T, K+1 layers are divided into, wherein K represents the object number in artwork, with G^sExemplified by, division is expressed as ∏={ g₀, g₁..., g_k}.Wherein, g_kIt is G^sA subgraph, its vertex set is with U_kRepresent.Similar, G^TVertex set with V_kRepresent.Then G^sAnd G^TBetween matching relationship be expressed asAssuming that the matching between subgraph is separate, then：

Estimated with geometric transformation, outward appearance and define Matching sub-image to (g_k, g_k') between similarity measure, use

Represent；In summary, the solution of graph structure matching problem is configurable to：

W=(K, ∏={ g₀, g₁..., g_k, Ψ={ Φ_k, Φ={ Φ_k})

Under bayesian theory framework, graph structure matching problem is described to maximize posterior probability：

W^*=argmaxp (W | G^s, G^T)=argmaxp (W) p (G^s, G^T|W)

Above formula is solved by markov chain Monte-Carlo (MCMC) method, while in order to efficiently calculate, by efficiently redirecting in solution space, globally optimal solution is rapidly converged to, to reach the matching of interframe characteristic point.

4. it is according to claim 1 based on video interpret interactive video stylized rendering method, it is characterised in that above-mentioned steps 3) region-competitive dividing method it is as follows：

On the basis of the matching relationship of interframe stabilization is obtained, pass through the advantage in Video segmentation of excavation regions competition mechanism, utilize the image matching algorithm for being layered graph structure, the matching relationship between former frame and present frame feature can be determined, the semantic information of so former frame is just traveled in present frame, then region-competitive partitioning algorithm is utilized according to the characteristic information of each matching area, present frame is divided into multiple semantic regions；

Given image I, corresponding image segmentation solution is defined as follows：

W={ (R₁, R₂... R_N), (θ₁, θ₂..., θ_N), (I₁, I₂..., I_N)}

Wherein, R_iThe region with same characteristic being partitioned into is represented,

θ_iRepresent region R_iThe parameter of corresponding characteristic probability distributed model, I_iRepresent region R_iCorresponding mark；

According to the matching relationship of feature in front and rear frame, it may be determined that cut zone number N.If the feature zonule set S={ S corresponding to each region₁, S₂..., S_N, for each region R_i, the zonule S occupied according to feature_iEstimate the initial parameter θ of model_i, obtain initial posterior probability P (θ_i| I (x, y)).According to MDL principles, posterior probability is converted into solution energy function minimum problem, obtained：

Wherein

Represent region R_iBoundary profile.The present invention estimates parameter { θ stage by stage using iterative manner_iAnd Γ, alternating iteration two benches, constantly reduce energy function in the various stages, so that constantly Reasoning With Learning goes out the final segmentation result of entire image；

During region-competitive, its characteristic probability distributed model is constantly updated in each region, while declining the ownership that principle fights for pixel according to steepest, updates respective boundary profile so that scope is constantly expanded in each region, finally gives the image segmentation result of present frame；

Its specific iterative step is：First stage, fixed Γ, according to current region segmentation state estimation { θ_i, try to achieve parameter θ under current state_iMaximal possibility estimation be used as its optimal solution

To minimize the cost for describing each region, therefore energy function is converted into：

Second stage, { θ_i, it is known that carrying out steepest decline to Γ, in order to quickly try to achieve the minimal solution of energy function, the present invention solves the equation of motion that steepest declines to the border Γ in all regions.For any point on boundary profile Γ

Have

Wherein,

For τ_kPointDirection vector, point

Which region belonged to, depending on point

It is adapted to the degree described by provincial characteristics probability Distribution Model；

To determine each pixel and interregional subordinate relation, the image segmentation algorithm process description based on competition mechanism is as follows：

In initial phase, the initial parameter of each class model is estimated according to the characteristic image block matched, the boundary point of all characteristic image blocks is added in queue undetermined, and calculate the posterior probability that all boundary points belong to all kinds of；

In the loop iteration stage, the boundary point i that present energy steepest declines is selected from queue undetermined, and then update all borders where boundary point i；Then under current cutting state, the model parameter in each region is recalculated using maximal possibility estimation；Using each provincial characteristics distributed model newly obtained, the posterior probability that all boundary points belong to all kinds of is recalculated；

So, the boundary point for constantly selecting present energy steepest to decline from queue undetermined updates corresponding border, update the feature distribution probabilistic model in each region according to current region segmentation state in good time simultaneously, multiple regions are mutually restricted, the ownership of image-region is competed simultaneously, until energy function convergence, so as to divide the image into as multiple regions.

5. the interactive video stylized rendering method according to claim 1 interpreted based on video, it is characterised in that the stylized module of above-mentioned video（2）Stylizing method step 4）Video stylization is based on interactive video semanteme segmentation module, and the selection of paintbrush is only determined by the material corresponding to the object area that identifies；

Above-mentioned paintbrush is all based on professional artist and a large amount of typical styles of writing is drawn on paper, then it is scanned and parameterizes, finally set up style of writing storehouse, drawn for each image-region, bottoming is carried out using big brush first, then brush size and opacity are gradually decreased to carry out meticulous depiction to the detail section of object, during drafting, using drafting strategy internal behind first edge：The drafting present invention of each tomographic image is drawn, and brush is alignd according to flow field first first since edge along the edge of line drawing；

In Video Rendering, in order to ensure stability of the paintbrush in time domain, the propagation of style of writing is carried out using thin-plate spline interpolation technology, in addition, style of writing is in communication process, also by calculating the area in style of writing region, style of writing is devised and deletes and increase mechanism；And using damping spring system is simulated, reduce " shake " effect of rendering result.

6. the interactive video stylized rendering method according to claim 1 interpreted based on video, it is characterised in that the stylized module of above-mentioned video（2）Stylizing method step 5）Based on semanteme parsing key frame non-photorealistic rendering method it is as follows：

How to design different artistic style style of writing models is one of stylized focus of attention of video, the works of different artistic expressions, differ from one another in style of writing expression, basic drafting strategy in video stylization is that the suitable style of writing of image content-based selection is drawn, style of writing storehouse is that a large amount of typical styles of writing are drawn on paper based on professional artist, then it is scanned and parameterizes, finally complete foundation, for the paintbrush that will be drawn

Include following information：The classification information of brush

, placement area scope

, color mapping

, transparency

, height field

And control point

, that is, have：

When designing style of writing model, the low level informations such as style of writing shape, texture are not only allowed for, while also combine its high-layer semantic information, so that in render process, each interpretation region of image/video has " pen " can be according to；When choosing style of writing, to interpret area classification as keyword, a collection of style of writing with identical category is selected from style of writing storehouse with simple and fast, and then a therefrom optional style of writing in a random basis；

" alignment " principle in being drawn for simulation oil painting, uses for reference original simple model theory, in each region

It is interior, calculate its original brief figure

Expression, brief figure is made up of the conspicuousness primitive of a group echo body surface feature, such as the spot on clothes, lines, gauffer；In render process, different paintbrush will be covered on these primitives to produce desired artistic effect；Interpret region

It is divided into the line drawing part for describing line drawing

And for describing the non-line drawing part with identical structural region

；

The field of direction

It is defined as：

The wherein field of direction

Initial value is line drawing

Gradient direction, direction is then traveled to non-line drawing region using diffusion equation

；

Render process to key frame is the continuous process chosen style of writing and put style of writing；To interpret region

Exemplified by, its non-line drawing part is rendered first

, then render line drawing part

；This is in order to ensure when the region rendered overlaps, the style of writing of line drawing part can be in upper strata；In non-line drawing part, optional one pixel region that is not rendered, using the center in the region as originating point, spreads along the field of direction to both sides, generates a flow pattern regions；The line on the basis of the axis in the region, the paintbrush chosen is transformed in the flow pattern regions, style of writing axis is alignd with region axis；It is similar to rendering for region line drawing part.

7. the interactive video stylized rendering method according to claim 1 interpreted based on video, it is characterised in that the stylized module of above-mentioned video（2）Stylizing method step 5）The style of writing transmission method of sequence frame is as follows：

Rendering for non-key frame is obtained by the rendering result " propagation " of key frame, the foundation of propagation is the space-time corresponding relation for interpreting region, in communication process, as the change in interpretation region is increasing, style of writing may gradually be leaked to region exterior, and the space with occurring in time domain to be rendered, so, in style of writing figure is propagated, it is necessary to while considering the addition of style of writing and deleting mechanism, otherwise, the jitter phenomenon that rendering result occurs；The propagation of style of writing, add and to delete mechanism as follows：

Style of writing is propagated：Order represents some interpretation region of video t key frame,

RepresentIn t+1 moment corresponding region, their image-region respectively with

、

Represent；With

、

Represent

、

Dense matching point (being calculated during video interpretation) in time domain；Assuming that

Table can pass through

The non-rigid transformation of table is obtained；When style of writing is propagated, the present invention wishes

On match point

Image-region new in t+1 frames can be mapped to

Match point

, based on considerations above, the present invention is from thin-plate spline interpolation model (Thin-plate Spline, TPS), and it can be

Middle key point

It is mapped toMatch point

, and for

In remaining non-key point pixel, TPS makes by minimizing energy function

Pixel grid occur elasticity（It is non-rigid）Deform and distort；

Style of writing is deleted：After being propagated in video due to paintbrush or when there occurs hiding relation or too many style of writing propagation frame number, region corresponding to some paintbrush can become less and less, therefore, the present invention will reject these paintbrush when their corresponding region areas are less than some given threshold value, equally, also to be deleted when the paintbrush of propagation falls outside corresponding zone boundary；

Style of writing increases, when there is new semantic region or already present semantic region becomes increasing（The expansion of such as clothes）The present invention must increase new paintbrush to cover these emerging regions, and in order to fill the space between paintbrush, the present invention only need to simply change size and the position of adjacent paintbrush, if the region not covered by paintbrush becomes bigger and bigger and exceeded some given threshold value, system can be automatically created new paintbrush to cover it；Nevertheless, the present invention is still unlikely to draw one to it at once when space occurs for the first time；Then, comparatively the present invention is provided with compares high threshold value, and postpones to render emerging region when they rise to sufficiently large；Then, the present invention puts algorithm to be filled up to the sufficiently large space of threshold value using general paintbrush, finally oppositely propagates and converts again these new paintbrush and go to fill void area that is previously having occurred but not rendering；The process of filling paintbrush can avoid continually converting paintbrush backward, while less some scrappy paintbrushes can be linked as larger paintbrush again, so as to reduce scintillation effect and other undesirable visual effects artificially caused；Similarly, since the present invention is that new paintbrush is added in the bottom, so they are drawn in below the paintbrush existed, This further reduces visual scintillation effect.

8. the interactive video stylized rendering method according to claim 1 interpreted based on video, it is characterised in that the stylized module of above-mentioned video（2）Stylizing method step 6）In be used for stabilization damping brush system it is as follows：

It is stabilization operation to the final step that video carries out stylized rendering, paintbrush adjacent in time domain and spatial domain is attached with spring, to simulate damping system；By the energy for minimizing the system, it is possible to reach the effect for removing shake；

For i-th of paintbrush of t, the present invention is used

Its centre coordinate and the geometric attribute of size are represented, and its initial value is designated as；The energy function of damping brush system is defined as follows：

With

For weight,,

；

The first item constraint paintbrush position will can not be partially too far away with initial position in formula：

Section 2 is the smoothness constraint carried out to paintbrush i in time domain in formula：

Section 3 all carries out smoothness constraint to adjacent paintbrush in time domain and spatial domain in formula；Note

For the adjacent paintbrush of i-th of paintbrush of t, for arbitrary neighborhood paintbrush

, the relative distance difference and difference in size between them are designated as

, and smooth item is defined as below：

Pass through Levenbergy-Marquard Algorithm for Solving energy minimization problems.

9. the interactive video stylized rendering method according to claim 8 interpreted based on video, it is characterised in that above-mentioned,

。