CN103226824A - Video retargeting system for maintaining visual saliency - Google Patents

Abstract

The invention provides a video retargeting system for maintaining visual saliency. The system comprises a salient image calculating module, a candidate seam extraction module, a curve deformation module, a salient curve detection module, a graded shape matching module and an optimum seam calculating module, wherein the salient curve detection module comprises a curve endpoint detection module and a curve tracking module; and the graded shape matching module comprises a shape tree construction module and a deformation cost calculating module. The video retargeting system greatly reduces curve deformation and distortion which are easily caused by a conventional video retargeting method, reserves visual saliency information in a video on the basis, maintains the time continuity, is lower in algorithm complexity, and has good subjective feeling and objective quality assessment.

Description

Keep the video Redirectional system of vision significance

Technical field

The present invention relates to a kind of system of image/video process field, specifically is a kind of video Redirectional system based on remarkable curve detection and layering Curve Matching.

Background technology

In recent years, along with the development and progress of terminal presentation facilities such as mobile phone, panel computer, notebook computer, video is redirected concern and the attention that (video retargeting) technology is subjected to academia day by day.The video redirecting technique refers to a kind ofly adjusts to arbitrary resolution with video from original resolution, with the technology of the display device that adapts to different size.Different with methods such as traditional stretching, cutting, fillings, the video redirecting technique has kept the information that has vision significance most largely in the process of adjusting resolution, thereby has splendid impression.Domestic and international in recent years many studies show that, the video redirecting technique mainly faces the technological challenge of three aspects: 1. keep vision significance, 2. reduce deformation, 3. retention time continuity.Keep vision significance to refer to when video is redirected and to keep the information that can cause the vision sensitivity as far as possible.The main object that reduction deformation refers in the video that is redirected front and back should have the similar shapes feature.The video that the retention time continuity refers to after being redirected should have coherent, level and smooth characteristic on time domain, avoid the generation of ripple and shake.

Find through literature search, proposed image redirecting technique in " Seam carving for content-aware image resizing " literary composition that S.Avidan and A.Shamir deliver first in " ACM SIGGRAPH " meeting in 2007 based on seam carving to prior art.Video redirecting technique based on seam carving has been proposed in " Improved seam carving for video retargeting " literary composition that M.Rubinstein, A.Shamir and S.Avidan deliver in " ACM SIGGRAPH " meeting in 2008 first.The basic thought of Seam carving is to adjust the resolution of image and video by the method in the minimum slit (seam) of continuous removal vision significance.Mainly comprise two steps: 1. specific image calculates; 2. calculate in the minimum slit of energy.Yet, be that because the slit of removing can be an arbitrary shape, the object in the video after causing being redirected can produce bigger deformation based on the subject matter of the video reorientation method of seam carving.

Summary of the invention

At defective of the prior art, the purpose of this invention is to provide a kind of video Redirectional system based on remarkable curve detection and layering Curve Matching.The present invention is by detecting the higher curve of vision significance in the original video, and in redirection process, take all factors into consideration and keep vision significance and reduce factor aspect the curve deformation two, try to achieve optimum slit, the form reduction degree and the euphorosia degree of the video after having guaranteed to be redirected.

The present invention is achieved by the following technical solutions:

The present invention includes: specific image computing module, candidate slot extraction module, curve deformation module, remarkable curve detection module, classification form fit module and optimum slit computing module.Wherein:

The specific image computing module calculates the specific image of each frame of input video, and the result is outputed to the candidate slot extraction module;

The candidate slot extraction module receives the result of specific image computing module, and extracts minimal energy path, and the result is outputed to curve deformation module and optimum slit computing module respectively;

Significantly the curve detection module is calculated the remarkable curve of each frame of input video, and the result is outputed to curve deformation module respectively mentions classification form fit module;

Curve deformation module receives the output of remarkable curve detection module and candidate slot extraction module, and calculates the deformation that curve is caused that removes of each candidate slot, and the result is outputed to classification form fit module;

Classification form fit module receives curve deformation module and the significantly output of curve detection module, and calculates the form fit cost of the curve before and after the deformation, and the result is outputed to optimum slit computing module;

Optimum slit computing module receives the output of classification form fit module and candidate slot extraction module, and optimum slit is calculated in comprehensive vision significance and curve deformation.

Preferably, described remarkable curve detection module comprises endpoint curve detection module and curve tracking module.Wherein: the endpoint curve detection module links to each other with input video and detects the starting point of the remarkable curve of each frame, and the curve tracking module links to each other with the endpoint curve detection module and from the direction search curve of end points according to maximization curve weight.

Preferably, described classification form fit module comprises shape tree constructing module and deformation cost computing module.Wherein: shape tree constructing module links to each other with remarkable curve detection module and the remarkable curve construction shape tree to extracting, and deformation cost computing module is set the form fit cost that constructing module links to each other and calculates the curve of deformation front and back with shape.

Preferably, described specific image computing module adopts the sign of gradient magnitude as the image vision conspicuousness.

Preferably, described candidate slot extraction module is to realize by the minimal energy path that dynamic programming method is asked for specific image.

Preferably, described optimum slit computing module is from vision significance and definition slit, remarkable curve deformation cost two aspects energy, and COMPREHENSIVE CALCULATING goes out the slit of energy minimum and removed.

Compared with prior art, the present invention has following beneficial effect:

The present invention greatly reduces the deformation that is redirected main curve in the rear video, has reduced to be redirected the content distortion that causes; Do not need every frame to calculate different slits, and by maintenance interframe slit smooth mode retention time continuity, but a frame section is got same slit vertical on the time domain, also kept time continuity when having reduced computation complexity; Do not need complicated visual attention location Model Calculation specific image, and adopt simple gradient feature, promptly reflected the vision sensitive information, reduced calculation cost, comparatively responsive to curve again.

Description of drawings

By reading the detailed description of non-limiting example being done with reference to the following drawings, it is more obvious that other features, objects and advantages of the present invention will become:

Fig. 1 is that system of the present invention forms connection diagram.

Embodiment

The present invention is described in detail below in conjunction with specific embodiment.Following examples will help those skilled in the art further to understand the present invention, but not limit the present invention in any form.Should be pointed out that to those skilled in the art, without departing from the inventive concept of the premise, can also make some distortion and improvement.These all belong to protection scope of the present invention.

Embodiment

As shown in Figure 1, present embodiment comprises: specific image computing module, candidate slot extraction module, curve deformation module, remarkable curve detection module, classification form fit module and optimum slit computing module.Wherein: the specific image computing module links to each other with input video and calculates the specific image of each frame, the candidate slot extraction module links to each other with the specific image computing module and extracts minimal energy path, remarkable curve detection module links to each other with input video and calculates the remarkable curve of each frame, curve deformation module links to each other with the candidate slot extraction module with remarkable curve detection module and calculates the deformation that curve is caused that removes of each candidate slot, optimum slit is calculated in link to each other with the candidate slot extraction module comprehensive vision significance and the curve deformation of link to each other with the remarkable curve detection module form fit cost of the curve that calculates the deformation front and back of classification form fit module and curve deformation module, optimum slit computing module and classification form fit module.

In the present embodiment, described specific image computing module calculates the specific image of each frame of input video, is specially: the video to input carries out uniform sampling, extracts a frame every the N frame, calculates its gradient magnitude

As its specific image, wherein I _x(x, y) and I _y(x y) is image I (x, y) gradient of X and Y direction respectively.For each frame in this video-frequency band, its specific image is

Wherein n is the totalframes of extraction.

The concrete course of work of candidate slot extraction module is as follows in the present embodiment:

1) for vision specific image m (x y), calculates its energygram, is specially:

$E(x,y)=\left\{\begin{array}{c}m(x,y),\mathrm{ify}=1\\ m(x,y)+\underset{-1\&le;i\&le;1}{\min }E(x+i,y-1),\mathrm{if}1<y\&le;H\end{array}\right.$

2) (x, y) each element in the end row begins its minimal energy path of bottom-up calculating p from energygram E _w(h): 1≤w≤W, 1≤h≤H is specially:

$p_w\left(h\right)=\left\{\begin{array}{c}w,\mathrm{ifh}=H\\ p_h(h+1)+\underset{-1\&le;i\&le;1}{\arg \min }E(p_w(h+1)+i,h),\mathrm{if}1\&le;h<H\end{array}\right.$

3) from whole W paths

In select the energy minimum the Nc bar as candidate slot, note is done

Described remarkable curve detection module comprises endpoint curve detection module and curve tracking module.Wherein: the endpoint curve detection module links to each other with input video and detects the starting point of the remarkable curve of each frame, and the curve tracking module links to each other with the endpoint curve detection module from the direction search curve of end points according to maximization curve weight.

The concrete course of work of endpoint curve detection module is as follows in the present embodiment:

1) input picture is carried out the Canny edge extracting, obtain two-value skirt response image E _d, if E _d(x, y)=1, then (x y) is marginal point to pixel; If E _d(x, y)=0, then (x y) is non-marginal point to pixel.

2) to skirt response E _dCarry out end-point detection, be specially: for E _dOn each point (x, y), detecting with it is response distribution situation in 3 * 3 fields at center, as if E _d(x, y)=1, and (x has and has only a marginal point in 8 fields y), then (x y) is spring of curve; If E _d(x, y)=1, and (x, the number of edge points that four limits exist in 8 fields y) is (2,1,0,0) from high to low, then (x y) is spring of curve; Otherwise (x y) does not regard spring of curve as.

The concrete course of work of curve tracking module is as follows in the present embodiment:

1) curve representation is the sequence C=(x of point ₁..., x _n), x wherein _i=(x _i, y _i) be the coordinate that i is ordered on the curve.The fundamental element of curve is directed line segment (oriented segments) s _i=x _I+1-x _i, feasible directed line segment has 16 kinds of S={ (2 ,-2), (1 ,-2), (0 ,-2), (1 ,-2), (2 ,-2), (2 ,-1), (2,0), (2,1), (2,2), (1,2), (0,2), (1,2), (2,2), (2,1), (2,0), (2 ,-1) }.The directed line segment of forming curve satisfies s _i∈ S, i=1 ..., n-1.

2) calculated curve response

I wherein _x(x, y) and I _y(x y) is image I (x, y) gradient of directions X and Y direction respectively.

3) from current some x _i=(its 16 feasible directions are on every side searched in x, y) beginning

Getting the maximum direction of its weight is the curve bearing of trend.Be specially: the directed line segment weight definition is w (s)=m (x _i+ s)+λ T (s, s _I-1), T (s, s wherein _I-1) be the curve smoothing item, it weighs the level and smooth degree of adjacent two sections directed line segments.λ is a weight parameter, the relative weighting of balance gray scale item and level and smooth item.λ is big more, and curve is level and smooth more; λ is more little, and curve grey scale change Shaoxing opera is strong.Curve smoothing item T (s, s _I-1) be defined as:

$T(s,s^{\&prime;})=\left\{\begin{array}{c}R-|s-s^{\&prime;}|,\mathrm{ifR}-|s-s^{\&prime;}|\&GreaterEqual;0\\ -\&infin;,\mathrm{ifR}-|s-s^{\&prime;}|<0\end{array}\right.$

The maximum direction of weight is s _i=argmin _{S ∈ S}W (s).If w (s _i) Q, then curve continues to extend, and current point is x _I+1=x _i+ s _iIf w (s _i)≤Q, then curve ends at x _iWherein Q is predefined line segment weight threshold.

Described curve deformation module is calculated slit p (i), the deformation that the removal of 1≤i＜H causes curve.Be specially: for certain the point (x on the curve _i, y _i), the horizontal ordinate after its deformation becomes

$x_i^{\&prime;}=\left\{\begin{array}{c}{x}_i,\mathrm{ifp}\left(y_i\right)>x_i\\ x_i-1,\mathrm{ifp}\left(y_i\right)\&le;x_i\end{array}\right.$

Described classification form fit module comprises shape tree constructing module and deformation cost computing module.Wherein: shape tree constructing module links to each other with remarkable curve detection module to remarkable curve construction shape tree, link to each other with the shape tree constructing module form fit cost of curve of calculating deformation front and back of deformation cost computing module.

The concrete course of work of shape tree constructing module is as follows in the present embodiment:

1) root node storage curve C=(x of shape tree ₁..., x _n) mid point

With respect to two-end-point x ₁=(x ₁, y ₁) and x _n=(x _n, y _n) coordinate: B (x _i| x ₁, x _n)=(x ' _i, y ' _i), be specially:

${x_i}^{\&prime;}=\frac{(x_n-x_1)(x_i-x_1)+(y_n-y_1)(y_i-y_1)}{{(x_n-x_1)}^2+{(y_n-y_1)}^2}-0.5$

$y_i^{\&prime;}=\frac{(x_n-x_1)(y_i-y_1)+(y_n-y_1)(x_i-x_1)}{{(x_n-x_1)}^2+{(y_n-y_1)}^2}$

2) left subtree of root node is with preceding half section curve C of identical recursive fashion storage curve C ^L=(x ₁..., x _i); The right subtree of root node is with the second half section curve C of identical recursive fashion storage curve C ^R=(x _I+1..., x _n).

3) continuous three point (x on the leaf node storage curve of shape tree _J-1, x _j, x _J+1) mid point x _jWith respect to other 2 x _J-1With x _J+1Coordinate.

The concrete course of work of deformation cost computing module is as follows in the present embodiment:

1) the virgin curve note is C=(x ₁..., x _n), curve after deformation note be C '=(x ' ₁..., x ' _n), the optimum matching of two curves is to seek 1 x ' on C ' _j, make x ₁Coupling x ' ₁, x _nCoupling x ' _n, x _iCoupling x ' _jCoupling cost minimum.Wherein, the coupling cost is defined as:

$d(C,C^{\&prime;})=\underset{j}{\min }\{d(C^L,C^{\&prime;L})+d(C^R,C^{\&prime;R})+\mathrm{\&alpha;}|B\left(x_i\right|x_1,x_n)-B\left(x_j^{\&prime;}\right|x_1^{\&prime;},x_n^{\&prime;})\left|\right\}$

D (C wherein ^L, C ' ^L) be the coupling cost of the left subtree of the left subtree of C and C ', d (C ^R, C ' ^R) be the coupling cost of the right subtree of the right subtree of C and C ', | B (x _i| x ₁, x _n)-B (x ' _j| x ' ₁, x ' _n) | be the Euclidean distance of mid point of curve coordinate.

2) left and right sides subtree of two curves is respectively with identical recursive fashion calculating optimum coupling d (C ^L, C ' ^L) and d (C ^R, C ' ^R).

3) on leaf node, the coupling cost of definition line segment and line segment is 0, that is: if C=(x ₁, x ₂), C '=(x ' ₁, x ' ₂), d (C, C ')=0 then; The coupling cost of definition line segment and curve is: if C=(x ₁, x ₂), C '=(x ' ₁..., x ' _m), then $d(C,C^{\&prime;})={\mathrm{\&Sigma;}}_{i=2}^{m-1}\left|B\left(y_i^{\&prime;}\right|x_1^{\&prime;},x_m^{\&prime;})\right|.$

The concrete course of work of optimum slit computing module is as follows in the present embodiment

1) for a certain candidate slot p=(p ₁..., p _H), calculate its remarkable energy

Wherein (x y) is specific image to S.

2) for a certain candidate slot p=(p ₁..., p _H), calculate its deformation cost E _D, be specially: the remarkable curve of all in the original image

With its curve after by slit p deformation

The deformation cost be Then the deformation cost of slit p is $E_D={\mathrm{\&Sigma;}}_{i=1}^{n}d(C_i,C_i^{\&prime;}).$

3) gross energy of slit p is E (p)=E _s+ β E _Dβ is a slit energy weight parameter.β is big more, and the curve deformation that cause in the slit is more little; β is more little, and the vision significance in slit is low more.Optimum slit $p^{*}=$ ${\arg \min }_{1\&le;j\&le;N_c}E\left(p_j\right).$

The present invention greatly reduces curve deformation and the distortion that the conventional video reorientation method causes easily, kept the remarkable information of vision in the video on this basis, and kept time continuity, algorithm complex is lower, has good subjective feeling and evaluating objective quality.The subjective testing experimental result shows that 78% surveyee thinks that video of the present invention is redirected the result and is better than classic method.Objective examination's experimental result shows that the remarkable shaped form that the redirected result of video of the present invention causes has in a disguised form reduced more than 10% than classic method.

More than specific embodiments of the invention are described.It will be appreciated that the present invention is not limited to above-mentioned specific implementations, those skilled in the art can make various distortion or modification within the scope of the claims, and this does not influence flesh and blood of the present invention.

Claims (6)

1. a video Redirectional system of keeping vision significance is characterized in that, comprising: specific image computing module, candidate slot extraction module, curve deformation module, remarkable curve detection module, classification form fit module and optimum slit computing module; Wherein:

The specific image computing module calculates the specific image of each frame of input video, and the result is outputed to the candidate slot extraction module;

The candidate slot extraction module receives the result of specific image computing module, and extracts minimal energy path, and the result is outputed to curve deformation module and optimum slit computing module respectively;

Significantly the curve detection module is calculated the remarkable curve of each frame of input video, and the result is outputed to curve deformation module respectively mentions classification form fit module;

Curve deformation module receives the output of remarkable curve detection module and candidate slot extraction module, and calculates the deformation that curve is caused that removes of each candidate slot, and the result is outputed to classification form fit module;

Classification form fit module receives curve deformation module and the significantly output of curve detection module, and calculates the form fit cost of the curve before and after the deformation, and the result is outputed to optimum slit computing module;

Optimum slit computing module receives the output of classification form fit module and candidate slot extraction module, and optimum slit is calculated in comprehensive vision significance and curve deformation.

2. the video Redirectional system of keeping vision significance according to claim 1 is characterized in that, described remarkable curve detection module comprises endpoint curve detection module and curve tracking module; Wherein: the endpoint curve detection module links to each other with input video and detects the end points of the remarkable curve of each frame, and the curve tracking module links to each other with the endpoint curve detection module and from the direction search curve of end points according to maximization curve weight.

3. the video Redirectional system of keeping vision significance according to claim 1 and 2 is characterized in that, described classification form fit module comprises shape tree constructing module and deformation cost computing module; Wherein: shape tree constructing module links to each other with remarkable curve detection module and to each remarkable curve construction shape tree, deformation cost computing module is set the form fit cost that constructing module links to each other and calculates the curve of deformation front and back with shape.

4. the video Redirectional system of keeping vision significance according to claim 1, it is characterized in that, described specific image computing module adopts the sign of gradient magnitude as the image vision conspicuousness, and the related width of cloth specific image of frame of video section is by obtaining frame section uniform sampling and weighting.

5. according to claim 1 or the 2 or 4 described video Redirectional systems of keeping vision significance, it is characterized in that described candidate slot extraction module is to realize by the minimal energy path that dynamic programming method is asked for specific image.

6. according to claim 1 or the 2 or 4 described video Redirectional systems of keeping vision significance, it is characterized in that, described optimum slit computing module is from vision significance and definition slit, remarkable curve deformation cost two aspects energy, and COMPREHENSIVE CALCULATING goes out the slit of energy minimum and removed.

Images