CN103533237B - A kind of method extracting key frame of video from video - Google Patents

Abstract

The present invention relates to a kind of method extracting key frame of video from video, belong to technical field of image processing.The method extracting key frame of video from video of the present invention, operator carry out video capture by device to scene interested.Frame of video, acceleration information, azimuth information and the dimensional information of device synchronous recording video in shooting process.Directly according to acceleration information, azimuth information and dimensional information after having shot, each frame frame of video is calculated its weight.Desired key frame of video is extracted finally according to weight and desired number of key frames.The method extracting key frame of video from video that the present invention proposes, can extract key frame of video more accurately from video by less amount of calculation.

Description

A kind of method extracting key frame of video from video

Technical field

The present invention relates to a kind of method extracting key frame of video from video, belong to technical field of image processing.

Background technology

Along with the increase of hand-held capture apparatus (such as mobile phone, digital camera and hand-supported camera) quantity in recent years, individual The number of videos that people user uses portable equipment to shoot also rolls up.Various video captures application on smart mobile phone, as Instagram etc., also have stimulated the propagation of these videos, and Instagram, in 24 hours reached the standard grade, just has 5,000,000 to regard Frequency is uploaded.

So many application causes to produce substantial amounts of video every day.The place different from Word message is, video information Cannot directly be retrieved, therefore finding useful information from substantial amounts of video is the work extremely taken time and effort.At present The method taked mainly relies on the content manually checking video, and is labeled it, and this is the method for very poor efficiency undoubtedly.

Therefore the extraction of key frame becomes and significantly works.The Key-frame Extraction Algorithm that presently, there are mainly has following several Kind: 1) extract key frame according to fixed time interval, 2) calculate adjacent a few frame difference in color (or gray scale), Determine whether it is key frame.3) method based on motion analysis.

It is the most directly perceived for extracting key frame according to Fixed Time Interval, calculates simplest way, but the shortcoming of this method is also It is obvious that key frame may not be certain to be equally distributed.It is simple that the most this kind of method is applicable to content, the video that the time is shorter.

The method calculating adjacent frame difference is relatively more reasonable, but relatively difficult in choosing of threshold value, and threshold value value crosses I Can cause selecting too many key frame, value is excessive, likely can cause leakage choosing.And this method calculates upper the most more complicated.

Main method based on motion analysis mainly calculates the quantity of motion in camera lens by optical flow analysis, on the ground that quantity of motion is minimum Side chooses key frame, and it is excessive that its shortcoming lies also in amount of calculation.

Summary of the invention

The present invention seeks to propose a kind of method extracting key frame of video from video, to solve existing key-frame extraction side The shortcoming of method, the acceleration information of engraving device time each in content, photographic head zoom behavior and the video in conjunction with photographic head shooting And filming apparatus towards etc. information, according to the intention of video capture person, be calculated key frame.

The method extracting key frame of video from video that the present invention proposes, comprises the following steps:

(1) use video capture device, floor scene, obtain video, set total T frame frame of video in video, and Record the zoom scale information of each shooting moment filming apparatus photographic head；

(2) use the frequency identical with shooting video, record each shooting moment filming apparatus edge in rectangular coordinate system respectively The linear acceleration information of x, y, z axle；

(3) use the frequency identical with shooting video, with aspect sensor record each shooting the moment shooting be equipped in above-mentioned directly Azimuth information in angle coordinate system；

(4) according to azimuth information, linear acceleration information and the dimensional information of above-mentioned record, from video, key frame is extracted, Comprise the following steps:

(4-1) characteristic information of device when shooting kth frame frame of video is extracted in video, including: kth frame frame of video shoots Filming apparatus azimuth information o in moment_k=[o_x,k,o_y,k,o_z,k]^T, wherein o_x,kWhen representing the shooting of kth frame frame of video Carve the angle of the roll angle of filming apparatus, i.e. device minor face and horizontal plane, o_y,kRepresent that the kth frame frame of video shooting moment claps Take the photograph the angle on the long limit of the luffing angle of device, i.e. device and horizontal plane, o_z,kRepresent kth frame frame of video shooting moment shooting dress The angle of vacillating now to the left, now to the right put, i.e. the direction of device top sensing and the angle of direct north；The kth frame frame of video shooting moment The acceleration information a of filming apparatus_k=[a_x,k,a_y,k,a_z,k]^T, wherein a_x,k,a_y,k,a_z,kFor device respectively directly The x of angle coordinate system, y, the acceleration in z-axis, dimensional information s_kRepresent the zoom scale of photographic head when shooting kth frame；

(4-2) use discrete cosine transform, video obtained above is carried out feature information extraction, obtains kth frame in video The frame of video characteristic information f of frame of video_k；

(4-3) repeat step (4-1) and step (4-2), obtain the filming apparatus side of each frame frame of video in above-mentioned video Position information, the acceleration information of filming apparatus, the zoom scale of photographic head and frame of video characteristic information；

(4-5) the acceleration weights omega of each frame frame of video in video is calculated_ak: ω_ak=exp (-λ₁||a_k||₂), Wherein λ₁Parameter is regulated for acceleration, | | a_k||₂Represent acceleration information a_kTwo norms of vector, λ₁Span can root Determining according to the order of magnitude of acceleration, span is: 0.1～1；

(4-6) calculate each frame frame of video in video yardstick weights omega_sk: ω_sk=exp (λ₂s_k), wherein λ₂Parameter, λ is regulated for yardstick₂Span be: 0.5～1；

(4-7) total weights omega of each frame frame of video in video is calculated_k: ω_k=ω_akω_sk；

(4-8) use K mean algorithm, the filming apparatus azimuth information in all frame of video shooting moment in above-mentioned video is carried out Cluster, obtains C cluster centre, and C is the parameter chosen according to information such as video lengths, and the span of C is: 1～T, T is the frame number of all frame of video in video, and all of frame of video is referred to the azimuth information with corresponding filming apparatus connects most The near apoplexy due to endogenous wind belonging to cluster centre；

(4-9) optimization object function is set up as follows:

$J=\underset{k=1}{\overset{T}{\Σ}}\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{kl}^2\left|\right|o_k-v_j^{\left(p\right)}|{|}_2^2,$

Constraints is:0≤μ_kj≤1

Wherein k is the sequence number of frame of video, and j is the classification of cluster centre, j ∈ [1, C], μ_kjIt is parameter to be solved, υ_jPoly- Class center, p is current iteration number of times；

(4-10) when initializing, if p=0,The vector that initial value is jth cluster centre；

(4-11) μ is calculated_kj:

(4-12) according to above-mentioned result of calculation, μ is updated_kjValue, calculate μ_kj:

${\mathrm{\μ}}_{kj}={\mathrm{\μ}}_{kj}/\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\μ}}_{kj}$

(4-13) according to step (4-12) calculated μ_kj, calculate

(4-14) an iteration ends threshold epsilon is set, ifThen make p=p + 1, and return step (4-11), ifThen carry out step (4-15), ε Span be: 0.01～0.001；

(4-15) by following formula, an initial key frame set K={t is obtained₁,t₁,…,t_C}:Wherein j ∈ [1, C]；

(4-16) similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K is calculatedWherein i, j ∈ [1, C]；

(4-17) similarity threshold is set, in traversal step (4-16) calculated initial key frame set K Any two frames, calculate the similarity of the frame of video characteristic information of any two framesWith similarity threshold Compare, ifAndFrom above-mentioned initial key frame set K, then delete t_j； IfAndFrom above-mentioned initial key frame set K, then delete t_i；IfIn above-mentioned initial key frame set K, then retain t_iAnd t_j, repeating this step, the set K obtained is Key frame of video, the span of δ is: 0.2～0.3.

The method extracting key frame of video from video that the present invention proposes, its advantage is, can shoot the same of video user Time, the motion of device during records photographing video, towards and the change of focal length, and according to the information of filming apparatus, thus it is speculated that go out User shooting time intention, the acceleration of such as device is less, and towards keep certain numerical value continue for some time time, It is believed that user is shooting a certain scene targetedly, namely this scene is crucial to user.Accordingly it is assumed that permissible From video, key frame of video is extracted more accurately by less amount of calculation.

Accompanying drawing explanation

Fig. 1 is the FB(flow block) of the method extracting key frame of video from video that the present invention proposes.

Detailed description of the invention

The method extracting key frame of video from video that the present invention proposes, its FB(flow block) is as it is shown in figure 1, include following step Rapid:

(1) use video capture device, floor scene, obtain video, set total T frame frame of video in video, and Record the zoom scale information of each shooting moment filming apparatus photographic head；

(2) use the frequency identical with shooting video, record each shooting moment filming apparatus edge in rectangular coordinate system respectively The linear acceleration information of x, y, z axle；

(3) use the frequency identical with shooting video, with aspect sensor record each shooting the moment shooting be equipped in above-mentioned directly Azimuth information in angle coordinate system；

(4) according to azimuth information, linear acceleration information and the dimensional information of above-mentioned record, from video, key frame is extracted, Comprise the following steps:

(4-1) characteristic information of device when shooting kth frame frame of video is extracted in video, including: kth frame frame of video shoots Filming apparatus azimuth information o in moment_k=[o_x,k,o_y,k,o_z,k]^T, wherein o_x,kWhen representing the shooting of kth frame frame of video Carve the angle of the roll angle of filming apparatus, i.e. device minor face and horizontal plane, o_y,kRepresent that the kth frame frame of video shooting moment claps Take the photograph the angle on the long limit of the luffing angle of device, i.e. device and horizontal plane, o_z,kRepresent kth frame frame of video shooting moment shooting dress The angle of vacillating now to the left, now to the right put, i.e. the direction of device top sensing and the angle of direct north；The kth frame frame of video shooting moment The acceleration information a of filming apparatus_k=[a_x,k,a_y,k,a_z,k]^T, wherein a_x,k,a_y,k,a_z,kFor device respectively directly The x of angle coordinate system, y, the acceleration in z-axis, dimensional information s_kRepresent the zoom scale of photographic head when shooting kth frame；

(4-2) use discrete cosine transform, video obtained above is carried out feature information extraction, obtains kth frame in video The frame of video characteristic information f of frame of video_k；

(4-3) repeat step (4-1) and step (4-2), obtain the filming apparatus side of each frame frame of video in above-mentioned video Position information, the acceleration information of filming apparatus, the zoom scale of photographic head and frame of video characteristic information；

(4-5) the acceleration weights omega of each frame frame of video in video is calculated_ak: ω_ak=exp (-λ₁||a_k||₂), Wherein λ₁Parameter is regulated for acceleration, | | a_k||₂Represent acceleration information a_kTwo norms of vector, λ₁Span can root Determining according to the order of magnitude of acceleration, span is: 0.1～1；

(4-6) calculate each frame frame of video in video yardstick weights omega_sk: ω_sk=exp (λ₂s_k), wherein λ₂Parameter, λ is regulated for yardstick₂Span be: 0.5～1；

(4-7) total weights omega of each frame frame of video in video is calculated_k: ω_k=ω_akω_sk；

(4-8) use K mean algorithm, the filming apparatus azimuth information in all frame of video shooting moment in above-mentioned video is carried out Cluster, obtains C cluster centre, and C is the parameter chosen according to information such as video lengths, and the span of C is: 1～T, T is the frame number of all frame of video in video, and all of frame of video is referred to the azimuth information with corresponding filming apparatus connects most The near apoplexy due to endogenous wind belonging to cluster centre；

(4-9) optimization object function is set up as follows:

$J=\underset{k=1}{\overset{T}{\Σ}}\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{kj}^2\left|\right|o_k-v_j^{\left(p\right)}|{|}_2^2,$

Constraints is:0≤μ_kj≤1

Wherein k is the sequence number of frame of video, and j is the classification of cluster centre, j ∈ [1, C], μ_kjIt is parameter to be solved, υ_jPoly- Class center, p is current iteration number of times；

(4-10) when initializing, if p=0,The vector that initial value is jth cluster centre；

(4-11) μ is calculated_kj:

(4-12) according to above-mentioned result of calculation, μ is updated_kjValue, calculate μ_kj:

${\mathrm{\μ}}_{kj}={\mathrm{\μ}}_{kj}/\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\μ}}_{kj}$

(4-13) according to step (4-12) calculated μ_kj, calculate

(4-14) an iteration ends threshold epsilon is set, ifThen make p=p + 1, and return step (4-11), ifThen carry out step (4-15), ε Span be: 0.01～0.001；

(4-15) by following formula, an initial key frame set K={t is obtained₁,t₁,…,t_C}:Wherein j ∈ [1, C]；

(4-16) similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K is calculatedWherein i, j ∈ [1, C]；

(4-17) similarity threshold is set, in traversal step (4-16) calculated initial key frame set K Any two frames, calculate the similarity of the frame of video characteristic information of any two framesWith similarity threshold Compare, ifAndFrom above-mentioned initial key frame set K, then delete t_j； IfAndFrom above-mentioned initial key frame set K, then delete t_i；IfIn above-mentioned initial key frame set K, then retain t_iAnd t_j, repeating this step, the set K obtained is Key frame of video, the span of δ is: 0.2～0.3.

The method extracting key frame of video from video of the present invention, scene interested is regarded by operator by device Frequency shooting.Frame of video, acceleration information, azimuth information and the dimensional information of device synchronous recording video in shooting process. Directly utilize acceleration information, azimuth information and dimensional information after having shot and each frame frame of video is calculated its weight.Finally Desired key frame of video is extracted according to weight and desired number of key frames.

Claims (1)

1. the method extracting key frame of video from video, it is characterised in that the method comprises the following steps:

(1) use video capture device, floor scene, obtain video, set total T frame frame of video in video, and Record the zoom scale information of each shooting moment filming apparatus photographic head；

(2) use the frequency identical with shooting video, record each shooting moment filming apparatus edge in rectangular coordinate system respectively The linear acceleration information of x, y, z axle；

(3) use the frequency identical with shooting video, with aspect sensor record each shooting the moment shooting be equipped in above-mentioned directly Azimuth information in angle coordinate system；

(4) according to azimuth information, linear acceleration information and the dimensional information of above-mentioned record, from video, key frame is extracted, Comprise the following steps:

(4-1) characteristic information of device when shooting kth frame frame of video is extracted in video, including: kth frame frame of video shoots Filming apparatus azimuth information o in moment_k=[o_x,k,o_y,k,o_z,k]^T, wherein o_x,kWhen representing the shooting of kth frame frame of video Carve the angle of the roll angle of filming apparatus, i.e. device minor face and horizontal plane, o_y,kRepresent that the kth frame frame of video shooting moment claps Take the photograph the angle on the long limit of the luffing angle of device, i.e. device and horizontal plane, o_z,kRepresent kth frame frame of video shooting moment shooting dress The angle of vacillating now to the left, now to the right put, i.e. the direction of device top sensing and the angle of direct north；The kth frame frame of video shooting moment The acceleration information a of filming apparatus_k=[a_x,k,a_y,k,a_z,k]^T, wherein a_x,k,a_y,k,a_z,kFor device respectively directly The x of angle coordinate system, y, the acceleration in z-axis, dimensional information S_kRepresent the zoom scale of photographic head when shooting kth frame；

(4-2) use discrete cosine transform, video obtained above is carried out feature information extraction, obtains kth frame in video The frame of video characteristic information f of frame of video_k；

(4-3) repeat step (4-1) and step (4-2), obtain the filming apparatus side of each frame frame of video in above-mentioned video Position information, the acceleration information of filming apparatus, the zoom scale of photographic head and frame of video characteristic information；

(4-5) the acceleration weights omega of each frame frame of video in video is calculated_ak: ω_ak=exp (-λ₁||a_k||₂), Wherein λ₁Parameter is regulated for acceleration, | | a_k||₂Represent acceleration information a_kTwo norms of vector, λ₁Span can root Determining according to the order of magnitude of acceleration, span is: 0.1～1；

(4-6) calculate each frame frame of video in video yardstick weights omega_sk: ω_sk=exp (λ₂s_k), wherein λ₂Parameter, λ is regulated for yardstick₂Span be: 0.5～1；

(4-7) total weights omega of each frame frame of video in video is calculated_k: ω_k=ω_akω_sk；

(4-8) use K mean algorithm, the filming apparatus azimuth information in all frame of video shooting moment in above-mentioned video is carried out Cluster, obtains C cluster centre, and C is the parameter chosen according to information such as video lengths, and the span of C is: 1～T, T is the frame number of all frame of video in video, and all of frame of video is referred to the azimuth information with corresponding filming apparatus connects most The near apoplexy due to endogenous wind belonging to cluster centre；

(4-9) optimization object function is set up as follows:

$J=\underset{k=1}{\overset{T}{\Σ}}\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\ω}}_k{\mathrm{\μ}}_{kj}^2{||o_k-v_j^{\left(p\right)}||}_2^2,$

Constraints is:

Wherein k is the sequence number of frame of video, and j is the classification of cluster centre, j ∈ [1, C], μ_kjIt is parameter to be solved, υ_jPoly- Class center, p is current iteration number of times；

(4-10) when initializing, if p=0,The vector that initial value is jth cluster centre；

(4-11) μ is calculated_kj:

(4-12) according to above-mentioned result of calculation, μ is updated_kjValue, calculate μ_kj:

${\mathrm{\μ}}_{kj}={\mathrm{\μ}}_{kj}/\underset{j=1}{\overset{C}{\Σ}}{\mathrm{\μ}}_{kj}$

(4-13) according to step (4-12) calculated μ_kj, calculate

(4-14) an iteration ends threshold epsilon is set, ifThen make p=p + 1, and return step (4-11), ifThen carry out step (4-15), ε Span be: 0.01～0.001；

(4-15) by following formula, an initial key frame set K={t is obtained₁,t₁,…,t_C}:Wherein j ∈ [1, C]；

(4-16) similarity of the frame of video characteristic information of any two width frame of video in above-mentioned initial key frame set K is calculatedWherein i, j ∈ [1, C]；

(4-17) similarity threshold is set, in traversal step (4-16) calculated initial key frame set K Any two frames, calculate the similarity of the frame of video characteristic information of any two framesWith similarity threshold Compare, ifAndFrom above-mentioned initial key frame set K, then delete t_j； IfAndFrom above-mentioned initial key frame set K, then delete t_i；IfIn above-mentioned initial key frame set K, then retain t_iAnd t_j, repeating this step, the set K obtained is Key frame of video, the span of δ is: 0.2～0.3.