CN101026695A - Anchorperson frame automatic extracting method and system based on news video structure characteristics - Google Patents

Abstract

Based on structural feature of news video, the invention discloses method for automatic picking up news video port broadcast frame. Maximal feature of the method is that pattern plate of port broadcast frame is not needed to give in advance. Through establishing meaningful similar image set, based on inherent characters of port broadcast frames in whole time period of news video as well as time redundancy of port broadcast frame, the disclosed method uses clustering method to extract out set of port broadcast frame in each time period from meaningful similar image set under condition without set of port broadcast frame. Based on timestamp of set of port broadcast frame in each time period, the invention can realize automatic partitioning news video based on port broadcast frames. The method possesses high accuracy and universality for extracting out set of port broadcast frame.

Description

Mouth based on the news video architectural feature is broadcast frame extraction method and system

Technical field

The present invention is a kind of structure according to news video, extracts mouth automatically and broadcasts frame, and the news video unit that realization is broadcast frame based on mouth is divided a kind of method and system of processing automatically.

Background technology

News video has distinct architectural feature, and the feature of all kinds of news programs in each area has stronger similarity.Accurately the starting point and the end point of news unit in the news video of location are the bases that the news dividing elements is handled automatically.Wherein, sign news unit starting point and end point most important characteristic are exactly that mouth is broadcast frame (news anchor's picture), therefore selecting mouth automatically and broadcast the key issue that frame is the automatic division in news unit in news video, is the core content of realizing news video automatic indexing, retrieval.

Broadcasting frame detection method for existing mouthful, be to extract mouthful all kinds of characteristics of image of broadcasting frame mostly, and counterpart is broadcast the class methods that frame is discerned on this basis.As document " video port of rule-based analysis broadcast frame detect " (Yu Junqing, Yan east, etc.Computer engineering and application, 2004 (6): 84-86) proposed to utilize the mouth of video dominant hue distribution characteristics to broadcast frame detection method, this method very moment is broadcast and is occurred picture-in-picture in the frame or easily produce flase drop when the curtain below literal occurs and disturbs frequently; Document " Knowledge guided in video database " (Swanberg D, Shu C F, Jian R.In:Nibblack W ed.IS﹠amp; T/SPIE.San Jose, CA:SPIE.1993:13 ~ 14.) mode of mating with prior structure standard form is carried out the detection that mouth is broadcast frame, for satisfying the needs that real-time detects, the quantity of standard form is restricted, thereby is difficult to adapt to the news program (news anchor itself or host's dressing meeting change) of different times; Document " the news video mouth is broadcast the research of frame detection method " (Ma Yufei, Bai Xuesheng, etc.The software journal, 2001:12 (3): 377 ~ 382) propose a kind of news video analytical method based on knowledge, the template matches of two-stage, this method with the bust profile as common template, ignore mouth and broadcast the effect of frame background color, and match dedicated templates with common template, obtain certain effect.But the content of news program is multifarious, and the bust picture that non-mouth is broadcast frame also can often appear in the video, and especially, the host's picture in some news program is non-standard bust, so be inappropriate with the standard bust as common template.In sum, existing mouth is broadcast the frame recognizer and is often only considered a mouthful picture feature of broadcasting frame, emphasizes with the template matches to be that identification is detected on the basis, the big complexity height of computing cost, and be difficult to adapt to all news programs, versatility is very limited.

Although there are differences between the different news videos, their mouth is broadcast frame and still had following common characteristic: 1) mouthful broadcasting the frame picture extensively is uniformly distributed among the whole section news, and time span is the longest, through the whole story of news video; 2) every section mouth is broadcast the frame picture and can be continued to occur a period of time, and characteristics of image remains unchanged therebetween.

It should be noted that the scene in many news videos, interview picture, profile picture and static text screen as the reporter and all satisfy These characteristics 2).Satisfy characteristics 1 and have only mouth to broadcast the frame scene).

The present invention is according to the architectural feature of above-mentioned news video, proposed a kind of no template mouth of broadcasting the frame characteristic distributions based on mouth and broadcast the frame detection algorithm, has characteristics such as practical, applied range, adaptive capacity are strong.Experimental result proves, the mouth of this method in basically can the various news videos of reliable recognition broadcast the frame picture.

Summary of the invention

The objective of the invention is not broadcast frame and extract a kind of method of having set up automatically, for the automatic division of news video unit provides a kind of method and system with good versatility for news video has masterplate condition end opening.

The present invention includes the content of the following aspects:

(a) obtain the similar diagram image set in each stage of video by the similarity analysis of sequence of video images; (b) similarity analysis is undertaken by three stage matching processs; (c) concentrate the descriptor frame of extraction one two field picture frame as each image set from each similar image, be called key frame, form the key frame collection thus, this image set exists with certain characteristic formp of image; (d) utilize the architectural feature of news video, concentrate from key frame by clustering method and extract the different periods, multi-form mouth is broadcast frame; (e) broadcast the automatic division that the frame collection carries out the news video unit based on mouth.

Content (a) comprising: (a1) similar collection predefine masterplate chooses; (a2) feature extraction of image.

Content (b) comprising: (b1) average matching process; (b2) histogram matching; And (b3) line segment gray scale matching process.

Content (c) comprising: the condition that (c1) defines meaning similar diagram image set according to the news video architectural feature; (c2) from the concentrated key frame of choosing of similar image.

Content (d) comprising: (d1) foundation of news video architectural feature; (d2) clustering method; And (d3) mouth is broadcast frame and is extracted.

Content (e) comprising: (e1) the time started point of the two kinds of forms in news video unit; (e2) news video unit concluding time point.

Description of drawings

Fig. 1 is the news video structure chart

Fig. 2 extracts the cluster analysis flow chart that mouth is broadcast frame

Embodiment

Main feature of the present invention is exactly to utilize the architectural feature of news video, broadcast under the prerequisite of frame masterplate at no predefined mouthful, by clustering method, automatically extract mouth and broadcast frame, thereby broadcast the similar collection of frame according to each mouthful, determine to contain mouth and broadcast the frame sequence frame and do not contain a starting point and the end point that mouth is broadcast the news video unit of frame sequence frame, realize the automatic division of news video unit.

Divide from semanteme, one section news video unit mainly by head, run-out, mouth is broadcast the frame camera lens and some news camera lenses are formed.Broadcast the frame characteristics by the priori of news video structure and mouth, can Reasonable Orientation news head and run-out, therefore as long as the characteristics of broadcasting frame according to mouth it is separated and can achieve the goal with the news scene.

1. image characteristics extraction

Excise detected two field picture lower edge and may comprise the part of character area, the influence of avoiding literal to occur.To be left several (being the five equilibrium pattern under the general situation) image subblocks of part, take the way of nine image subblocks of five equilibrium in the system of the present invention.Add up the gray average and the grey level histogram of each sub-piece respectively, for reducing computing cost, grey level histogram is after smoothing processing, and histogrammic tonal gradation is 64.When the news mouth is broadcast frame and picture-in-picture occurred, owing to take the way of image block, only can be influential to the gray average and the grey level histogram of minority image subblock; Gray value along image on the equally spaced collection of the horizontal or longitudinal direction p bar line segment, p too conference causes computing cost to increase, and then contains face very little too for a short time, in the system of the present invention, selects p=9.

2. the predefine masterplate chooses

When analyzing beginning, carry out characteristic matching with first two field picture of video as masterplate in advance and follow-up picture frame, find out similar image collection.After treating that the set of last similar image is determined, as new pre-masterplate, and continue above-mentioned matching operation, find out new similar image set with the back frame of tail frame in the last similar image set.

3. three stage matching process of image

For accelerating the matching operation process, according to the size of matching process operand, the matching ratio that carries out image by mode from less to more, detailed process is:

(1) presses formula ${\mathrm{\ρ}}_1^i=1-\sqrt{\frac{{|\mathrm{average}}_m^i-{\mathrm{avergage}}_c^i|}{255}}$ (i=1,2 ..., 9), nine image subblocks of current frame image and the corresponding image subblock average of predefine masterplate are mated, wherein average _m ⁱBe the average gray of pre-i image subblock of template, average _c ⁱBe the mean value of i image subblock of detected frame, ρ ₁ ⁱ( $0\≤{\mathrm{\ρ}}_1^i\≤1$ ) be i image subblock average matching factor.If nine coefficient ρ ₁ ⁱIn majority (native system is defined as six) coefficient is arranged greater than threshold value C ₁, the match is successful then to judge average, changes (2) over to and carry out the histogram coupling.If above-mentioned condition does not satisfy, then the decidable present frame does not match with pre-masterplate, and with the pre-masterplate of present frame as next similar diagram image set, seeks new similar diagram image set by three stage matching process.

(2) press formula ${\mathrm{\ρ}}_2^i=\frac{\underset{n=0}{\overset{64}{\mathrm{\Σ}}}(f_i\left(n\right)-{\stackrel{\‾}{f}}_i)(g_i\left(n\right)-{\stackrel{\‾}{g}}_i)}{\sqrt{\underset{n=0}{\overset{64}{\mathrm{\Σ}}}{(f_i\left(n\right)-{\stackrel{\‾}{f}}_i)}^2}\sqrt{\underset{n=0}{\overset{64}{\mathrm{\Σ}}}{(g_i\left(n\right)-{\stackrel{\‾}{g}}_i)}^2}}$ (i=1,2 ..., 9) and nine image subblocks of current frame image and the pre-pairing image subblock of masterplate are carried out histogram coupling, wherein f _i(n) be the grey level histogram of i image subblock of pre-template, g _i(n) be the grey level histogram of i image subblock of detected frame,

With

Be respectively the grey level histogram average of pre-template and f image subblock of detected frame, ρ ₂ ⁱ( $-1\≤{\mathrm{\ρ}}_2^i\≤1$ ) be i image subblock histogram coupling similarity factor.As nine matching factor ρ ₂ ⁱIn have six coefficients greater than coefficient greater than threshold value C ₂, the match is successful then to judge histogram, changes the coupling that (3) carry out the straight line intensity profile over to.If above-mentioned condition is false, judge that then this present frame and pre-masterplate do not match, and with the pre-masterplate of this frame as next similar diagram image set.

(3) press formula ${\mathrm{\ρ}}_3^i=\frac{\underset{n=m}{\overset{N-m}{\mathrm{\Σ}}}(F_i\left(n\right)-{\stackrel{\‾}{F}}_i)(G_i\left(n\right)-{\stackrel{\‾}{G}}_i)}{\sqrt{\underset{n=m}{\overset{N-m}{\mathrm{\Σ}}}{(F_i\left(n\right)-{\stackrel{\‾}{F}}_i)}^2}\sqrt{\underset{n=m}{\overset{N-m}{\mathrm{\Σ}}}{(G_i(n+k)-{\stackrel{\‾}{G}}_i)}^2}}$ (k=-m ..., 0 ... m) the intensity profile curve to nine straight lines on the relevant position of nine straight line intensity profile curves of current frame image and pre-masterplate carries out the gray scale matching ratio, wherein (i=1,2 ..., p), k is the move left and right distance, and N is the sum of pixel on the line segment, F _i(n) and G _i(n) be intensity profile curve on pre-template and the detected frame i bar line segment,

With

Be gray average on pre-template and the detected frame i bar line segment, ρ ₃ ⁱ( $-1\≤{\mathrm{\ρ}}_3^i\≤1$ ) be i bar line segment intensity profile curve similarity factor.In the matching process, consider and mouthful broadcast in the frame sequence, frame possibility slightly rocking in front and back because of host's head, cause the skew of regional area intensity profile, therefore the matching way of straight line intensity profile curve has taked the mode of move left and right coupling to carry out, and therefrom selects the matching factor ρ that maximum matching factor is made straightway intensity profile curve ₃ ⁱIn nine straight line intensity profile Curve Matching coefficients, six coefficients are arranged greater than threshold value C ₃, then judge present frame with masterplate coupling, and the similar image that this frame is included into pre-masterplate place concentrated, get the next frame of present frame then, repeating step (1) (2) (3) is until the last frame image that finds this similar diagram image set.If above-mentioned condition is false, judge that then present frame and pre-masterplate do not match, with the pre-masterplate of present frame as next similar diagram image set, repeating step (1) (2) (3) is until finding next new similar diagram image set.

By above-mentioned matching process, whole section news video can be divided into a series of similar image set.

4. choosing and pairing key frame of meaningful similar diagram image set

In the news video, host's picture is longer general perdurabgility, based on this fact, can be to similar image set the carrying out judgement of time redundancy, be less than or equal to threshold value T1 when the perdurabgility of certain similar diagram image set, judge that then this similar diagram image set is that non-mouth is broadcast the similar collection of frame, and this similar diagram image set is removed.When perdurabgility of similar diagram image set greater than T1, then defining this similar diagram image set is significant similar diagram image set.So-called significant similar diagram image set refers to this similar diagram image set and meet the feature that mouth is broadcast frame similar diagram image set on time redundancy, and promptly this similar diagram image set is that possible mouth is broadcast frame similar diagram image set.Remove the key frame of a two field picture from concentrated of each significant similar image, produced the key frame images collection thus, and write down the first frame of each similar diagram image set and the pairing timestamp of tail frame as this similar diagram image set.

5. utilizing clustering method to concentrate from significant similar image selects mouth multi-form each period, the same news video and broadcasts frame

According to the Structural Characteristics of news video, the news mouth is broadcast frame and generally can be play in certain news unit and occur once more after finishing, and this shows that the news mouth broadcasts frame and not only possess similar redundancy in a certain period, and also possesses similar redundancy on the difference periods; Broadcast between the frame similar diagram image set at two different period mouths, generally have long time different news camera lenses to occur.Therefore, mouth broadcast the similar collection of frame occur once more must experience long time; The 3rd feature that mouth is broadcast the similar collection of frame is that mouth is broadcast still similar those non-mouthfuls of the similar collection of frame different periods with other in whole news video and broadcast frame similar diagram image set and compare, have maximum decentralization, it is to be distributed in the whole news video in a kind of uncertain mode that mouth is broadcast frame.Based on these features, can comform by clustering method and heterogeneously broadcast the similar collection of frame like selecting outlet in the image set.Specific practice is as follows:

If the frame sequence of news video consists of Clip={f ₁, f ₂..., f _N, by 3. described three stage matching process video is divided into a series of similar diagram image sets, get rid of insignificant similar diagram image set by 4. described methods, and therefore obtain a series of significant similar diagram image sets

KeyScene＝{c ₁，c ₂，...，c _M} (M＜N)

Get a two field picture as key frame concentrated of each significant similar image, and therefore obtain the key frame collection

KeyFrame＝{x ₁，x ₂，...，x _M}

Construction set X _p(p=1,2 ..., K) also claim corresponding with it x _iKey element for this set.

X _p={ x _q| x _qWith x _pBe complementary (p≤M, M key frame number)

Thus, get the set X={X of dissimilar point set in twos ₁, X ₂..., X _KK＜M wherein.

Note ${\mathrm{\ξ}}_p\left(x_q\right)=\left\{\begin{array}{c}1,x_q\∈X_p\\ 0,x_q\∉X_p\end{array}\right.,$ Claim $n\left(X_p\right)=\underset{q=1}{\overset{k}{\mathrm{\Σ}}}{\mathrm{\ξ}}_{p(}\left(x_q\right)$ Be point set X _pSize tolerance.

Note $\mathrm{\ζ}(x_m,x_n)=\begin{array}{c}\left\{\begin{array}{c}1,(x_m\∈X_p,x_n\∈X_q,p=q)\\ \∞,(p\≠q)\end{array}\right.\end{array}(p,q=1,\·\·\·,K)$

$\mathrm{\η}(x_m,x_n)=\left\{\begin{array}{c}1,(x_m\∈X_p,x_n\∈X_q,p=q)\\ -\∞(p\≠q)\end{array}\right.(p,q=1,\·\·\·,K)$

d ₂(x _m，x _n)＝η(x _m，x _n)|t(m)-t(n)|(m，n＝1，…，M)

Here t (i) is similar diagram image set C _iThe 1st frame is to video start frame f ₁Time span for  p, get D _1p= _d1 (x _m, x _n) x _m∈ X _p, x _n∈ X _q, D _2p={ d ₂(x _m, x _n) | x _m∈ X _p, x _n∈ X _q), wherein (q=1 ..., K), claim d ₁(X _p)=supD _1pBe point set X _pMinimum diameter, d ₂(X _p)=infD _2pBe point set X _pMaximum gauge.So $X_p^a=\left\{X_p\right|n\left(X_p\right)>T_2,d_1\left(X_p\right)>T_3,d_2\left(X_p\right)>T_4,p=1,\·\·\·,K\}$ For the mouth of same form is broadcast the frame collection, this collection is broadcast key frame for similar mouth of different periods and is formed.

By above-mentioned clustering method, different periods, multi-form mouth can be broadcast frame (broadcast frame such as the single mouth that occurs in the same news video, double mouth is broadcast frame) and concentrate from similar image respectively and separate.

6. broadcast the news video dividing elements of frame based on mouth

By clustering method, obtain different period mouths and broadcast the similar diagram image set, write down each mouthful and broadcast the timestamp that similar image is concentrated first frame and tail frame, broadcast between the first frame of frame similar diagram image set and the pairing sun of tail frame according to different period mouths and to stab, provide respectively to comprise and mouthful broadcast the frame sequence frame and do not comprise the time started point that mouth is broadcast the news unit of frame sequence frame, and can contain the time started point that mouth is broadcast the frame sequence frame by next news unit, determine the concluding time point of current news unit.By the time starting point of two kinds of news unit, the news video unit can be made two kinds of different divisions, a kind of is that the news unit is to begin and begin with host's picture; Another kind is the appearance that the news unit of division does not comprise host's picture.

7. result of the test

According to the method that invention proposes, we choose the news hookup of the Chinese Central Television's one cover, news when international, Shanghai TV Station global news station, and 13 kinds of different news programs such as news mill have carried out mouth and have broadcast frame and detect test on DELL Pentium 4PC machine.Comprise contents such as advertisement, news head, news in brief, news unit in the news video.Wherein part test result is as shown in the table:

News program	Double picture	Single picture	Mouth is broadcast the frame species number	Mouth is broadcast the frame sum	Flase drop	Omission	Accuracy rate
								News hookup	2	9	3	11	0	0	100％
News when international	0	9	1	9	0	0	100％
								Global news station	0	9	1	9	0	0	100％
The news mill	14	26	3	40	0	0	100％

From experimental result, the present invention has obtained gratifying result, and can get rid of the influence of advertisement, the first-class disturbing factor of newsreel.To these 13 kinds of different news programs, the verification and measurement ratio rate that mouth is broadcast frame reaches 100%, and does not have the generation of flase drop situation.Prove absolutely that the inventive method takes into full account the distribution character that mouth is broadcast frame.This method is broadcast under the prerequisite of frame masterplate at Wu Kou, has very high detection accuracy and good universality and practicality.

Claims (13)

1. broadcast frame extraction method and system based on the mouth of news video architectural feature, comprising: the similarity analysis according to video obtains significant similar diagram image set, and can produce a key frame images feature set thus; By clustering method, can concentrate from significant similar image to extract the day part mouth and broadcast frame similar diagram image set; Broadcast the timestamp of frame similar diagram image set based on mouth, can carry out broadcasting the automatic division of the news unit of frame news video based on mouth.

2. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 1, it is characterized in that this method is a kind of analysis afterwards processing method.

3. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 1, it is characterized in that this method is a kind of no template method that the chosen in advance mouth is broadcast the frame masterplate that need not.

According to claim 1 based on the news video architectural feature mouthful broadcast frame extraction method and system, it is characterized in that mouth that this method can utilize the inventive method to extract broadcasts the automatic division of the news video unit that frame carries out.

5. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 1, it is characterized in that it is to concentrate from significant similar image to extract that mouth is broadcast frame.

6. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 1, it is characterized in that it is to extract along the architectural feature of time domain direction according to the news video structure that mouth is broadcast frame.

7. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 1, it is characterized in that the similar diagram image set is to utilize the similitude coupling of characteristics of image to obtain.

According to claim 2 based on the news video architectural feature mouthful broadcast frame extraction method and system, it is characterized in that mouth that this method can be used as other broadcasts the check or the error correction method of frame extract real-time system.

According to claim 2 based on the news video architectural feature mouthful broadcast frame extraction method and system, it is characterized in that this method can be after news video have been play certain hour or after news video all finished, counterpart was broadcast frame and extracted automatically.

10. broadcast frame extraction method and system according to the mouth based on the news video architectural feature of claim 4, it is characterized in that this method can combine with any news captions extraction system, realize the more meticulous division of news video unit.

11. according to claim 4,8 based on the news video architectural feature mouthful broadcast frame extraction method and system, it is characterized in that this method can broadcast the frame real time extracting method with any one mouthful and combine use.

12. the mouth based on the news video architectural feature according to claim 5 is broadcast frame extraction method and system, the matching process that it is characterized in that characteristics of image is to finish by the coupling of gray average, grey level histogram, image cathetus intensity profile curve.

13. according to claim 6 based on the news video architectural feature mouthful broadcast frame extraction method and system, the architectural feature that it is characterized in that news video is that same period mouth is broadcast frame and possessed bigger temporal redundancy; Two different period mouths are broadcast long time span between the frame; Mouth is broadcast frame and is distributed in whole news video on the period, possesses maximum dispersion.

Images