CN107920275A - Video broadcasting method, device, terminal and storage medium - Google Patents
Video broadcasting method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN107920275A CN107920275A CN201711469028.XA CN201711469028A CN107920275A CN 107920275 A CN107920275 A CN 107920275A CN 201711469028 A CN201711469028 A CN 201711469028A CN 107920275 A CN107920275 A CN 107920275A
- Authority
- CN
- China
- Prior art keywords
- video
- video flowing
- prediction
- time point
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims description 22
- 238000005070 sampling Methods 0.000 claims description 16
- 230000003247 decreasing effect Effects 0.000 claims description 14
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 12
- 210000002569 neuron Anatomy 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000003066 decision tree Methods 0.000 description 5
- 239000000284 extract Substances 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000012706 support-vector machine Methods 0.000 description 4
- 230000003139 buffering effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 210000004027 cell Anatomy 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 206010016256 fatigue Diseases 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 210000003127 knee Anatomy 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4333—Processing operations in response to a pause request
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8547—Content authoring involving timestamps for synchronizing content
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
This application discloses a kind of video broadcasting method, device, terminal and storage medium, belong to Computer Applied Technology field.This method includes：Prediction video flowing is obtained, the prediction video flowing is the video flowing for having cached in currently playing video and not yet having played；By the prediction video flowing input prediction model, the first time out point predicted, the prediction model trains to obtain according to Sample video stream and sample time point；When the video playing is to the first time out point, pause plays the video, by the above method, by obtaining the prediction video flowing for having cached in video and not yet having played.Scheme shown in the application directly predicts the time point for being adapted to pause by the prediction model pre-set from video flowing, and pause waits the enough videos of caching when being played to the time point that this is adapted to pause, avoid network it is obstructed when at the time point for being not suitable for pause there is a situation where interim card, improve the result of broadcast of Online Video.
Description
Technical field
The invention relates to Computer Applied Technology field, more particularly to a kind of video broadcasting method, device, terminal
And storage medium.
Background technology
User by the videoconference client in terminal either Webpage requesting video when videoconference client or webpage in regard
Frequency player from the video flowing of server pull user's program request and is buffered in local on backstage, while plays regarding for caching on foreground
Frequency flows.
When the network bandwidth between terminal and server is smaller, in fact it could happen that video player is from server pull video
Situation of the speed of stream less than the speed that foreground plays.In the related art, video player is when playing Online Video, can be with
The playable duration for the video flowing that periodic detection has been cached and do not played, when playable duration is less than a certain thresholding, pause regards
Frequency plays, to wait backstage to pull and cache more video flowings.
The content of the invention
Video broadcasting method, device, terminal and storage medium provided by the embodiments of the present application.The technical solution is as follows：
First aspect, there is provided a kind of video broadcasting method, the described method includes：
Prediction video flowing is obtained, the prediction video flowing is the video for having cached in currently playing video and not yet having played
Stream；
By it is described prediction video flowing input prediction model, the first time out point predicted, it is described first pause when
Between point be time point in the corresponding reproduction time section of the prediction video flowing, the prediction model be according to Sample video stream and
What sample time point was trained；
When the video playing is to the first time out point, pause plays the video.
Second aspect, there is provided a kind of video play device, described device include：
Video flowing acquisition module, for obtain predict video flowing, it is described prediction video flowing be in currently playing video
Caching and the video flowing not yet played；
Prediction module, for predicting video flowing input prediction model, the first time out point predicted, institute by described
It is the time point in the corresponding reproduction time section of the prediction video flowing to state the first time out point, and the prediction model is basis
What Sample video stream and sample time point were trained；
Suspend module, for when the video playing is to the first time out point, pause to play the video.
The third aspect, there is provided a kind of terminal, the memory that the terminal includes processor, is connected with the processor,
And the programmed instruction on the memory is stored in, the processor realizes that first aspect provides when performing described program instruction
Video broadcasting method.
Fourth aspect, a kind of computer-readable medium are stored thereon with programmed instruction, and described program instruction is held by processor
The video broadcasting method that first aspect provides is realized during row.
The beneficial effect that technical solution provided by the embodiments of the present application is brought is：
By obtaining the prediction video flowing for having cached in video and not yet having played, prediction video flowing input prediction model is obtained
To the first time out point of prediction, and when video playing is to the first time out point, pause plays video, i.e. the application institute
The scheme shown directly predicts the time point for being adapted to pause by the prediction model pre-set from video flowing, and is playing
Pause, which waits, when being adapted to the time point of pause to this caches enough video, avoid network it is obstructed when be not suitable for pause
There is a situation where interim card at time point, the result of broadcast of Online Video is improved.
Brief description of the drawings
In order to illustrate more clearly of the technical solution in the embodiment of the present application, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present application, for
For those of ordinary skill in the art, without creative efforts, other can also be obtained according to these attached drawings
Attached drawing.
Fig. 1 is the structure diagram for the Online Video play system that the application one embodiment provides；
Fig. 2 is the flow chart of the video broadcasting method shown in one exemplary embodiment of the application；
Fig. 3 is a kind of schematic diagram down-sampled to video flowing progress that embodiment illustrated in fig. 2 is related to；
Fig. 4 is a kind of model prediction schematic diagram that embodiment illustrated in fig. 2 is related to；
Fig. 5 is another model prediction schematic diagram that embodiment illustrated in fig. 2 is related to；
Fig. 6 is the block diagram for the video play device that the application one embodiment provides；
Fig. 7 is the block diagram for the terminal that the application one embodiment provides.
Embodiment
To make the purpose, technical scheme and advantage of the application clearer, below in conjunction with attached drawing to the application embodiment party
Formula is described in further detail.
First, to this application involves some nouns be introduced.
Prediction model：It is a kind of machine learning mould for the time out point being used in the data prediction video according to input
Type.
Prediction model is obtained according to Sample video stream and sample time point training.Wherein, Sample video stream is advance
Which or which time point is adapted to the video flowing of pause in known corresponding reproduction time section.Sample time point refers in sample
It is adapted to the time point of pause in video flowing.Wherein, sample time point can be that mark personnel mark in Sample video stream in advance
The time point of acquisition.
Above-mentioned prediction model can be disaggregated model, such as, which can be neural network model (such as convolution
Neural network model) and Logic Regression Models etc..
, can be with reference to each in Sample video stream for example mark personnel are in Sample video stream when mark sample time point
The image content and audio content of video frame are labeled, for example, mark personnel can be by the predefined type in Sample video stream
The corresponding play time of picture is labeled as sample time point, wherein, which can be that pure color picture is (such as pure
It is black), the picture not comprising personage or the picture not comprising subtitle etc.；Correspondingly, mark personnel can also be by Sample video stream
In predefined type audio content corresponding time point be labeled as sample time point, wherein, which can be with
Refer to the audio content of no dialogue.Wherein, the selection principle of above-mentioned sample time point, can be in samples played video flowing,
Pause will not excessively influence the viewing experience of user at above-mentioned sample time point, such as, sample time point can be two sections of phases
To a certain play time between the more independent story of a play or opera.
Alternatively, Sample video stream and sample time point are stored in sample storehouse.The sample storehouse includes at least one set of sample, often
Group sample includes one section of Sample video stream and the corresponding sample time point of this section of video flowing.
It should be noted that unless otherwise specified, at the time point involved in the subsequently each embodiment of the application, be video
In playing progress rate time point.Such as time point for 0 it is small when 0 divide 30 seconds, refer to the 30th second in video playing progress.
Alternatively, prediction model includes but not limited to：Deep neural network (Deep Neural Network, DNN) mould
Type, Recognition with Recurrent Neural Network (Recurrent Neural Networks, RNN) model, convolutional neural networks (Convolutional
Neural Network, CNN) it is model, support vector machines (Support Vector Machine, SVM), embedded
(embedding) model and gradient lifting decision tree (Gradient Boosting Decision Tree, GBDT) model etc.
At least one of model, the present embodiment will not enumerate herein.
Wherein, DNN models are a kind of deep learning frames.DNN models include input layer, at least one layer of hidden layer (in or
Interbed) and output layer.Alternatively, input layer, at least one layer of hidden layer (or intermediate layer) and output layer include at least one god
Through member, neuron is handled for docking received data.Alternatively, the quantity of the neuron between different layers can phase
Together；Alternatively, can not also be same.
RNN models are a kind of neutral nets with feedback arrangement.In RNN models, the output of neuron can be under
One timestamp is applied directly to itself, i.e. input of the i-th layer of neuron at the m moment, except (i-1) layer neuron this when
Outside the output at quarter, its own output at (m-1) moment is further included.
CNN is a kind of depth feed forward-fuzzy control, generally includes the processing such as convolutional layer, pond layer and activation primitive
Layer, the artificial neuron of CNN can respond the surrounding cells in a part of coverage, have well in scenes such as image procossings
Performance.
Support vector machines is a kind of sorting technique based on Statistical Learning Theory.When giving one group of training sample, support
Vector machine learns to carry out two class classification by using supervised.Order interval is maximum on feature space, and final realize will be given
Training sample is divided into two classes.
Embedding models are to be based on entity and relation distribution vector representation, by the relation in each triple example
Regard the translation from entity head to entity tail as.Wherein, triple example includes main body, relation, object, and triple example can be with table
It is shown as (main body, relation, object)；Main body is entity head, and object is entity tail.Such as：The father of Xiao Ming is big bright, then passes through three
Tuple example is expressed as (Xiao Ming, father are big bright).
GBDT models are a kind of decision Tree algorithms of iteration, which is made of more decision trees, and the result of all trees is tired out
Add up as final result.Each node of decision tree can obtain a predicted value, and by taking the age as an example, predicted value is to belong to
The average value at owner's age of age corresponding node.
Fig. 1 is the structure diagram of the Online Video play system shown in the exemplary embodiment of the application, this is
System includes terminal 110 and server 120.
Terminal 110 has communication function, and terminal 110 includes but not limited to：Mobile phone, tablet computer, wearable device, intelligence
At least one of energy robot, smart home device, pocket computer on knee and desktop computer.
Operating system 111 and client 112 are installed in terminal 110.
Alternatively, operating system 111 includes but not limited to：Windows systems, linux system, IOS (iPhone OS) system
System, Android (Android) system or windowsPhone systems etc..
Client 112 has Online Video playing function, such as, which can be video player client
Or webpage client of embedded video playing component etc..Client 112 is asked when playing Online Video to server 120
The corresponding video flowing of the video is pulled, and the video flowing pulled is buffered in local.When playing video, client 112 according to
Acquiescence or the playing progress rate of user's instruction play the video flowing cached.Also, during video is played, client 112
Prediction video flowing can be obtained, which is the video flowing for having cached in currently playing video and not yet having played；Visitor
Family end 112 will predict video flowing input prediction model, and the first time out point predicted, which is pre-
Survey the time point in the corresponding reproduction time section of video flowing；Client 112 when video playing is to the first time out point,
Pause plays the video.
Alternatively, terminal 110 is connected by wireless network or cable network with server 120.
Server 120 can be an independent server host；Or or multiple servers host form
Server cluster.
Server 120 is used to provide the video flowing that client 112 asks to pull.
Alternatively, the quantity of above-mentioned terminal 110 can be at least one that the quantity of server 120 can also be at least one
A, the present embodiment is not construed as limiting this.
Alternatively, in the application, wireless network or cable network use standard communication techniques and/or agreement.Network is usual
For internet, it may also be any network, includes but not limited to LAN (Local Area Network, LAN), Metropolitan Area Network (MAN)
(Metropolitan Area Network, MAN), wide area network (Wide Area Network, WAN), mobile, wired or nothing
Any combinations of gauze network, dedicated network or Virtual Private Network).In certain embodiments, using including hypertext markup
Language (HyperText Mark-up Language, HTML), extensible markup language (Extensible Markup
Language, XML) etc. technology and/or form represent the data by network exchange.In addition can also use such as safe
Socket layer (Secure Socket Layer, SSL), Transport Layer Security (Trassport Layer Security, TLS), void
Intend dedicated network (Virtual Private Network, VPN), Internet Protocol Security (Internet Protocol
Security, IPsec) etc. conventional encryption techniques encrypt all or some links.In further embodiments, can also make
Substitute or supplement above-mentioned data communication technology with customization and/or the exclusive data communication technology.
Alternatively, the executive agent of each step is the client in terminal in the application, which can be shown in Fig. 1
Online Video play system in client 112.
Fig. 2 is the flow chart of the video broadcasting method shown in the exemplary embodiment of the application.The video playing side
Method includes following steps.
Step 201, prediction video flowing is obtained, which is to have cached in currently playing video and not yet played
Video flowing.
Client can be performed periodically from currently playing video during a certain Online Video is played
The step of caching and the video flowing acquisition prediction video flowing not yet played.For example client is using 1min as a cycle, i.e., every point
Clock, which obtains, once predicts video flowing, and performs subsequent prediction step according to the prediction video flowing got.
Obtain predict video flowing when, video that client can will have been cached in currently playing video and not yet played
In stream, all or part of video flowing is retrieved as above-mentioned prediction video flowing.
Optionally, when obtaining prediction video flowing, in the video flowing that client will have been cached and not yet played, reproduction time
The video flowing of the first scheduled duration the latest is retrieved as prediction video flowing.
In the embodiment of the present application, in order to reduce the complexity of subsequent prediction, while reduce same section of video flowing is multiple
The possibility of prediction video flowing is retrieved as, when obtaining prediction video flowing, client can be obtained only in the video flowing cached,
The a bit of video flowing of reproduction time the latest.For example client can will have been cached and do not played when obtaining prediction video flowing
Video flowing in, reproduction time the latest, and playing duration be 20s video flowing be retrieved as prediction video flowing.For example, it is assumed that client
Hold the reproduction time of video flowing for having cached and not yet having played in the playing duration of whole video 0 it is small when 15 divide 18 seconds to 0
In hour 18 points of sections of 05 second, then obtain predict video flowing when, client can by 0 in the playing duration of video it is small when
17 points 45 seconds to 0 it is small when 18 divide the corresponding video flowing in 05 second section be retrieved as prediction video flowing.
Optionally, prediction video flowing is being obtained, client obtains the current play time point of video with having been cached in video
And the first duration between the latest time point not yet played；When the first duration is in the first scheduled duration and the second scheduled duration
Between when, according to it is default acquisition the cycle perform obtain prediction video flowing the step of, wherein, second scheduled duration be more than first
Scheduled duration.
In practical applications, when the Network status between terminal and server is preferable, or network bandwidth is higher, client
The speed for pulling video flowing is held to be usually not less than the playback rate of video, at this time, it is not necessary to suspend video playing to wait video
The caching of stream.Therefore, in the embodiment of the present application, client can obtain the current of video before prediction video flowing is obtained
The first duration between play time and the play time the latest of the video flowing cached, when first duration is less than a certain
During a threshold value (i.e. above-mentioned second scheduled duration), the duration (i.e. the first scheduled duration) for the prediction video flowing simultaneously greater than to be obtained
When, client can periodically obtain prediction video flowing.
Such as, it is assumed that the second scheduled duration is 1 minute, and the first scheduled duration is 20s, when playable duration (is currently broadcast
The latest time point of video flowing for putting time point and having cached and not yet having played) when being less than 1 minute and being more than 20s, client can
To think that pause waits caching inevitable, client can be obtained with every 10s and once predict video flowing at this time, every time by
Last 20s video flowings in the video flowing of caching are retrieved as prediction video flowing, are more than 1 minute until can play duration, Huo Zhe little
In 20s.
Step 202, video flowing input prediction model, the first time out point predicted will be predicted.
Wherein, the first time out point is the time point predicted in the corresponding reproduction time section of video flowing, and prediction model is
Trained according to Sample video stream and sample time point.
In the embodiment of the present application, client get every time prediction video flowing after, you can with will prediction video flowing make
For prediction data, it is input in prediction model, is predicted by prediction model and obtain the first time out point.Wherein, this is first temporary
It is in prediction video flowing to stop time point, it is contemplated that user's viewing will not be excessively influenced when being appropriate for the time point of pause, and suspending
The time point of experience.
Optionally, video flowing input prediction model will be predicted, during the first time out point predicted, client is to pre-
Survey video flowing and carry out down-sampled processing, obtain it is down-sampled after video flowing, and will be down-sampled after video flowing input prediction model,
Obtain the first time out point.
In practical applications, predict that the data volume included in video flowing may be relatively more, if according in prediction video flowing
All data be predicted, then the forecasting efficiency of prediction model can be caused relatively low, and if reduce prediction video flowing broadcasting
Length, then may be too short because of prediction video flowing, comprising information content it is very few and cause the accuracy of prediction result to reduce.Cause
This, in order to improve the forecasting efficiency of model, while ensures certain prediction effect, in the embodiment of the present application, can be to prediction
Video flowing carries out down-sampled processing, and by the video flowing input prediction model after down-sampled processing, on the one hand, subtracted by down-sampled
The data volume of few input prediction model, on the other hand, although the down-sampled data volume that input prediction model is greatly reduced,
The useful information content that video flowing after down-sampled is included is not greatly reduced, so as to ensure certain prediction prediction
Accuracy, compared with not carrying out scheme that is down-sampled and directly inputting prediction model to prediction video flowing, by prediction video flowing drop
The scheme of input prediction model after sampling, can ensure enough forecasting accuracies while forecasting efficiency is greatly improved.
Optionally, down-sampled processing is carried out to prediction video flowing, obtain it is down-sampled after video flowing when, client is to prediction
Video flowing carries out drop frame sampling, obtains the first video flowing, is included in the first video flowing according to default sample rate from prediction video flowing
In each video frame for collecting, include picture frame and audio frame in each video frame；To the picture frame in each video frame into
Row resolution decreasing samples, and obtains the picture frame after resolution decreasing；Drop precision sampling is carried out to the audio frame in each video frame, is obtained
Audio frame after precision must drop；By the picture frame after resolution decreasing and drop precision after audio frame be combined into it is down-sampled after regard
Frequency flows.
Due in addition to comprising picture frame, also including the audio frame played with image frame synchronization, and this Shen in video flowing
Please be also otherwise varied for the down-sampled mode of picture frame and audio frame in embodiment.
Please refer to Fig.3, it illustrates the invention relates to a kind of down-sampled schematic diagram is carried out to video flowing.Such as
Shown in Fig. 3, video flowing is considered as the stream data being made of several video frame, includes what is be played simultaneously in each video frame
Picture frame and audio frame, as shown in figure 3, when carrying out drop frame sampling to prediction video flowing, are first according to default default sampling
Rate, in units of video frame, drop frame sampling is carried out to prediction video flowing, such as, it 1/4 is that client can be when default sample rate
To sample out a video frame in every adjacent 4 video frame, and using the video flowing of each video frame sampled out composition as
First video flowing.By dropping frame sampling, client can significantly subtract in the case where keeping the reduction that information content will not be excessive
The frame number of few video frame.
In figure 3, after client is by dropping frame sampling the first video flowing of acquisition, respectively to each in the first video flowing
A picture frame and each audio frame are further sampled.
When each picture frame in the first video flowing samples, client can carry out each picture frame drop point
Resolution processing, obtains the picture frame after resolution decreasing, such as, it is assumed that the original resolution of each picture frame is 1080p, client
The resolution ratio of each picture frame can be reduced to 480p from 1080p, in picture frame is ensured by way of resolution decreasing sampling
Comprising information content in the case of, the data volume of each picture frame is greatly decreased.
When each audio frame samples in the first video flowing, client can be to each audio frame into being about to precision
Processing, obtains the audio frame after drop precision, wherein, the precision of audio frame can be represented by bit depth, such as, it is assumed that it is each
The original bit depth of audio frame is 16, and client can be by way of decrease depth-sampling, by the locating depth of each audio frame
Degree is reduced to 8, and in the case of the information content included in ensureing audio frame, the data volume of each audio frame is greatly decreased.
In figure 3, client obtain resolution decreasing after picture frame and will after the audio frame after precision, combine obtain drop adopt
Video flowing after sample.
Please refer to Fig.4, it illustrates the invention relates to a kind of model prediction schematic diagram.As shown in figure 4, it is directed to
The situation comprising picture frame and audio frame, prediction model can only extract the picture frame in video flowing, and root at the same time in video flowing
The first time out point is obtained according to the picture frame prediction extracted；Alternatively, prediction model can also only extract the sound in video flowing
Frequency frame, and the first time out point is obtained according to the audio frame prediction extracted；Regarded alternatively, prediction model can also extract at the same time
Picture frame and audio frame in frequency stream, and combine the picture frame in video flowing and audio frame the first time out point of acquisition.
Optionally, individual machine learning model is included in prediction model, picture frame and audio frame in video flowing is combined
When obtaining the first time out point, the picture frame in video flowing and audio frame are inputted the list by client together as input data
A machine learning model, to obtain above-mentioned first time out point.
Optionally, two machine learning models, i.e. the first prediction model and the second prediction model are included in prediction model,
It will predict video flowing input prediction model, during the first time out point predicted, client will be predicted each in video flowing
A picture frame inputs the first prediction model, obtains the first time point set of prediction, at least one is included in first time point set
A time point；And each audio frame predicted in video flowing is inputted into the second prediction model, obtain the second time point set of prediction
Close, at least one time point is included in the second time point set；Obtained further according to first time point set and the second time point set
Obtain the first time out point.
Refer to Fig. 5, it illustrates the invention relates to another model prediction schematic diagram.It is as shown in figure 5, pre-
Survey in model and include the first prediction model and the second prediction model, wherein the first prediction model is to be used to be carried out temporarily according to picture frame
The model of point prediction between stopping time, the second prediction model are the models that time out point prediction is carried out according to audio frame, are being predicted
Cheng Zhong, client will predict prediction model regarding from input after video flowing (or down-sampled after video flowing) input prediction model
Frequency extracts picture frame and audio frame respectively in flowing, and the picture frame extracted is input to the first prediction model, obtains by the
The first time point set that the time out point that one prediction model predicts is formed, meanwhile, the sound that prediction model will extract
Frequency frame is input to the second prediction model, the second time point that the time out point for obtaining being predicted by the second prediction model is formed
Set, finally, client synthesis first time point set and the second time point set obtain the first time out point.
Optionally, when obtaining the first time out point according to first time point set and the second time point set, client
End determines to whether there is time point j in the second time point set, time point i is in first time point set for time point i
Random time point, it is poor that the time difference between time point j and time point i is less than preset time；Exist when in the second time point set
During time point j, the first time out point is obtained according to time point i and time point j.
In the embodiment of the present application, the first time out point is obtained in comprehensive first time point set and the second time set
When, the Each point in time during client can gather first time is compared with the Each point in time in the second time point set
It is right, the first time out point is determined according to similar time point in different time set (it is poor no more than preset time to be separated by).
Such as, it is assumed that above-mentioned preset time difference is 2s, the sometime point t in gathering for first time1, the time point
t136 divide 1 second when corresponding reproduction time in video is 0 small, and there is sometime point t in the second time point set2, should
Time point t2Corresponding reproduction time in video for 0 it is small when 36 divide 2 seconds, then t1And t2Between time difference be 1s, less than default
Time difference, at this time, client can be according to t1And t2To obtain the first time out point.
Optionally, when obtaining the first time out point according to time point i and time point j, client by time point i, when
Between point j or the middle time point of time point i and time point j be retrieved as the first time out point.
Such as using time point i as above-mentioned t1, time point j is above-mentioned t2Exemplified by, client can be by t1And t2In it is a certain
A time point is retrieved as the first time out point, alternatively, client can also be by t1And t2Middle time point (i.e. 0 36 divides when small
1 second 500 milliseconds) be retrieved as the first time out point.
Optionally, before it will predict video flowing input prediction model, client is to each picture in prediction video flowing
Face frame carries out color lump processing, obtains the video flowing of color lumpization processing, and the video flowing input prediction model that color lump is handled.
Such as above-mentioned down-sampled processing obtain it is down-sampled after video flowing after, client can also be to after down-sampled
Video flowing in each picture frame carry out color lump processing, obtain color lumpization processing video flowing.
Step 203, when video playing is to the first time out point, pause plays the video.
In the embodiment of the present application, when the client terminal playing video reaches the first time out point that above-mentioned prediction obtains,
Can be with the broadcasting of the automatic pause video, to wait the video flowing for pulling enough data volumes or enough playing durations.
Optionally, when pause plays video, client determines that first is temporary when video playing is to the first time out point
Stop whether there is the second time out point after time point, the second time out point is another pause obtained by prediction model
Time point；When the second time out point is not present after the first time out point, the step of pause plays video is performed.
In the embodiment of the present application, client can periodically obtain prediction video flowing and predict time out point, because
This, when client obtains the first time out point from prediction, during the first time out point is played to, may may proceed to
Predict one or more than one other time out points, at this time, in order to avoid frequently suspending video, ensure to use
Smoothly viewing is experienced at family, and client can continue to play video, until next time out point.
Optionally, every time after definite first time out point, determine to whether there is not yet before the first time out point
The 3rd time out point being played to, if so, then cancelling the 3rd time out point.
In alternatively possible implementation, when client obtains a time out point from prediction, to being played to this
During time out point, if predicting above-mentioned first time out point, video is carried out in order to avoid frequently temporary
Stop, ensure that user smoothly experience by viewing, client can cancel upper time out point, correspondingly, when client is from pre-
Survey and obtain the first time out point, during the first time out point is played to, continue to predict one or one with
On other time out points, then client can also cancel the first time out point.
Optionally, client can obtain what is cached and not yet played when video playing is to the first time out point
The remaining playing duration of video flowing, when the residue playing duration is not more than three scheduled durations, client executing pause plays
The step of video.
In the embodiment of the present application, when client obtains the first time out point from prediction, to when being played to the first pause
Between put during, may because of between terminal and server network environment improve etc. reason, client pull foot
Enough video flowings, at this time, it is not necessary to video is suspended, therefore, when client terminal playing video reaches above-mentioned first pause
During time point, can determine first the remaining playing duration of video flowing that has cached whether long enough, if so, then client can be with
Do not suspend broadcasting, and be to continue with playing video, until next time out point, on the contrary, however, it is determined that the video flowing cached
Remaining playing duration falls short of, then client can suspend video playing at the first time out point, to wait caching more
Video flowing.
Optionally, in the embodiment of the present application, can be with after client suspends video playing at the first time out point
The playable duration for the video flowing for having cached and not yet having played is periodically detected, is more than the 4th scheduled duration when this can play duration
When, continue to play the video.
By the scheme shown in the embodiment of the present application, client is regarded to what is finished receiving during user's viewing video
Frequency stream carries out background process, using advance trained prediction model, determines relatively to be adapted to pause in the progress bar buffered
The time out point of broadcasting；When interim card is inevitable, suspend broadcasting in advance at identified time out point, to wait
Buffering.It is the video of 30 minutes for total duration in a specific example, when residue buffered duration less than 2 minutes,
Client redefined a time out point every 10 seconds.Determination process is as follows：Interception last 20 seconds of progress bar of buffering
Video flowing simultaneously carries out down-sampled processing, and by picture, all down-sampled video flowing inputs to prediction model, prediction model and exports this
The time out point for being adapted to pause to play in 20 seconds, it is automatic at object time point when residue buffered duration less than 20 seconds
Pause plays.
Wherein, above-mentioned prediction model is trained according to training data and obtained, which can be the mark manually provided
The 20s video flowings of time out point, the training of prediction model can be completed before client is developed or equipment is dispatched from the factory, and
And the prediction model can also be by being updated online.
In conclusion the scheme shown in the embodiment of the present application, by obtaining the prediction for having cached in video and not yet having played
Video flowing, the first time out point that prediction video flowing input prediction model is predicted, and it is temporary to first in video playing
When stopping time point, pause plays video, i.e., scheme shown in the application is by the prediction model that pre-sets from video flowing
The time point for being adapted to pause is directly predicted, and pause waits caching is enough to regard when being played to the time point that this is adapted to pause
Frequently, avoid network it is obstructed when be not suitable for pause time point at there is a situation where interim card, improve broadcasting for Online Video
Put effect.
In addition, the scheme shown in the embodiment of the present application, when being played to up to the first time out point, it may be determined that this first
It whether there is other time out points after time out point, alternatively, determine whether residue playing duration exceedes certain threshold value,
If other time out points are not present after the first time out point, alternatively, remaining playing duration is no more than certain threshold
Value, just performs pause step, avoids unnecessary pausing operation.
In addition, the scheme shown in the embodiment of the present application, after determining the first time out point, when can first be suspended
Between put before, and other time out points for being not yet played to are cancelled, to avoid unnecessary pausing operation.
Following is the application device embodiment, can be used for performing the application embodiment of the method.It is real for the application device
The details not disclosed in example is applied, refer to the application embodiment of the method.
Fig. 6 is refer to, the block diagram of the video play device provided it illustrates the application one embodiment, this is regarded
Frequency playing device can be implemented in combination with as some or all of of terminal by software, hardware or both.The device can be with
Including：Video flowing acquisition module 601, prediction module 602 and pause module 603.
Video flowing acquisition module 601, video flowing is predicted for obtaining, and the prediction video flowing is in currently playing video
The video flowing for having cached and not yet having played；
Prediction module 602, for predicting video flowing input prediction model, the first time out predicted by described
Point, the first time out point are the time points in the corresponding reproduction time section of the prediction video flowing, the prediction model
Train to obtain according to Sample video stream and sample time point；
Suspend module 603, for when the video playing is to the first time out point, being regarded described in pause broadcasting
Frequently.
Optionally, the prediction module 602, is specifically used for,
Down-sampled processing, the video flowing after acquisition is down-sampled are carried out to the prediction video flowing；
By it is described it is down-sampled after video flowing input the prediction model, obtain the first time out point.
Optionally, down-sampled processing is being carried out to the prediction video flowing, obtain it is down-sampled after video flowing when, it is described pre-
Module 602 is surveyed, is specifically used for,
Drop frame sampling carried out to the prediction video flowing, obtains the first video flowing, included in first video flowing according to
Default sample rate includes picture frame and audio from each video frame predicted and collected in video flowing, each video frame
Frame；
Resolution decreasing sampling is carried out to the picture frame in each video frame, obtains the picture frame after resolution decreasing；
Drop precision sampling is carried out to the audio frame in each video frame, obtains the audio frame after drop precision；
By the picture frame after the resolution decreasing and it is described drop precision after audio frame be combined into it is described it is down-sampled after
Video flowing.
Optionally, the prediction model includes the first prediction model and the second prediction model, the prediction module 602, tool
Body is used for,
Each picture frame in the prediction video flowing is inputted into first prediction model, obtains the first time of prediction
Point set, includes at least one time point in the first time point set；
Each audio frame in the prediction video flowing is inputted into second prediction model, obtains the second time of prediction
Point set, includes at least one time point in the second time point set；
The first time out point is obtained according to the first time point set and the second time point set.
Optionally, when obtaining first pause according to the first time point set and the second time point set
Between when putting, the prediction module 602, is specifically used for,
For time point i, determine to whether there is time point j in the second time point set, the time point i is described
Random time point in first time point set, the time difference between the time point j and the time point i are less than preset time
Difference；
When in the second time point set there are during the time point j, according to the time point i and the time point j
Obtain the first time out point.
Optionally, the video flowing acquisition module 601, is specifically used for,
By in the video flowing for having cached and not yet having played, the video flowing of the first scheduled duration of reproduction time the latest obtains
It is taken as the prediction video flowing.
Optionally, the video flowing acquisition module 601, is specifically used for,
Obtain the current play time point of the video and the latest time point for having cached and not yet having played in the video
Between the first duration；
When first duration is between first scheduled duration and the second scheduled duration, according to default acquisition
Cycle performs described the step of obtaining prediction video flowing, and second scheduled duration is more than first scheduled duration.
The application also provides a kind of computer-readable medium, is stored thereon with programmed instruction, and programmed instruction is held by processor
The video broadcasting method that above-mentioned each embodiment of the method provides is realized during row.
Present invention also provides it is a kind of comprising instruction computer program product, when run on a computer so that
Computer performs the video broadcasting method that above-mentioned each embodiment of the method provides.
With reference to figure 7, the block diagram of the terminal provided it illustrates one exemplary embodiment of the application.In the application
Terminal can include it is one or more such as lower components：Processor 710 and memory 720.
Processor 710 can include one or more processing core.Processor 710 utilizes various interfaces and connection
Various pieces in whole terminal, by running or performing the instruction being stored in memory 720, program, code set or instruction
Collection, and the data being stored in memory 720 are called, perform the various functions and processing data of terminal.Alternatively, processor
710 can use Digital Signal Processing (Digital Signal Processing, DSP), field programmable gate array
(Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic
Array, PLA) at least one of example, in hardware realize.Processor 710 can integrating central processor (Central
Processing Unit, CPU) and modem etc. in one or more of combinations.Wherein, CPU mainly handles operation system
System and application program etc.；Modem is used to handle wireless communication.It is understood that above-mentioned modem can not also
It is integrated into processor 710, is realized separately through chip piece.
Alternatively, above-mentioned each embodiment of the method carries under being realized when processor 710 performs the programmed instruction in memory 720
The video broadcasting method of confession.
Memory 720 can include random access memory (Random Access Memory, RAM), can also include read-only
Memory (Read-Only Memory).Alternatively, which includes non-transient computer-readable medium (non-
transitory computer-readable storage medium).Memory 720 can be used for store instruction, program, generation
Code, code set or instruction set.Memory 720 may include storing program area and storage data field, wherein, storing program area can store
It is used for realization the instruction, the instruction at least one function, the finger for being used for realization above-mentioned each embodiment of the method for operating system
Order etc.；Storage data field can be stored uses created data etc. according to terminal.
One of ordinary skill in the art will appreciate that hardware can be passed through by realizing all or part of step of above-described embodiment
To complete, relevant hardware can also be instructed to complete by program, the program can be stored in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only storage, disk or CD etc..
The foregoing is merely the preferred embodiment of the application, not to limit the application, it is all in spirit herein and
Within principle, any modification, equivalent replacement, improvement and so on, should be included within the protection domain of the application.
Claims (10)
- A kind of 1. video broadcasting method, it is characterised in that the described method includes：Prediction video flowing is obtained, the prediction video flowing is the video flowing for having cached in currently playing video and not yet having played；By the prediction video flowing input prediction model, the first time out point predicted, the first time out point It it is the time point in the corresponding reproduction time section of the prediction video flowing, the prediction model is according to Sample video stream and sample What time point trained；When the video playing is to the first time out point, pause plays the video.
- 2. according to the method described in claim 1, it is characterized in that, it is described by it is described prediction video flowing input prediction model, obtain To the first time out point of prediction, including：Down-sampled processing, the video flowing after acquisition is down-sampled are carried out to the prediction video flowing；By it is described it is down-sampled after video flowing input the prediction model, obtain the first time out point.
- 3. according to the method described in claim 2, it is characterized in that, described carry out down-sampled processing to the prediction video flowing, Obtain it is down-sampled after video flowing, including：Drop frame sampling is carried out to the prediction video flowing, the first video flowing is obtained, is included in first video flowing according to default Sample rate includes picture frame and audio frame from each video frame predicted and collected in video flowing, each video frame；Resolution decreasing sampling is carried out to the picture frame in each video frame, obtains the picture frame after resolution decreasing；Drop precision sampling is carried out to the audio frame in each video frame, obtains the audio frame after drop precision；By the picture frame after the resolution decreasing and it is described drop precision after audio frame be combined into it is described it is down-sampled after video Stream.
- 4. method according to any one of claims 1 to 3, it is characterised in that the prediction model includes the first prediction model It is described by the prediction video flowing input prediction model with the second prediction model, the first time out point predicted, bag Include：Each picture frame in the prediction video flowing is inputted into first prediction model, obtains the first time point set of prediction Close, at least one time point is included in the first time point set；Each audio frame in the prediction video flowing is inputted into second prediction model, obtains the second time point set of prediction Close, at least one time point is included in the second time point set；The first time out point is obtained according to the first time point set and the second time point set.
- It is 5. according to the method described in claim 4, it is characterized in that, described according to the first time point set and described second Time point set obtains the first time out point, including：For time point i, determine to whether there is time point j in the second time point set, the time point i is described first Random time point in time point set, it is poor that the time difference between the time point j and the time point i is less than preset time；When there are during the time point j, being obtained in the second time point set according to the time point i and time point j The first time out point.
- 6. method according to any one of claims 1 to 3, it is characterised in that described obtain predicts video flowing, including：By in the video flowing for having cached and not yet having played, the video flowing of the first scheduled duration of reproduction time the latest is retrieved as The prediction video flowing.
- 7. according to the method described in claim 6, it is characterized in that, it is described obtain prediction video flowing, including：Obtain the video current play time point and the video in cached and between the latest time point that not yet plays The first duration；When first duration is between first scheduled duration and the second scheduled duration, according to the default acquisition cycle Described the step of obtaining prediction video flowing is performed, second scheduled duration is more than first scheduled duration.
- 8. a kind of video play device, it is characterised in that described device includes：Video flowing acquisition module, video flowing is predicted for obtaining, and the prediction video flowing has been cached in currently playing video And the video flowing not yet played；Prediction module, for by the prediction video flowing input prediction model, the first time out point predicted, described the Time out point is the time point in the corresponding reproduction time section of the prediction video flowing, and the prediction model is according to sample What video flowing and sample time point were trained；Suspend module, for when the video playing is to the first time out point, pause to play the video.
- 9. a kind of terminal, it is characterised in that the memory that the terminal includes processor, is connected with the processor, Yi Jicun The programmed instruction on the memory is stored up, the processor is realized when performing described program instruction as claim 1 to 7 is any The video broadcasting method.
- 10. a kind of computer-readable recording medium, it is characterised in that be stored thereon with programmed instruction, described program instruction is located Manage the video broadcasting method realized when device performs as described in claim 1 to 7 is any.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711469028.XA CN107920275B (en) | 2017-12-29 | 2017-12-29 | Video playing method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711469028.XA CN107920275B (en) | 2017-12-29 | 2017-12-29 | Video playing method, device, terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107920275A true CN107920275A (en) | 2018-04-17 |
| CN107920275B CN107920275B (en) | 2020-06-09 |
Family
ID=61894453
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711469028.XA Expired - Fee Related CN107920275B (en) | 2017-12-29 | 2017-12-29 | Video playing method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107920275B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111064788A (en) * | 2019-12-18 | 2020-04-24 | 深圳前海达闼云端智能科技有限公司 | Signal transmission method, robot, and computer-readable storage medium |
| CN111327956A (en) * | 2020-02-13 | 2020-06-23 | 杭州海康威视系统技术有限公司 | Video playing method and device and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103905897A (en) * | 2014-04-23 | 2014-07-02 | 无锡天脉聚源传媒科技有限公司 | Video playing device and method thereof |
| JP5643242B2 (en) * | 2012-02-15 | 2014-12-17 | 日本電信電話株式会社 | Media player parameter estimation apparatus, method, and program |
| US8930577B2 (en) * | 2011-09-13 | 2015-01-06 | Microsoft Corporation | Synchronizing streaming video between viewers over a network |
| CN104918060A (en) * | 2015-05-29 | 2015-09-16 | 北京奇艺世纪科技有限公司 | Method and device for selecting position to insert point in video advertisement |
| CN105979345A (en) * | 2015-12-03 | 2016-09-28 | 乐视致新电子科技（天津）有限公司 | Video data processing method and device |
| CN106028088A (en) * | 2016-06-16 | 2016-10-12 | 腾讯科技（深圳）有限公司 | Insertion method and device of media data |
-
2017
- 2017-12-29 CN CN201711469028.XA patent/CN107920275B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8930577B2 (en) * | 2011-09-13 | 2015-01-06 | Microsoft Corporation | Synchronizing streaming video between viewers over a network |
| JP5643242B2 (en) * | 2012-02-15 | 2014-12-17 | 日本電信電話株式会社 | Media player parameter estimation apparatus, method, and program |
| CN103905897A (en) * | 2014-04-23 | 2014-07-02 | 无锡天脉聚源传媒科技有限公司 | Video playing device and method thereof |
| CN104918060A (en) * | 2015-05-29 | 2015-09-16 | 北京奇艺世纪科技有限公司 | Method and device for selecting position to insert point in video advertisement |
| CN105979345A (en) * | 2015-12-03 | 2016-09-28 | 乐视致新电子科技（天津）有限公司 | Video data processing method and device |
| CN106028088A (en) * | 2016-06-16 | 2016-10-12 | 腾讯科技（深圳）有限公司 | Insertion method and device of media data |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111064788A (en) * | 2019-12-18 | 2020-04-24 | 深圳前海达闼云端智能科技有限公司 | Signal transmission method, robot, and computer-readable storage medium |
| CN111327956A (en) * | 2020-02-13 | 2020-06-23 | 杭州海康威视系统技术有限公司 | Video playing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107920275B (en) | 2020-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10733452B2 (en) | Brand safety in video content | |
| CN109145784B (en) | Method and apparatus for processing video | |
| CN108712661B (en) | Live video processing method, device, equipment and storage medium | |
| EP3791338B1 (en) | Generating a customized machine-learning model to perform tasks using artificial intelligence | |
| JP7479750B2 (en) | Virtual video live broadcast processing method and device, electronic device | |
| CN108012162A (en) | Content recommendation method and device | |
| CN110119757A (en) | Model training method, video category detection method, device, electronic equipment and computer-readable medium | |
| WO2019242222A1 (en) | Method and device for use in generating information | |
| US20220132222A1 (en) | Prediction model training via live stream concept association | |
| WO2018217501A1 (en) | Using artificial intelligence and machine learning to automatically share desired digital media | |
| US11727265B2 (en) | Methods and apparatus to provide machine programmed creative support to a user | |
| US20180075877A1 (en) | Speaker segmentation and clustering for video summarization | |
| CN113486833B (en) | Multi-modal feature extraction model training method and device and electronic equipment | |
| CN107111620A (en) | Video editing using context data and the content discovery using group | |
| US20230206420A1 (en) | Method for detecting defect and method for training model | |
| JP2017112609A (en) | Video conversation method and system using bidirectional transmission of visual or auditory effect | |
| WO2022028304A1 (en) | Multimedia data processing method and apparatus, device and readable storage medium | |
| CA3028328A1 (en) | Method and system for automatically producing video highlights | |
| CN109448709A (en) | A kind of terminal throws the control method and terminal of screen | |
| WO2023116233A1 (en) | Video stutter prediction method and apparatus, device and medium | |
| WO2021019342A1 (en) | Synchronized sound generation from videos | |
| US20210117409A1 (en) | Automated content generation from statistical data | |
| CN106535003A (en) | Bullet screen information processing method, device and system | |
| CN107920275A (en) | Video broadcasting method, device, terminal and storage medium | |
| CN108289201A (en) | Video data handling procedure, device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant after: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. Address before: Changan town in Guangdong province Dongguan 523860 usha Beach Road No. 18 Applicant before: GUANGDONG OPPO MOBILE TELECOMMUNICATIONS Corp.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200609 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |