CN110177308A - Mobile terminal and its audio-video frame losing method in record screen, computer storage medium - Google Patents
Mobile terminal and its audio-video frame losing method in record screen, computer storage medium Download PDFInfo
- Publication number
- CN110177308A CN110177308A CN201910299887.1A CN201910299887A CN110177308A CN 110177308 A CN110177308 A CN 110177308A CN 201910299887 A CN201910299887 A CN 201910299887A CN 110177308 A CN110177308 A CN 110177308A
- Authority
- CN
- China
- Prior art keywords
- audio
- frame
- video
- video frame
- terminal capabilities
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/4424—Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless
- H04N21/64784—Data processing by the network
- H04N21/64792—Controlling the complexity of the content stream, e.g. by dropping packets
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
This application discloses the audio-video frame losing method in a kind of mobile terminal and its record screen, computer storage medium, which includes: to obtain the audio frame and video frame that record screen generates, and obtain terminal capabilities at that time and/or network state；Judge whether terminal capabilities and/or network state meet audio video transmission condition；If it is not, then abandoning audio frame and/or video frame according to preset rules based on terminal capabilities and/or network state.By above-mentioned audio-video frame losing method, partial video frame can be abandoned under and/or the insufficient situation of network state insufficient in terminal device performance, so that subsequent audio/video frames can be sent in time.
Description
Technical field
This application involves technical field of video processing, lose more particularly to the audio-video in a kind of mobile terminal and its record screen
Frame method, computer storage medium.
Background technique
IOS (original name iPhone OS, be renamed as IOS from after IOS 4) is the proprietary movement developed by some mobile devices
Operating system, the equipment supported include iPhone, iPod touch and iPad.
Replaykit is the frame that IOS is broadcast live release to screen, and the effect of ReplayKit frame is from screen recording
Or stream transmission video, and carry out the audio of self-application and microphone.Using ReplayKit frame, user can record from screen
Video processed, from application program and microphone records audio.
When returning to audio, video data due to Replaykit it is possible that terminal device performance is insufficient or network bandwidth not
Situations such as sufficient, causes the audio, video data in memory to be unable to get and timely sends.
Summary of the invention
To solve the above problems, this application provides the audio-videos in a kind of mobile terminal and its record screen to mend frame method, meter
Calculation machine storage medium can abandon part audio-video when or network bandwidth deficiency insufficient in terminal device performance, to guarantee
Audio, video data in memory can timely be sent.
The technical solution that the application uses is: providing a kind of audio-video frame losing method recorded in screen, the video is lost
Frame method includes: to obtain the audio frame and video frame that record screen generates, and obtain terminal capabilities at that time and/or network state；Sentence
Break the terminal capabilities and/or whether the network state meets audio video transmission condition；If it is not, being then based on the terminal capabilities
And/or the network state abandons the audio frame and/or the video frame according to preset rules.
Another technical solution that the application uses is: providing a kind of mobile terminal, the mobile terminal includes: acquisition mould
Block for obtaining the audio frame and video frame that record screen generates, and obtains terminal capabilities at that time and/or network state；Handle mould
Block, for judging whether terminal capabilities and/or network state meet audio video transmission condition；Processing module is also used to described
When terminal capabilities and/or the network state are unsatisfactory for the audio video transmission condition, based on the terminal capabilities and/or described
Network state abandons audio frame and/or video frame according to preset rules.
Another technical solution that the application uses is: a kind of mobile terminal is provided, the mobile terminal include processor,
Memory and display screen；Wherein, the display screen is described for storing program data for showing picture, the memory
Processor is for executing described program data to realize such as above-mentioned audio-video frame losing method.
Another technical solution that the application uses is: providing a kind of computer storage medium, the computer storage is situated between
Program data is stored in matter, described program data are when being executed by processor, for realizing such as above-mentioned audio-video frame losing side
Method.
Audio-video frame losing method in a kind of record screen provided by the present application includes: the audio that acquisition for mobile terminal record screen generates
Frame and video frame, and obtain terminal capabilities at that time and/or network state；Judge whether terminal capabilities and/or network state are full
Footsteps transmission of video condition；If it is not, then based on terminal capabilities and/or network state according to preset rules abandon audio frame and/or
Video frame；When terminal capabilities and/or network state are unable to satisfy real-time full and receive all audio-videos, mobile terminal is according to reality
When terminal capabilities and/or network state take different frame losing logics to abandon part audio/video frames so that subsequent sound regard
Frequency frame can be sent in time.
Detailed description of the invention
In order to more clearly explain the technical solutions in the embodiments of the present application, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, the drawings in the following description are only some examples of the present application, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.Wherein:
Fig. 1 is the flow diagram of the audio-video frame losing method first embodiment in record screen provided by the present application；
Fig. 2 is the flow diagram of the audio-video frame losing method second embodiment in record screen provided by the present application；
Fig. 3 is the flow diagram of the audio-video frame losing method 3rd embodiment in record screen provided by the present application；
Fig. 4 is the flow diagram of the audio-video frame losing method fourth embodiment in record screen provided by the present application；
Fig. 5 is the structural schematic diagram of one embodiment of mobile terminal provided by the present application；
Fig. 6 is the structural schematic diagram of another embodiment of mobile terminal provided by the present application；
Fig. 7 is the structural schematic diagram of one embodiment of computer storage medium provided by the present application.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description.It is understood that specific embodiment described herein is only used for explaining the application, rather than to the limit of the application
It is fixed.It also should be noted that illustrating only part relevant to the application for ease of description, in attached drawing and not all knot
Structure.Based on the embodiment in the application, obtained by those of ordinary skill in the art without making creative efforts
Every other embodiment, shall fall in the protection scope of this application.
Term " first ", " second " in the application etc. be for distinguishing different objects, rather than it is specific suitable for describing
Sequence.In addition, term " includes " and " having " and their any deformations, it is intended that cover and non-exclusive include.Such as comprising
The process, method, system, product or equipment of a series of steps or units are not limited to listed step or unit, and
It is optionally further comprising the step of not listing or unit, or optionally further comprising for these process, methods, product or equipment
Intrinsic other step or units.
Referenced herein " embodiment " is it is meant that a particular feature, structure, or characteristic described can wrap in conjunction with the embodiments
It is contained at least one embodiment of the application.Each position in the description occur the phrase might not each mean it is identical
Embodiment, nor the independent or alternative embodiment with other embodiments mutual exclusion.Those skilled in the art explicitly and
Implicitly understand, embodiment described herein can be combined with other embodiments.
Refering to fig. 1, Fig. 1 is the process signal of the audio-video frame losing method first embodiment in record screen provided by the present application
Figure.The audio-video frame losing method of Fig. 1 be applied to a kind of mobile terminal with audio/video frames transmission-receiving function and processing function or
Electronic equipment, wherein mobile terminal or electronic equipment can carry IOS operating system.
The audio-video frame losing method of the present embodiment includes:
Step 101: obtaining the audio frame and video frame that record screen generates, and obtain terminal capabilities at that time and/or network-like
State.
In IOS system, it mainly includes audio frame and video frame that the data of ReplayKit development kit, which return,.
Wherein, audio frame includes RPSampleBufferTypeAudioApp, i.e. App (application, application program)
Internal audio frequency data content, hereinafter referred to as App audio and RPSampleBufferTypeAudioApp, i.e. microphone audio number
According to content, hereinafter referred to as Mic audio.
Wherein, video frame, i.e. RPSampleBufferTypeVideo are naked video requency frame data content.
It should be understood that ReplayKit frame is a frame for carrying out record screen, and during recording screen, a side
There are the audio from App, such as video App, game App, music App in face, on the other hand comes from the received audio of microphone again,
As user speaks.
Optionally, App audio thread safe list, Mic audio thread safe list and video are respectively created when initialization
Thread-safe list.When receiving ReplayKit audio frame, thread-safe list is inserted into respective thread, so that
Replaykit can be rapidly returned to, and accomplish the thread for not blocking ReplayKit as much as possible.
It is understood that the thread of ReplayKit is served only for reading and writing using audio, microphone audio and view
Frequency frame.
It should be understood that if there is congestion situations, then will lead to when App audio, Mic audio and video frame are read
Audio is paused and jump problem, and video frame is discontinuously with picture come the problem of knock-oning, so it should be ensured that not blocking ReplayKit's
Thread.
Optionally, in one embodiment, when receiving App audio, it is inserted into App audio thread safe list；It is connecing
When receiving Mic audio, it is inserted into Mic audio thread safe list；When receiving video frame, it is inserted into video thread security column
Table.
Specifically, in initialization, creation video thread safe list is connecing in the video frame for receiving ReplayKit
Video frame is inserted into video thread safe list in take-up journey, Replaykit is rapidly returned to, is accomplished as much as possible not
Block the thread of ReplayKit.
It will be further appreciated that being application audio, microphone audio and the video frame for getting record screen and generating, record is shielded
The application audio and microphone audio of generation give independent first thread, give independent second for the video frame that record screen generates
Thread.
While obtaining above-mentioned audio frame and video frame, mobile terminal can also detect the terminal device performance of itself with
And the data such as network state in data transmission are obtained, to judge whether the transmission process of audio frame and video frame is normal, with true
Can the audio frame and video frame that determined in memory timely be sent.
Step 102: judging whether terminal capabilities and/or network state meet audio video transmission condition.
Wherein, terminal capabilities can be evaluated by a terminal capabilities index or multiple terminal capabilities indicator combinations.For example, eventually
Holding performance indicator may include CPU processing speed, the memory speed of service, video card capabilities etc..Similarly, network state can also be by
One network performance index or multiple network performance index combination evaluations.For example, network performance index may include flow trend,
Network speed size, network operator, network type etc..
Audio video transmission condition can be set to some Evaluation threshold of terminal capabilities or some evaluation of network state
Some overall merit threshold value of threshold value or terminal capabilities and network state.
For example, audio video transmission condition can be set to some threshold speed of terminal CPU processing, as the real-time CPU of terminal
Processing speed is less than the threshold speed, then illustrates that the terminal capabilities of mobile terminal is unable to satisfy audio video transmission condition, i.e., can not
Accomplish to send the audio frame and video frame in memory in time.
Alternatively, audio video transmission condition can be set to some network speed threshold value, when terminal sends audio frame and video
While frame, detects and obtain real-time network velocity magnitude.If real-time network velocity magnitude is less than the network speed threshold value, say
Bright network environment at that time is unable to satisfy audio video transmission condition, i.e., can not accomplish to send the audio frame and video in memory in time
Frame.
Alternatively, audio video transmission condition may be arranged as the weighted value of CPU processing speed threshold value and network speed threshold value,
Wherein, weighted value can be set to 50%CPU processing speed threshold value and 50% network speed threshold value.In other embodiments, add
The distribution of weight can also be adjusted based on the considerations of technical staff, and details are not described herein.
When the terminal capabilities and/or network state that obtain in real time meet audio video transmission condition, mobile terminal continue into
The recording of row audio-video；When the terminal capabilities and/or network state that obtain in real time are unable to satisfy audio video transmission condition, then into
Enter step 103.
Step 103: audio frame and/or video frame are abandoned according to preset rules based on terminal capabilities and/or network state.
Wherein, the case where audio video transmission condition being unable to satisfy for terminal capabilities and network state, mobile terminal according to
Different frame losing logics abandons part audio frame and/or video frame.
Specifically, when being unable to satisfy audio video transmission condition due to terminal capabilities, mobile terminal can use the first frame losing
Logic abandons part audio/video frames；When being unable to satisfy audio video transmission condition due to network state, mobile terminal can be using the
Two frame losing logics abandon some or all audio/video frames.
In the insufficient or insufficient situation of network state due to terminal capabilities, mobile terminal can be patrolled using different frame losings
Volume, identical frame losing logic can also be used.Further, under identical terminal capabilities or network state, for audio frame
And video frame, mobile terminal can also use different frame losing logics, details are not described herein.
In the present embodiment, mobile terminal also obtains at that time while obtaining the audio frame and video frame of recording screen generation
Terminal capabilities and/or network state；It is mobile when terminal capabilities and/or network state are unable to satisfy audio video transmission condition
Terminal based on terminal capabilities and/or network state using identical or different frame losing logic abandon some or all audio frames and/
Or video frame；By abandoning some or all audio frames and/or video frame, abandon the audio frame generated in part record screen and/or
Video frame, so that the audio frame and/or video frame in memory can timely be sent；The device resource for occupying calling program
Control in the reasonable scope, is maintained in controlled range but also audio-video delay is broadcast live.
For the step 102 in embodiment illustrated in fig. 1, the application further provides another specific method.Specifically
Referring to Fig. 2, Fig. 2 is the flow diagram of the audio-video frame losing method second embodiment in record screen provided by the present application.
Specifically in above-mentioned steps 102, terminal capabilities can be by CPU processing speed, the memory speed of service, video card capabilities etc.
Performance indicator evaluation.In the present embodiment, terminal capabilities can also be evaluated by CPU usage and/or EMS memory occupation size.Its
In, the cpu resource that CPU usage, that is, running of mobile terminal program occupies indicates the operation program of terminal at some time point
The case where.CPU usage is higher, illustrates that terminal has run many programs in synchronization, will lead to the process of audio video transmission,
Therefore, CPU usage is also an important indicator for evaluating terminal capabilities.
In order to improve the evaluation accuracy of terminal capabilities, the audio-video frame losing method of the present embodiment is specifically proposed with lower section
Method:
Step 201: judging whether EMS memory occupation size is greater than EMS memory occupation threshold value.
Wherein, while obtaining above-mentioned audio frame and video frame, acquisition for mobile terminal self EMS memory occupies size, that is, deposits
The space size of storage audio frame and video frame is had been used in storage space.
Further, the preset EMS memory occupation threshold value of acquisition for mobile terminal, and EMS memory occupation threshold value and EMS memory occupation is big
It is small to be compared, to judge whether EMS memory occupation size is greater than EMS memory occupation threshold value；When EMS memory occupation size is greater than EMS memory occupation
When threshold value, then illustrates that the state of terminal is unsatisfactory for audio video transmission condition, enter step 202.
Specifically, in default EMS memory occupation threshold value, EMS memory occupation threshold value can be lower than 50M.Since ReplayKit belongs to
The Extension module of IOS system, it is maximum allowable in save as 50M, the memory that 500 milliseconds of naked video data needs is much super
50M is crossed, therefore, EMS memory occupation threshold value should be lower than 50M, in case the memory for touching the Extension module of IOS system is online.
In other embodiments, mobile terminal also itself available CPU usage, the i.e. program of running of mobile terminal
The cpu resource of occupancy.
Further, the preset CPU usage threshold value of acquisition for mobile terminal, and by CPU usage threshold value and CPU usage
It is compared, to judge whether CPU usage is greater than CPU usage threshold value；When CPU usage is greater than CPU usage threshold value,
Then illustrate that the state of terminal is unsatisfactory for audio video transmission condition, enters step 202.
Specifically, CPU usage threshold value can also be defined by technical staff based on concrete condition.For example,
It is more than 40% to be set as threshold value of warning that Extension module or the application program of record screen, which occupy CPU, is urgent threshold more than 60%
Value is threshold limit value etc. more than 80.
Step 202: the interframe space based on video frame abandons the video frame of default span, wherein video frame is before encoding
Naked video frame.
Wherein, the present embodiment proposes a kind of frame losing logic, can also use other frame losing logics in other embodiments.
Specifically, mobile terminal abandons the video frame of default span according to the interframe space of video frame, for example, whenever movement
When terminal receives 3 frame video frame, mobile terminal abandons 1 frame video frame therein.Wherein, presetting span can also accordingly adjust
It is whole, for example, mobile terminal can abandon 1 frame video frame therein often to receive 5 frame video frames.
Wherein, the video frame abandoned in step 202 is the naked video frame before coding, does not need to abandon the video after coding
Frame.
For audio frame, mobile terminal can also use the frame losing logic of step 202.Wherein, above-mentioned threshold value setting
It can be consistent with video frame losing mode when performance deficiency with frame losing algorithm, the threshold value of different numerical value also can be set.
But unlike video frame, mobile terminal cannot directly abandon original audio frame, need to abandon App audio thread-safe
Audio frame in list, Mic audio thread safe list.
In the present embodiment, mobile terminal device performance deficiency is divided into cpu performance deficiency and EMS memory occupation is excessively high caused
Free memory is insufficient；Mobile terminal can abandon the part generated in record screen using identical frame losing logic based on both of these case
Audio frame and/or video frame, so that the audio frame and/or video frame in memory can timely be sent.
For the step 102 in embodiment illustrated in fig. 1, the application further provides another specific method.Specifically
Referring to Fig. 3, Fig. 3 is the flow diagram of the audio-video frame losing method 3rd embodiment in record screen provided by the present application.
Specifically in above-mentioned steps 102, network state can be by flow trend, network speed size, network operator, net
The evaluation of the performance indicators such as network type.In the present embodiment, network state can also be evaluated by network bandwidth size.Wherein, bandwidth
It is called bandwidth, refers to the data that can be transmitted at a fixed time, i.e., can transmit the ability of data in transmission pipeline.Cause
This, network bandwidth size is also to evaluate an important indicator of network state.
In order to improve the evaluation accuracy of network state, the audio-video frame losing method of the present embodiment is specifically proposed with lower section
Method:
Step 301: judging whether network bandwidth size is less than first network bandwidth threshold.
Wherein, while obtaining above-mentioned audio frame and video frame, acquisition for mobile terminal real-time network amount of bandwidth.
Further, the preset first network bandwidth threshold of acquisition for mobile terminal, and by first network bandwidth threshold and net
Network amount of bandwidth is compared, to judge whether network bandwidth size is less than first network bandwidth threshold；When network bandwidth size
When less than first network bandwidth threshold, then illustrates that the state of network is unsatisfactory for audio video transmission condition, enter step 302.
Step 302: judging whether network bandwidth size is greater than the second network bandwidth threshold value.
Wherein, mobile terminal further obtains preset second network bandwidth threshold value, and by the second network bandwidth threshold value with
Network bandwidth size is compared, to judge whether network bandwidth size is greater than the second network bandwidth threshold value.Wherein, the second network
Bandwidth threshold is less than first network bandwidth threshold.
If mobile terminal judges that network bandwidth size is greater than the second network bandwidth threshold value, and is less than first network bandwidth threshold
Value then illustrates that the state of the network of mobile terminal connection is in alert status, enters step 303；If mobile terminal judges network
Amount of bandwidth then illustrates that the state of the network of mobile terminal connection is in a state of emergency, enters less than the second network bandwidth threshold value
Step 304.
Step 303: abandoning video non-key frame.
Wherein, naked video frame is inserted into video thread safe list after coding, and mobile terminal is based on every frame video frame
Video frame after coding is divided into video non-key frame and key frame of video by content.At the state of the network of mobile terminal connection
When alert status, mobile terminal disposably abandons all video non-key frames in video thread safe list.
Step 304: abandoning key frame of video and video non-key frame.
Wherein, when the state of the network of mobile terminal connection is in a state of emergency, mobile terminal disposably abandons video
All video non-key frames and key frame of video in thread-safe list.
In the present embodiment, after the state of network of the mobile terminal based on connection abandons coding using different frame losing logics
Video non-key frame and/or key frame of video so that the video frame in memory can timely be sent.
After the step 103 in embodiment illustrated in fig. 1, the application further provides another specific method.
It is the flow diagram of the audio-video frame losing method fourth embodiment in record screen provided by the present application referring specifically to Fig. 4, Fig. 4.
Specifically in above-mentioned steps 103, mobile terminal is based on terminal capabilities and/or network state abandon part audio frame and/
Or video frame.Since in record screen technology, treated audio frame and video frame need to carry out audio-visual synchronization, at this time, it may be necessary to
Timestamp is calculated to audio frame and video frame, it is final to carry out audio-visual synchronization processing.
In order to handle frame losing treated audio frame and video frame synchronization, the audio-video frame losing method of the present embodiment is specific
Propose following methods:
Step 401: setting audio counter.
Wherein, audio counter is arranged in mobile terminal, and audio counter is initialized as 0, abandons for calculating mobile terminal
Audio frame quantity.
Step 402: when abandoning each frame audio frame, audio counter adds 1.
Wherein, when mobile terminal abandons each frame audio frame, audio counter cumulative 1.
Step 403: setting video count device.
Wherein, video count device is arranged in mobile terminal, and audio counter is initialized as 0, abandons for calculating mobile terminal
Video frame quantity.
Step 404: when abandoning each frame video frame, video count device adds 1, to keep audio frame and video frame synchronization.
Wherein, when mobile terminal abandons each frame audio frame, audio counter cumulative 1.
When mobile terminal synchronizes processing to audio frame and video frame, numerical value of the mobile terminal based on audio counter
Timestamp is calculated with the numerical value of video count device, to keep audio-visual synchronization.
In the present embodiment, mobile terminal counts the audio frame of discarding by setting audio counter and video count device
And video frame, and the numerical value of the numerical value based on audio counter and video count device calculates timestamp, to keep audio-video same
Step.
For the audio-video frame losing method for realizing above-described embodiment, present invention also provides a kind of mobile terminals, please specifically join
Fig. 5 is read, Fig. 5 is the structural schematic diagram of one embodiment of mobile terminal provided by the present application.
The mobile terminal 50 includes obtaining module 51 and processing module 52.
Wherein, it obtains module 51 and is used to obtain the audio frame and video frame that record screen generates, and obtain terminal capabilities at that time
And/or network state.
Processing module 52 is for judging whether terminal capabilities and/or network state meet audio video transmission condition；It is also used to
When the terminal capabilities and/or the network state are unsatisfactory for the audio video transmission condition, based on the terminal capabilities and/
Or the network state abandons audio frame and/or video frame according to preset rules.
For the audio-video frame losing method for realizing above-described embodiment, present invention also provides another mobile terminals, specifically ask
It is the structural schematic diagram of another embodiment of mobile terminal provided by the present application refering to Fig. 6, Fig. 6.
The mobile terminal 60 includes processor 61, memory 62 and display screen 63.
The mobile terminal 60 can be the terminal using IOS as operating system, such as mobile phone, tablet computer, intelligent wearable device
Etc., wherein the display screen 63 can be liquid crystal display (LCD), be also possible to organic light-emitting diode (OLED) display screen (OLED),
It is used to show video pictures, such as the operation interface of App or the operation interface of system in mobile terminal 60.
Program data is stored in memory 62, processor 61 is for executing the program data to realize above-described embodiment
Audio-video frame losing method.
It is the structural schematic diagram of one embodiment of computer storage medium provided by the present application please continue to refer to Fig. 7, Fig. 7, it should
Program data 71 is stored in computer storage medium 70, the program data 71 is above-mentioned to realize when being executed by processor
The audio-video frame losing method of embodiment.
Embodiments herein is realized in the form of SFU software functional unit and when sold or used as an independent product, can
To be stored in a computer readable storage medium.Based on this understanding, the technical solution of the application substantially or
Say that all or part of the part that contributes to existing technology or the technical solution can embody in the form of software products
Out, which is stored in a storage medium, including some instructions are used so that a computer equipment
(can be personal computer, server or the network equipment etc.) or processor (processor) execute each implementation of the application
The all or part of the steps of mode the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory
(ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk
Etc. the various media that can store program code.
The foregoing is merely presently filed embodiments, are not intended to limit the scope of the patents of the application, all to utilize this
Equivalent structure or equivalent flow shift made by application specification and accompanying drawing content, it is relevant to be applied directly or indirectly in other
Technical field similarly includes in the scope of patent protection of the application.
Claims (10)
1. the audio-video frame losing method in a kind of record screen, which is characterized in that the audio-video frame losing method includes:
The audio frame and video frame that record screen generates are obtained, and obtains terminal capabilities at that time and/or network state；
Judge whether the terminal capabilities and/or the network state meet audio video transmission condition；
If it is not, then abandoning the audio frame and/or institute according to preset rules based on the terminal capabilities and/or the network state
State video frame.
2. audio-video frame losing method according to claim 1, which is characterized in that the terminal capabilities includes CPU usage；
It is described that the step of whether terminal capabilities and/or network state meet audio video transmission condition judged, comprising:
Judge whether the CPU usage is greater than CPU usage threshold value；
If so, the interframe space based on the video frame abandons the video frame of default span, wherein the video frame is coding
Preceding naked video frame.
3. audio-video frame losing method according to claim 1, which is characterized in that the terminal capabilities includes that EMS memory occupation is big
It is small；
It is described that the step of whether terminal capabilities and/or network state meet audio video transmission condition judged, comprising:
Judge whether the EMS memory occupation size is greater than EMS memory occupation threshold value；
If so, the interframe space based on the video frame abandons the video frame of default span, wherein the video frame is coding
Preceding naked video frame.
4. audio-video frame losing method according to claim 1, which is characterized in that the terminal capabilities includes CPU usage；
It is described that the step of whether terminal capabilities and/or network state meet audio video transmission condition judged, comprising:
Judge whether the CPU usage is greater than CPU usage threshold value；
If so, the interframe space based on the audio frame abandons the audio frame of default span, wherein the audio frame is coding
Audio frame afterwards.
5. audio-video frame losing method according to claim 1, which is characterized in that the network state includes that Netowrk tape is roomy
It is small；
It is described that the step of whether terminal capabilities and/or network state meet audio video transmission condition judged, comprising:
Judge whether the network bandwidth size is less than first network bandwidth threshold；
If so, abandoning the video frame after coding and/or the audio frame after coding according to preset rules.
6. audio-video frame losing method according to claim 5, which is characterized in that the video frame after coding includes video
Non-key frame and key frame of video；
It is described judge the step of whether the network bandwidth size is less than first network bandwidth threshold after, comprising:
If so, further judging whether the network bandwidth size is greater than the second network bandwidth threshold value, wherein first net
Network bandwidth threshold is greater than the second network bandwidth threshold value；
If so, abandoning the video non-key frame；
If it is not, then abandoning the key frame of video and the video non-key frame.
7. audio-video frame losing method as claimed in one of claims 1-6, which is characterized in that described to be based on the terminal capabilities
And/or the network state according to preset rules abandon audio frame and/or the step of video frame after, further includes:
Audio counter is set；
When abandoning audio frame described in each frame, the audio counter adds 1；
Video count device is set；
When abandoning video frame described in each frame, the video count device adds 1, to keep the audio frame and the video frame same
Step.
8. a kind of mobile terminal, which is characterized in that the mobile terminal includes:
Module is obtained, for obtaining the audio frame and video frame that record screen generates, and acquisition terminal capabilities at that time and/or network-like
State；
Processing module, for judging whether terminal capabilities and/or network state meet audio video transmission condition；
Processing module is also used to when the terminal capabilities and/or the network state are unsatisfactory for the audio video transmission condition,
Audio frame and/or video frame are abandoned according to preset rules based on the terminal capabilities and/or the network state.
9. a kind of mobile terminal, which is characterized in that the mobile terminal includes processor, memory and display screen；Wherein, institute
Display screen is stated for showing picture, the memory is for storing program data, and the processor is for executing described program number
Such as the described in any item audio-video frame losing methods of claim 1-7 are realized accordingly.
10. a kind of computer storage medium, which is characterized in that program data is stored in the computer storage medium, it is described
Program data is when being executed by processor, for realizing the described in any item audio-video frame losing methods of such as claim 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910299887.1A CN110177308A (en) | 2019-04-15 | 2019-04-15 | Mobile terminal and its audio-video frame losing method in record screen, computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910299887.1A CN110177308A (en) | 2019-04-15 | 2019-04-15 | Mobile terminal and its audio-video frame losing method in record screen, computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110177308A true CN110177308A (en) | 2019-08-27 |
Family
ID=67689468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910299887.1A Pending CN110177308A (en) | 2019-04-15 | 2019-04-15 | Mobile terminal and its audio-video frame losing method in record screen, computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110177308A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111107104A (en) * | 2019-12-31 | 2020-05-05 | 广州酷狗计算机科技有限公司 | Video transmitting method, video receiving method, device, equipment and storage medium |
| CN111475245A (en) * | 2020-04-08 | 2020-07-31 | 腾讯科技（深圳）有限公司 | Dynamic picture display method and device, electronic equipment and computer storage medium |
| CN111654761A (en) * | 2020-04-20 | 2020-09-11 | 威比网络科技(上海)有限公司 | Video processing method, system, device and medium based on online education |
| CN112383676A (en) * | 2020-11-03 | 2021-02-19 | 北京百度网讯科技有限公司 | Video file processing method and device, electronic equipment and storage medium |
| CN113038128A (en) * | 2021-01-25 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Data transmission method and device, electronic equipment and storage medium |
| CN115484494A (en) * | 2022-09-15 | 2022-12-16 | 云控智行科技有限公司 | Method, device and equipment for processing digital twin video stream |
| WO2022267733A1 (en) * | 2021-06-25 | 2022-12-29 | 荣耀终端有限公司 | Method for dynamically adjusting frame-dropping threshold value, and related devices |
| WO2023050921A1 (en) * | 2021-09-30 | 2023-04-06 | 中兴通讯股份有限公司 | Video and audio data sending method, display method, sending end and receiving end |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104620595A (en) * | 2012-10-11 | 2015-05-13 | 坦戈迈公司 | Proactive video frame dropping |
| CN105992023A (en) * | 2015-02-11 | 2016-10-05 | 杭州海康威视数字技术股份有限公司 | Video image data processing method and apparatus thereof |
| CN106993197A (en) * | 2017-04-10 | 2017-07-28 | 杭州星犀科技有限公司 | A kind of frame losing method based on encoder |
| US20180176636A1 (en) * | 2016-12-15 | 2018-06-21 | Guangzhou Ucweb Computer Technology Co., Ltd | Method, device, and electronic terminal for video recording |
| CN109089131A (en) * | 2018-09-21 | 2018-12-25 | 广州虎牙信息科技有限公司 | A kind of record screen live broadcasting method, device, equipment and storage medium based on IOS system |
-
2019
- 2019-04-15 CN CN201910299887.1A patent/CN110177308A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104620595A (en) * | 2012-10-11 | 2015-05-13 | 坦戈迈公司 | Proactive video frame dropping |
| CN105992023A (en) * | 2015-02-11 | 2016-10-05 | 杭州海康威视数字技术股份有限公司 | Video image data processing method and apparatus thereof |
| US20180176636A1 (en) * | 2016-12-15 | 2018-06-21 | Guangzhou Ucweb Computer Technology Co., Ltd | Method, device, and electronic terminal for video recording |
| CN106993197A (en) * | 2017-04-10 | 2017-07-28 | 杭州星犀科技有限公司 | A kind of frame losing method based on encoder |
| CN109089131A (en) * | 2018-09-21 | 2018-12-25 | 广州虎牙信息科技有限公司 | A kind of record screen live broadcasting method, device, equipment and storage medium based on IOS system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111107104A (en) * | 2019-12-31 | 2020-05-05 | 广州酷狗计算机科技有限公司 | Video transmitting method, video receiving method, device, equipment and storage medium |
| CN111107104B (en) * | 2019-12-31 | 2022-02-18 | 广州酷狗计算机科技有限公司 | Video transmitting method, video receiving method, device, equipment and storage medium |
| CN111475245A (en) * | 2020-04-08 | 2020-07-31 | 腾讯科技（深圳）有限公司 | Dynamic picture display method and device, electronic equipment and computer storage medium |
| CN111654761A (en) * | 2020-04-20 | 2020-09-11 | 威比网络科技(上海)有限公司 | Video processing method, system, device and medium based on online education |
| CN112383676A (en) * | 2020-11-03 | 2021-02-19 | 北京百度网讯科技有限公司 | Video file processing method and device, electronic equipment and storage medium |
| CN112383676B (en) * | 2020-11-03 | 2024-02-09 | 北京百度网讯科技有限公司 | Video file processing method and device, electronic equipment and storage medium |
| CN113038128A (en) * | 2021-01-25 | 2021-06-25 | 腾讯科技（深圳）有限公司 | Data transmission method and device, electronic equipment and storage medium |
| WO2022267733A1 (en) * | 2021-06-25 | 2022-12-29 | 荣耀终端有限公司 | Method for dynamically adjusting frame-dropping threshold value, and related devices |
| WO2023050921A1 (en) * | 2021-09-30 | 2023-04-06 | 中兴通讯股份有限公司 | Video and audio data sending method, display method, sending end and receiving end |
| CN115484494A (en) * | 2022-09-15 | 2022-12-16 | 云控智行科技有限公司 | Method, device and equipment for processing digital twin video stream |
| CN115484494B (en) * | 2022-09-15 | 2024-04-02 | 云控智行科技有限公司 | Digital twin video stream processing method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110177308A (en) | Mobile terminal and its audio-video frame losing method in record screen, computer storage medium | |
| CN104837052B (en) | Play the method and device of multi-medium data | |
| CN113542795B (en) | Video processing method and device, electronic equipment and computer readable storage medium | |
| US20140192136A1 (en) | Video chatting method and system | |
| CN110377434A (en) | A kind of service blowout method, device and message-oriented middleware | |
| CN109275012A (en) | A kind of live streaming barrage display methods and device | |
| CN114189700A (en) | Live broadcast card pause prompting method and device, computer equipment and storage medium | |
| CN112218115B (en) | Control method and device for streaming media audio and video synchronization and computer equipment | |
| Hong et al. | Optimizing cloud-based video crowdsensing | |
| CN112199174A (en) | Message sending control method and device, electronic equipment and computer readable storage medium | |
| CN115017002B (en) | Frequency prediction method and frequency prediction device | |
| CN114650437A (en) | Video publishing method, device, equipment and storage medium | |
| CN110012315A (en) | A kind of video broadcasting method, device and electronic equipment | |
| CN106162109B (en) | A kind of remote monitoring method and monitoring device | |
| CN110960857B (en) | Game data monitoring method and device, electronic equipment and storage medium | |
| CN112911390B (en) | Video data playing method and terminal equipment | |
| CN112711515A (en) | Real-time monitoring method and device and electronic equipment | |
| CN106482742A (en) | The acquisition methods of meter step data and device | |
| CN114630170B (en) | Audio and video synchronization method and device, electronic equipment and storage medium | |
| CN106921886A (en) | The multimedia data playing method and device of a kind of terminal | |
| CN110245120A (en) | The daily record data processing method of streaming computing system and streaming computing system | |
| CN102956208B (en) | Method, device and system for counting image frame rates of terminal | |
| CN115811483A (en) | Network state monitoring method and device, electronic equipment and storage medium | |
| US11343299B2 (en) | Method and system for real time measuring quality of video call service | |
| CN115883719B (en) | Notification processing method and notification processing device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190827 |