CN110213636A

CN110213636A - Video frame generating method, device, storage medium and the equipment of Online Video

Info

Publication number: CN110213636A
Application number: CN201810398279.1A
Authority: CN
Inventors: 向晨宇
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-04-28
Filing date: 2018-04-28
Publication date: 2019-09-06
Anticipated expiration: 2038-04-28
Also published as: CN110213636B

Abstract

The embodiment of the present application discloses video frame generating method, device, storage medium and the equipment of a kind of Online Video, belongs to field of computer technology.It include different types of sub-data flow in the video data stream the described method includes: receiving video data stream, every seed data stream has different traffic identifier；For every seed data stream, a mission thread and a task queue are created, and is added to the data segment for belonging to same video frame in the sub-data flow as a task in the task queue；For mission thread corresponding to the task queue there are task, the task is handled using the mission thread, obtains each processing result；Each processing result is rendered, video frame is obtained.Terminal capabilities both can be improved in the embodiment of the present application, can also be improved the fluency of picture broadcasting.

Description

Video frame generating method, device, storage medium and the equipment of Online Video

Technical field

The invention relates to field of computer technology, in particular to the video frame generating method of a kind of Online Video, Device, storage medium and equipment.

Background technique

Direct broadcasting room supports the online voice communication of more people and video communication.Each direct broadcasting room may include at least one main broadcaster User, after at least one spectators user enters direct broadcasting room, direct broadcasting room can also include at least one spectators user.In the live streaming phase Between, with online interaction, interaction mode can include between main broadcaster user and main broadcaster user, between main broadcaster user and spectators user but not It is limited to send barrage and gifts.At this point, live streaming client needs to clap barrage sub-data flow, present sub-data flow and main broadcaster These three different types of sub-data flows of the video sub-data flow taken the photograph generate a video and play out.Here live streaming client The live streaming client of live streaming client and/or spectators user including main broadcaster user.

In the related technology, live streaming client receives video data stream, includes different types of subnumber in the video data stream According to stream, and each sub-data flow has different traffic identifier；A mission thread is created, and for the creation pair of every seed data stream The data segment for belonging to same video frame in every seed data stream is added to the subdata as a task by the task queue answered It flows in corresponding task queue；Successively the task in each task queue is rendered using the mission thread, is obtained each Rendering result；Notice hardware renders each rendering result, obtains video frame.

For mission thread when handling the task in a task queue, task in other task queues can seize this The process resource of business thread in this way, the mission thread can interrupt the processing to current task, and goes to handle appointing in other queues Business, prevent the picture of current task plays not smooth from being fully displayed in video frame so as to cause picture.

Summary of the invention

The embodiment of the present application provides video frame generating method, device, storage medium and the equipment of a kind of Online Video, uses Task in solution different task queue seizes the process resource of same mission thread, and picture is caused to play slack ask Topic.The technical solution is as follows:

On the one hand, a kind of video frame generating method of Online Video is provided, which comprises

Video data stream is received, includes different types of sub-data flow in the video data stream, every seed data stream tool There is different traffic identifier；

For every seed data stream, a mission thread and a task queue are created, and will be belonged in the sub-data flow It is added in the task queue in the data segment of same video frame as a task；

For mission thread corresponding to the task queue there are task, the task is carried out using the mission thread Processing, obtains processing result；

Each processing result is rendered, video frame is obtained.

On the one hand, a kind of video frame generating apparatus of Online Video is provided, described device includes:

Receiving module includes different types of sub-data flow in the video data stream, often for receiving video data stream Seed data stream has different traffic identifier；

Creation module, every seed data stream for being received for the receiving module, create a mission thread and One task queue, and it is added to described appoint for the data segment for belonging to same video frame in the sub-data flow as a task It is engaged in queue；

Processing module, for utilizing the mission thread for mission thread corresponding to the task queue there are task The task is handled, processing result is obtained；

Generation module obtains video frame for rendering to each processing result that the processing module obtains.

On the one hand, a kind of presentation of video frames method of live video, for being broadcast live in client, the method packet are provided It includes:

Live video stream is obtained, the live video stream includes live streaming picture data stream and interactive data stream, the interaction Data flow include at least one of present data flow and barrage data flow, the live video stream, the present data flow and The barrage data flow has different traffic identifier；

For every kind of data flow, a mission thread and a task queue are created, and will be belonged in the data flow same The data segment of one live streaming video frame is added in the task queue as a task；

Each processing result is rendered to obtain live video frame, the live video frame includes live streaming picture and interaction Data；

Play the live video frame.

On the one hand, a kind of presentation of video frames device of live video, for being broadcast live in client, described device packet are provided It includes:

Module is obtained, for obtaining live video stream, the live video stream includes live streaming picture data stream and interaction number According to stream, the interactive data stream includes at least one of present data flow and barrage data flow, the live video stream, described Present data flow and the barrage data flow have different traffic identifier；

Creation module, every kind of data flow for obtaining for the acquisition module, creates a mission thread and one Task queue, and it is added to the task queue for the data segment for belonging to same video frame in the data flow as a task In；

Generation module, for being rendered to obtain live video frame to each processing result that the processing module obtains, The live video frame includes live streaming picture and interactive data；

Playing module, the live video frame generated for playing the generation module.

On the one hand, provide a kind of computer readable storage medium, be stored in the storage medium at least one instruction, At least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or refer to Collection is enabled to be loaded by the processor and executed the video frame generating method to realize Online Video as described above, alternatively, described At least one instruction, an at least Duan Chengxu, the code set or instruction set are loaded by the processor and are executed to realize The presentation of video frames method of live video as described above.

On the one hand, a kind of electronic equipment is provided, the electronic equipment includes processor and memory, in the memory It is stored at least one instruction, described instruction is loaded by the processor and executed the view to realize Online Video as described above Frequency frame generating method, alternatively, described instruction is loaded by the processor and executed the view to realize live video as described above Frequency frame playback method.

The beneficial effect of technical solution provided by the embodiments of the present application includes at least:

Due to task seize be handle the task mission thread process resource, so, to each mission thread When creating a task queue, asking for the process resource of same mission thread can be seized to avoid the task in different task queue Topic improves the property of terminal so as to avoid the problem for causing the reduced performance of terminal, fever hot due to seizing process resource Energy.In addition, since each mission thread can handle each task, enable the picture of each task in video frame complete Full rendering, to improve the fluency of picture broadcasting.In addition, handling task by multiple tasks thread parallel, can accelerate The processing progress of task avoids the problem of picture plays Caton, improves the fluency of picture broadcasting.

Detailed description of the invention

In order to more clearly explain the technical solutions in the embodiments of the present application, make required in being described below to embodiment Attached drawing is briefly described, it should be apparent that, the drawings in the following description are only some examples of the present application, for For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other Attached drawing.

Fig. 1 is a kind of interface schematic diagram of the competing class live streaming client of electricity provided according to exemplary partial embodiment；

Fig. 2 is a kind of schematic diagram at the direct broadcasting room interface provided according to exemplary partial embodiment；

Fig. 3 is the structural schematic diagram that system is generated according to a kind of video frame that exemplary partial embodiment provides；

Fig. 4 is the method flow diagram of the video frame generating method for the Online Video that the application one embodiment provides；

Fig. 5 is the interaction schematic diagram of the runloop that the application one embodiment provides and mission thread；

Fig. 6 is the method flow diagram of the video frame generating method for the Online Video that another embodiment of the application provides；

Fig. 7 is the flow chart of task processing and rendering that another embodiment of the application provides；

Fig. 8 is the schematic diagram of process flow when exiting backstage that another embodiment of the application provides；

Fig. 9 is the method flow diagram of the presentation of video frames method for the live video that the application one embodiment provides；

Figure 10 is the structural block diagram of the video frame generating apparatus for the Online Video that the application one embodiment provides；

Figure 11 is the structural block diagram of the presentation of video frames device for the live video that the application one embodiment provides；

Figure 12 is the structural block diagram for the terminal that the application another embodiment provides.

Specific embodiment

To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with attached drawing to the application Embodiment is described in further detail.

Before to the embodiment of the present application carrying out that explanation is explained in detail, first the application scenarios of the embodiment of the present application are carried out Explanation.

The embodiment of the present application is applied to broadcast at least two distinct types of data generation one online video that plays In the scene put, it is illustrated below with the typical scene of two of them:

1, live scene

Live scene includes direct broadcasting room live scene and live telecast scene, is carried out separately below to both live scenes Explanation.

1) direct broadcasting room live scene

Main broadcaster user enters direct broadcasting room by the live streaming client of main broadcaster user, spectators user passes through the live streaming of spectators user After client enters direct broadcasting room, between main broadcaster user and main broadcaster user, between main broadcaster user and spectators user can with online interaction, Interaction mode includes but is not limited to send barrage and gifts.At this point, live streaming client is needed barrage sub-data flow, present These three different types of sub-data flows of the video sub-data flow that sub-data flow and main broadcaster shoot generate a videos and play out, To embody the enjoyment of interaction.Here live streaming client includes the live streaming client and/or the live streaming of spectators user of main broadcaster user Client.

Here live streaming client can be the competing class live streaming client of electricity for live game process, be also possible to be used for Client is broadcast live in the amusement class that the entertainments such as song and dance, talk show are broadcast live, and the present embodiment does not limit the type of live streaming client It is fixed.

By taking the competing class live streaming client of electricity as an example, referring to FIG. 1, it illustrates the Shipping Options Page interface of " king " game, in Fig. 1 " BA merciless " and " penguin " be two direct broadcasting rooms, when the user clicks when " BA is merciless ", into the direct broadcasting room of " BA mercilessness ", from mark Label page boundary face jumps to direct broadcasting room interface shown in Fig. 2；Referring to FIG. 2, the region of its top half is playing " BA is merciless " Direct broadcasting room in ongoing game video, intermediate region shows barrage, and the region of lower half portion shows present class Type.

It should be noted that Fig. 2 in interface three different regions show that three kinds of data are illustrated, in practical reality Now, can also be by barrage, present Overlapping display on video pictures, the present embodiment is not to the display shape of various types of data Formula limits.

2) live telecast scene

TV in addition to broadcasting program number evidence, can also play the two dimensional code, short of program side's contact method when playing program The different types of data such as program data, interactive data can be generated a video by the interactive datas such as signal code, TV applications It plays out.

2, Video Applications play scene

Video playing is applied in addition to playing video data, can also play barrage, advertisement link relevant to video data, The different types of data such as video data, interactive data can be generated one by the interactive datas such as advertising slogan, video playing application Video plays out.

In the present embodiment, a mission thread and a task queue are created for each type of sub-data flow, in this way, When processing task, the task in different task queue will not mutually seize the process resource of mission thread, enable each task It is enough fully processed, to ensure that the fluency that picture plays.

Next the system architecture of the embodiment of the present invention is introduced.

Referring to FIG. 3, it illustrates a kind of structural schematic diagrams of video frame generation system provided by the embodiments of the present application.It should It includes at least one terminal 310 and server 320 that video frame, which generates system,.Terminal 310 and server 320 by cable network or Wireless network establishes connection.

Terminal 310 is that have the equipment of data transmit-receive function, such as smart phone.It can be equipped with above in terminal 310 The live streaming client, Television clients, video playing client etc..

For being equipped with live streaming client in terminal 310, then client is broadcast live can be divided into according to the difference of user: The spectator client that the main broadcaster's client and spectators user that main broadcaster user uses use.Wherein, main broadcaster user is to possess interaction power Interactive data and local collected video data can be uploaded to server by the user of limit and live video upload permissions, Then the interactive data and video data are transmitted to main broadcaster's client and spectator client by server, main broadcaster's client can be with The video data and interactive data for receiving server transmission, are then shown；Spectators user is the use for possessing interaction permission Interactive data can be uploaded to server by family, and the interactive data is transmitted to main broadcaster's client and spectators client by server End, spectator client can receive the video data and interactive data of server transmission, be then shown.In addition to this, The client that the client and spectators user that main broadcaster user uses use is essentially identical.

Server 320 can be one or multiple servers, can also be a cloud computing center.In other words, it takes Being engaged in device 320 can be by a server realization；Can also combine realization by multiple servers, every server undertake it is identical or Different functions, for example have the server for registering and logging, the server for saving user's head portrait, for saving channel The server of information and configuration information, server for saving picture or video etc.；Server 320 can also use cloud meter Realize that cloud computing center is that whole piece service cluster is formed by virtual computing platforms in calculation center.

It is illustrated in Fig. 3 with showing two terminals 310 and server 320.

Referring to FIG. 4, it illustrates the sides of the video frame generating method of the Online Video of the application one embodiment offer The video frame generating method of method flow chart, the Online Video can be applied in the client in terminal shown in Fig. 3.This is online The video frame generating method of video, comprising:

Step 401, video data stream is received, includes different types of sub-data flow, every seed number in the video data stream There is different traffic identifier according to stream.

Video data stream is server to the data flow of the video of client push.For example, user is entered by client After direct broadcasting room, video data stream of the server to client push；Alternatively, server is pushed away to client after user opens TV The video data stream sent.

The video data stream of different video includes different types of sub-data flow, and every seed data stream is for providing in video A kind of element and every seed data stream have different traffic identifier, with show distinguish.For example, the video data stream of direct broadcasting room is at least Including video sub-data flow, barrage sub-data flow, present sub-data flow；The video data stream of video playing application includes at least view Frequency sub-data flow, barrage sub-data flow, advertisement sub-data flow.

In the present embodiment, client first sends video acquisition request to server, and server is requested according to the video acquisition Video data stream is determined, by the video data stream to client.

In the present embodiment, traffic identifier can be carried according to the data IP generated in video data stream In the packet header of (Internet Protocol, Internet protocol) data packet, in this way, client is receiving each IP data packet Afterwards, the packet header of each IP data packet can be parsed, it, will be each with the traffic identifier for every kind of traffic identifier being resolved to Data in a IP data packet form a kind of sub-data flow, and video data stream is finally divided into multiple sub-data flows.

Step 402, a mission thread and a task queue are created for every seed data stream, and by the subdata The data segment for belonging to same video frame in stream is added in task queue as a task.

Mission thread is the thread for being handled task.Wherein, thread is referred to as Lightweight Process, is that program is held The minimum unit of row stream.

Task queue is the queue for caching task, and each task is to belong to the data of same video frame in sub-data flow Section.Wherein, task queue is effective always when direct broadcasting room is not turned off, that is, always can be to this during direct broadcasting room is not turned off Task is added in business queue, and reads task from the task queue.

In the present embodiment, client creates a mission thread and a task queue to every seed data stream.For example, view The data flow of frequency includes video sub-data flow, barrage sub-data flow and present sub-data flow, then client can be video subnumber According to stream one mission thread of creation and a task queue, a mission thread and a task team are created for barrage sub-data flow Column create a mission thread and a task queue for present sub-data flow.

For each sub-data flow, which can be divided into several tasks and be added to the subdata by client It flows in corresponding task queue.When sub-data flow is divided into several tasks, one kind is achieved in that in sub-data flow The data of fixed duration are intercepted, using obtained data segment as a task, which is to belong to one in sub-data flow The data of video frame.For example, fixed duration is 1s, then after receiving sub-data flow, the data that interception receives in 0-1 seconds, Obtained data segment is divided into a task to be added in task queue；The data received in interception 1-2s, by what is obtained Data segment is divided into a task and is added in task queue, and so on.

It should be noted that continuing for the video data stream to be divided into and appoint during client receives video data stream Business is added in corresponding task queue.During this period, each mission thread executes step 403 to handle corresponding task queue In task.

Step 403, for mission thread corresponding to the task queue there are task, using the mission thread to task into Row processing, obtains processing result.

In the present embodiment, in the task of generation, client can also using the timestamp of video frame as the timestamp of task, In this way, can determine mission thread according to timestamp to the processing sequence of task, below to the process of mission thread processing task It is introduced.

Circulation executes following steps when each mission thread processing task: the wherein time is searched from corresponding task queue An earliest task of stamp starts the corresponding runloop of the mission thread (operation circulation), which is registered as The input in mode that task registration is the runloop (operation recycles) by the registration observer in the mode of the runloop Source, and the timing source in the mode of runloop is set, during running the runloop according to timing source, utilize the task line Journey such as is decapsulated to the task, is decoded at the processing, obtains a processing result.

Wherein, runloop includes that the input source to be monitored, timing source and the registration observer to be notified, input source are For the task of registration, timing source is the duration that mission thread handles a task, and registration observer is mission thread.? When runloop each run, a mode can be specified, the mode is corresponding with input source, and input source corresponding with the mode is at this Monitored when secondary runloop is run, input source not corresponding with the mode then placed in a suspend state, waits matched mode Runloop starting when could be monitored.In other words, a kind of interested input source can only be monitored by running a runloop.

In the present embodiment, a runloop corresponds to a mission thread, can handle one in a mission thread After task, next task is registered in runloop, to guarantee the mission thread by runloop (operation recycles) Continuous service.Referring to FIG. 5, it illustrates the interaction schematic diagrames of runloop and mission thread, wherein Input Sources It is input source, Port is the input source based on port, and Custom is customized input source, and Timer Sources is timing source, HandlePort is to handle the input source based on port, and customSrc is the processing customized input source of user, and mySelector is The source of Selector is handled, timerFired is processing timing source.

It should be noted that between different task thread can with the task in the corresponding task queue of parallel processing, To accelerate the processing progress of task.

Step 404, each processing result is rendered, obtains video frame.

Client first can be rendered to obtain rendering result to each processing result, reinform hardware to rendering result into Row rendering obtains video frame.Wherein, the process rendered to each processing result is detailed in the description in step 604-607, this Place does not repeat.

Here hardware can be GPU (Graphics Processing Unit, graphics processor), be also possible to other Processor, this embodiment is not limited.After Hardware Render obtains video frame, video frame can be carried out according to certain frame per second It plays, that is, upper screen is carried out to video frame.

In conclusion the video frame generating method of Online Video provided by the embodiments of the present application, since what task was seized is The process resource of the mission thread of the task being handled, so, when creating a task queue to each mission thread, can keep away Exempt from the problem of task in different task queue seizes the process resource of same mission thread, so as to avoid due to seizing processing The problem that resource causes the reduced performance of terminal, fever hot, improves the performance of terminal.In addition, due to each mission thread It can handle each task, the picture of each task in video frame rendered completely, is broadcast to improve picture The fluency put.In addition, handling task by multiple tasks thread parallel, the processing progress of task can be accelerated, avoid picture Face plays the problem of Caton, improves the fluency of picture broadcasting.

Referring to FIG. 6, it illustrates the sides of the video frame generating method of the Online Video of another embodiment offer of the application The video frame generating method of method flow chart, the Online Video can be applied in the client in terminal shown in Fig. 3.This is online The video frame generating method of video, comprising:

Step 601, video data stream is received, includes different types of sub-data flow, every seed number in the video data stream There is different traffic identifier according to stream.

Step 602, a mission thread and a task queue are created for every seed data stream, and by the subdata The data segment for belonging to same video frame in stream is added in task queue as a task.

Step 603, for mission thread corresponding to the task queue there are task, using the mission thread to task into Row processing, obtains processing result.

Wherein, the implementation process of step 601-603 is identical as the implementation process of step 401-403, does not repeat herein.

Step 604, the processing result respectively obtained is sent to rendering thread using each mission thread.

Rendering thread is the thread rendered for the processing result to task.

The mission thread of each processing task is sent the processing result after often obtaining a processing result Give rendering thread.

Step 605, each processing result is added in screen buffer using rendering thread.

Thread is rendered according to the sequence for receiving processing result, each processing result is added in screen buffer, under Two kinds of situations in adding procedure are illustrated.

1) when current time rendering thread receives a processing result, rendering thread directly adds the processing result Into screen buffer.

For example, the list cached in screen buffer are as follows: the processing of the processing result-> video task 2 of video task 1 As a result the processing result of the processing result-> present task 2 of-> present task 1, if the place that current time rendering thread receives Manage the processing result the result is that video task 3, then after the processing result is added in screen buffer, the list are as follows: view The place of the processing result-> present task 2 of the processing result-> present task 1 of the processing result-> video task 2 of frequency task 1 Manage the processing result of result-> video task 3.

2) when current time rendering thread receives at least two processing results:

If it is determined that at least two processing result corresponds to different timestamps, then according to timestamp by early to late suitable At least two processing result is successively added in screen buffer by sequence.

If it is determined that at least two processing result corresponds to identical timestamp, i.e., when there are at least two mission threads are same When sending the processing result for corresponding to same video frame to rendering thread, each processing result is added to using rendering thread May include the following steps in screen buffer:

Step 6051, at least two processing results for corresponding to same video frame are received using rendering thread.

Step 6052, the processing priority of the corresponding sub-data flow of each processing result is determined using rendering thread.

Since rendering thread is according to processing result to be added to the sequence in screen buffer to render processing result, it is The processing result of important task is first rendered in guarantee same video frame, needs to come according to the processing priority of sub-data flow Determine that each processing result is added to the sequence in screen buffer.

In the present embodiment, it can be arranged according to the importance sub-data streams of sub-data flows various types of in video preferential Grade, height and the importance positive correlation of the priority of sub-data flow.That is, sub-data flow is more important, priority is higher. Still it is illustrated, then may be used by taking video data stream includes video sub-data flow, present sub-data flow and barrage sub-data flow as an example The highest priority of video sub-data flow is arranged, the priority that barrage sub-data flow is arranged is minimum.

Step 6053, it is polled using rendering thread according to processing priority, the processing that each poll is obtained As a result it is added in screen buffer.

For example, rendering thread receives the processing result of video task and the processing result of present task, and video simultaneously The priority of sub-data flow is higher than the priority of present sub-data flow, then renders the processing whether thread elder generation poll has video task As a result, the processing result of the video task is added in screen buffer when being polled to the processing result of video task；Wash with watercolours Whether poll has the processing result of present task to appoint the present when being polled to the processing result of present task to dye thread again The processing result of business is added in screen buffer.

Step 606, successively each processing result in screen buffer is rendered using rendering thread, is obtained each Rendering result.

Wherein, successively each processing result in screen buffer is rendered using rendering thread, obtains each wash with watercolours Dye is as a result, may include the following steps:

Step 6061, the drafting context of the corresponding process of client is assigned to rendering thread.

Context is drawn for generating context environmental, is used when GPU is carried out to rendering result and rendered upper screen.Wherein, One process corresponds to a drafting context, and process here is the corresponding process of client, i.e. creation mission thread and wash with watercolours Contaminate the process of thread.

Step 6062, it is added to the sequence in screen buffer according to processing result using rendering thread, successively from view A processing result is read in frequency buffer area.

Step 6063, for each processing result, it is added to drafting up and down using rendering thread creation texture, and by texture Wen Zhong renders processing result, obtains intermediate data, and intermediate data is added in texture, obtains rendering result.

Texture in computer graphics had both included the texture of body surface on ordinary meaning, even if body surface presentation is recessed The rill of convex injustice, the multicolour pattern being also included on the smooth surface of object, in other words, texture refer to the flower of body surface Line.

Here intermediate data can be RGB (Red Green Blue, RGB) data.

Still by taking the list in step 605 as an example, then it will first draw context and be assigned to rendering thread, rendering thread is read The processing result of video task 1 creates texture, and texture is added to and is drawn in context, to the processing result of video task 1 It is rendered, obtains RGB, RGB data is added in texture, obtain the rendering result of video task 1；It renders thread and reads view The processing result of frequency task 2 creates texture, and texture is added to and is drawn in context, to the processing result of video task 2 into Row rendering, obtains RGB, RGB data is added in texture, obtain the rendering result of video task 2；It renders thread and reads present The processing result of task 1 creates texture, and texture is added to and is drawn in context, carries out to the processing result of present task 1 Rendering, obtains RGB, RGB data is added in texture, obtain the rendering result of present task 1, and so on.

Referring to FIG. 7, it illustrates establish task queue, handled each task, processing for poll be arranged The flow chart that priority, poll processing result are added in screen buffer, and are rendered.

Step 607, when obtaining the corresponding each rendering result of a video frame every time, start rendering to GPU transmission and refer to It enables, beginning render instruction is used to indicate GPU and starts the corresponding each rendering result of render video frame, obtains video frame.

In the present embodiment, after obtaining each rendering result, client can notify GPU to carry out wash with watercolours to rendering result immediately Dye and play, since other rendering results of same video frame at this time may be not ready, will lead to picture play it is not smooth, So optionally, GPU can also be reinformed to the video frame after often obtaining the corresponding all rendering results of a video frame Corresponding all rendering results are rendered, and are played again after obtaining video frame.

When client receives the running background instruction of user's sending, backstage is withdrawn into according to running background instruction and is transported Row.In the related technology, if closed line journey, thread not will continue to handle task client, and GPU is notified to generate video Frame plays out, since the operating system of terminal does not allow the client of running background to continue to play video frame, so, operation system System will cause client collapse to prevent client terminal playing video frame.If client closes thread, user may can forget Just in running background client, the utilization rate of client is reduced.In the present embodiment, when client is withdrawn into running background, also Step 608 is executed, to solve the above problems.

Step 608, it when receiving running background instruction, is sent to GPU and stops render instruction, stopped render instruction and be used for Instruction GPU stopping renders rendering result.

In the present embodiment, client not closed line journey in running background, and mission thread will continue to at task Reason, and processing result is sent to rendering thread, processing result is added in screen buffer by rendering thread, and indicates GPU Stopping renders rendering result.In this way, would not also play video frame since GPU does not render rendering result, grasp Make system not and will cause client collapse.In addition, due to client not closed line journey, so, the line of such as audio processing etc Journey can also continue to the sound of output video, to prompt user's client also running, to improve the utilization rate of client.

Optionally, this method further include: when receiving running background instruction, notice rendering thread stops poll.

Polling purpose is preferentially to be added to the processing result of task important in same video frame in screen buffer, Preferentially to be rendered to obtain rendering result to the processing result, so that hardware precedence renders and plays the rendering result.In visitor Family end is in running background, since GPU does not need to render rendering result and played, so, do not need concern processing knot Fruit is added to the sequence in screen buffer, that is, rendering thread can be at random by corresponding at least two processing of same video frame As a result it is added in screen buffer, at this point, client can notify rendering thread to stop poll, to save resource.

It should be noted that rendering thread starts poll when receiving at least two processing results at the same time, without to It renders thread and sends instruction to notify it to start poll.

Optionally, this method further include: when receiving running background instruction, removed using rendering thread and draw context In data texturing, by draw context be assigned to process.

In client in running background, it is also necessary to remove data texturing, otherwise, remaining data texturing will affect video Buffer area.

In a kind of possible implementation, when client receives running background instruction, notice rendering thread stops wheel It askes, rendering thread stops poll, and the data texturing in context is drawn in cleaning, will draw context and be assigned to process, and to GPU It sends and stops render instruction, to indicate that GPU stopping renders rendering result, please refer to process shown in Fig. 8.

In an example, after using method provided in this embodiment, CPU (Central Processing Unit, in Central processor) occupancy from unused this method when measure 38.9% be reduced to 25.6%, and memory is from the unused party The 250MB measured when method is reduced to 190MB, greatly improves the performance of terminal.In addition, side provided in this embodiment is not used Before method, client has 0.5% collapse rate when being withdrawn into running background, after using method provided in this embodiment, client End will not generate collapse when being withdrawn into running background.

When receiving running background instruction, notice rendering thread stops poll, to save resource.

When receiving running background instruction, the data texturing in drafting context is removed to avoid residual using rendering thread Influence of the data texturing stayed to screen buffer.

Referring to FIG. 9, it illustrates the sides of the presentation of video frames method of the live video of another embodiment offer of the application Method flow chart, the presentation of video frames method of the live video can be applied in the live streaming client in terminal shown in Fig. 3.It should The presentation of video frames method of live video, comprising:

Step 901, live video stream is obtained, which includes live streaming picture data stream and interactive data stream, is somebody's turn to do Interactive data stream includes at least one of present data flow and barrage data flow, live video stream, present data flow and barrage Data flow has different traffic identifier.

Live video stream is server to live streaming client push, the video flowing for playing in direct broadcasting room.

Live video stream further includes main broadcaster user and spectators the user generation in interaction in addition to including live streaming picture data stream Interactive data stream.Optionally, interactive data stream includes at least one of present data flow and barrage data flow.Wherein, gift The data flow that object data flow generates when being gifts, barrage data flow are the data flows generated when sending barrage.

In one possible implementation, live streaming client first sends live video acquisition request, service to server Device determines live video stream according to the live video acquisition request, which is sent to live streaming client.

Step 902, for every kind of data flow, a mission thread and a task queue are created, and will be in the data flow The data segment for belonging to same live video frame is added in task queue as a task.

In the present embodiment, live streaming client creates a mission thread and a task queue to every kind of data flow.For example, Live video stream includes live streaming picture data stream, barrage data flow and present data flow, then client, which is broadcast live, to be live streaming picture Face data stream creates a mission thread and a task queue, creates a mission thread and a task for barrage data flow Queue creates a mission thread and a task queue for present data flow.

For each data flow, which can be divided into several tasks and be added to the data flow by live streaming client In corresponding task queue.

It is a kind of possible when being achieved in that interception is fixed in a stream when data flow is divided into several tasks Long data, using obtained data segment as a task, which is to belong to a live video frame in data flow Data.For example, fixed duration is 1s, then after receiving data flow, the data that interception receives in 0-1 seconds, the number that will be obtained A task is divided into according to section to be added in task queue；The data received in interception 1-2s, obtained data segment is divided It is added in task queue for a task, and so on.

Step 903, for mission thread corresponding to the task queue there are task, using the mission thread to task into Row processing, obtains processing result.

Wherein, the implementation process of step 903 and the implementation process of step 403 are identical, do not repeat herein.

Step 904, the processing result respectively obtained is sent to rendering thread using each mission thread.

For the mission thread of each processing task, after mission thread often obtains a processing result, i.e., by the processing As a result it is sent to rendering thread.

Step 905, each processing result is added in screen buffer using rendering thread.

2) when current time rendering thread receives at least two processing results:

If it is determined that at least two processing result corresponds to identical timestamp, i.e., when there are at least two mission threads are same When sending the processing result for corresponding to same live video frame to rendering thread, each processing result added using rendering thread It is added in screen buffer, may include the following steps:

Step 9051, at least two processing results for corresponding to same live video frame are received using rendering thread.

Step 9052, the processing priority of the corresponding data flow of each processing result is determined using rendering thread.

Since rendering thread is according to processing result to be added to the sequence in screen buffer to render processing result, it is The processing result of important task is first rendered in the same live video frame of guarantee, optionally, can be according to the place of data flow Priority is managed to determine that each processing result is added to the sequence in screen buffer.

In the present embodiment, priority, the height of the priority of data flow can be arranged according to the importance of each data flow With importance positive correlation.That is, data flow is more important, priority is higher.It still include live streaming picture with live video stream It is illustrated for face data stream, present data flow and barrage data flow, then the priority of live streaming picture data stream can be set Highest, the priority that barrage data flow is arranged are minimum.

Step 9053, it is polled using rendering thread according to processing priority, the processing that each poll is obtained As a result it is added in screen buffer.

For example, rendering thread receives the processing result of live streaming picture task and the processing result of present task simultaneously, and The priority that picture data stream is broadcast live is higher than the priority of present data flow, then renders whether thread elder generation poll has live streaming picture to appoint The processing result of business adds the processing result of the live streaming picture task when being polled to the processing result of live streaming picture task Into screen buffer；Whether poll has the processing result of present task to rendering thread again, when the processing for being polled to present task When as a result, the processing result of the present task is added in screen buffer.

Step 906, successively each processing result in screen buffer is rendered using rendering thread, is obtained each Rendering result.

Step 9061, the drafting context that the corresponding process of client is broadcast live is assigned to rendering thread.

Context is drawn for generating context environmental, is used when GPU is carried out to rendering result and rendered upper screen.Wherein, One process corresponds to a drafting context, and process here is the corresponding process of live streaming client, i.e. creation mission thread With the process of rendering thread.

Step 9062, it is added to the sequence in screen buffer according to processing result using rendering thread, successively from view A processing result is read in frequency buffer area.

Step 9063, for each processing result, it is added to drafting up and down using rendering thread creation texture, and by texture Wen Zhong renders processing result, obtains intermediate data, and intermediate data is added in texture, obtains rendering result.

Step 907, it when obtaining the corresponding each rendering result of a live video frame every time, is sent to GPU and starts wash with watercolours Order is had a finger in every pie, beginning render instruction is used to indicate GPU and starts to render the corresponding each rendering result of live video frame, obtains live streaming view Frequency frame, the live video frame include live streaming picture and interactive data.

In the present embodiment, after obtaining each rendering result, live streaming client can notify GPU immediately to rendering result into It row rendering and plays, since other rendering results of same live video frame at this time may be not ready, will lead to picture and broadcast Put it is not smooth, so, optionally, can also be reinformed after often obtaining the corresponding all rendering results of a live video frame GPU renders the corresponding all rendering results of the live video frame, obtains live video frame.

Step 908, live video frame is played.

When live streaming client receives the running background instruction of user's sending, after being withdrawn into according to running background instruction Platform operation.In the related technology, if closed line journey, mission thread and rendering thread not will continue to carry out task live streaming client Processing, and GPU is notified to generate live video frame and play out, since the operating system of terminal does not allow the live streaming visitor of running background Family end continues to play live video frame, so, operating system will cause live streaming client collapse to prevent live streaming client terminal playing Live video frame.If client, which is broadcast live, closes thread, user may can forget that client just is broadcast live in running background, reduce The utilization rate of client is broadcast live.In the present embodiment, when live streaming client is withdrawn into running background, step 608 is executed, also with solution The certainly above problem.

Step 909, it when receiving running background instruction, is sent to GPU and stops render instruction, stopped render instruction and be used for Instruction GPU stopping renders rendering result.

In the present embodiment, be broadcast live client not closed line journey in running background, and mission thread will continue to task into Row processing, and processing result is sent to rendering thread, processing result is added in screen buffer by rendering thread, and indicates GPU stopping renders rendering result.In this way, would not also play video since GPU does not render rendering result Frame, operating system not will cause live streaming client collapse.In addition, since client not closed line journey is broadcast live, so, such as audio The thread of processing etc can also continue to the sound of output video, to prompt user's live streaming client also running, to mention The utilization rate of height live streaming client.

Polling purpose is that the processing result of task important in same live video frame is preferentially added to video buffer Qu Zhong, preferentially to be rendered to obtain rendering result to the processing result, so that GPU is preferentially rendered and played the rendering result. In live streaming client in running background, since GPU does not need to render rendering result and played, so, it does not need to close Note processing result is added to the sequence in screen buffer, that is, rendering thread can be corresponding by same live video frame at random At least two processing results are added in screen buffer, at this point, live streaming client can notify rendering thread to stop poll, with Save resource.

In live streaming client in running background, it is also necessary to remove data texturing, otherwise, remaining data texturing will affect Screen buffer.

In a kind of possible implementation, when live streaming client receives running background instruction, notice rendering thread stops Only poll, rendering thread stop poll, and the data texturing in context is drawn in cleaning, will draw context and are assigned to process, and It is sent to GPU and stops render instruction, to indicate that GPU stopping renders rendering result.

In conclusion the presentation of video frames method of live video provided by the embodiments of the present application, since what task was seized is The process resource of the mission thread of the task being handled, so, when creating a task queue to each mission thread, can keep away Exempt from the problem of task in different task queue seizes the process resource of same mission thread, so as to avoid due to seizing processing The problem that resource causes the reduced performance of terminal, fever hot, improves the performance of terminal.In addition, due to each mission thread It can handle each task, the picture of each task in live video frame rendered completely, to improve straight Broadcast the fluency of picture broadcasting.In addition, handling task by multiple tasks thread parallel, the processing progress of task can be accelerated, The problem of live streaming picture plays Caton is avoided, the fluency that live streaming picture plays is improved.

Referring to FIG. 10, it illustrates the knots of the video frame generating apparatus of the Online Video of the application one embodiment offer The video frame generating apparatus of structure block diagram, the Online Video can be applied in the client in terminal shown in Fig. 3.This is regarded online The video frame generating apparatus of frequency, comprising:

Receiving module 1010 includes different types of sub-data flow in video data stream, often for receiving video data stream Seed data stream has different traffic identifier；

Creation module 1020, every seed data stream for receiving for receiving module 1010, creates a task line Journey and a task queue, and it is added to task team for the data segment for belonging to same video frame in sub-data flow as a task In column；

Processing module 1030, for utilizing mission thread for mission thread corresponding to the task queue there are task Task is handled, processing result is obtained；

Generation module 1040 obtains video frame for rendering to each processing result that processing module 1030 obtains.

Optionally, generation module 1040 are also used to:

The processing result respectively obtained is sent to rendering thread using each mission thread；

Each processing result is added in screen buffer using rendering thread；

Successively each processing result in screen buffer is rendered using rendering thread, obtains each rendering knot Fruit；

Notice GPU renders each rendering result, obtains video frame.

Optionally, when there are the processing that at least two mission threads correspond to same video frame to rendering thread transmission simultaneously When as a result, generation module 1040 is also used to:

At least two processing results for corresponding to same video frame are received using rendering thread；

The processing priority of the corresponding sub-data flow of each processing result is determined using rendering thread；

It is polled using rendering thread according to processing priority, the processing result that each poll is obtained is added to In screen buffer.

Optionally, generation module 1040, when being also used to platform operating instruction upon receipt, notice rendering thread stops wheel It askes.

Optionally, generation module 1040 are also used to:

The drafting context of the corresponding process of client is assigned to rendering thread；

It is added to the sequence in screen buffer according to processing result using rendering thread, successively from screen buffer Read a processing result；

For each processing result, using rendering thread creation texture, and texture is added to and is drawn in context, to place Reason result is rendered, and obtains intermediate data, intermediate data is added in texture, obtain rendering result.

Optionally, the device further include:

Module is removed, when for platform operating instruction upon receipt, removes the line drawn in context using rendering thread Data are managed, context will be drawn and be assigned to process.

Optionally, generation module 1040 are also used to when obtaining the corresponding each rendering result of a video frame every time, to Hardware, which is sent, starts render instruction, and beginning render instruction is used to indicate GPU and starts the corresponding each rendering result of render video frame, Obtain video frame.

Optionally, the device further include:

Sending module sends to GPU when for platform operating instruction upon receipt and stops render instruction, stops rendering and refers to Order is used to indicate GPU stopping and renders to rendering result.

In conclusion the video frame generating apparatus of Online Video provided by the embodiments of the present application, since what task was seized is The process resource of the mission thread of the task being handled, so, when creating a task queue to each mission thread, can keep away Exempt from the problem of task in different task queue seizes the process resource of same mission thread, so as to avoid due to seizing processing The problem that resource causes the reduced performance of terminal, fever hot, improves the performance of terminal.In addition, due to each mission thread It can handle each task, the picture of each task in video frame rendered completely, is broadcast to improve picture The fluency put.In addition, handling task by multiple tasks thread parallel, the processing progress of task can be accelerated, avoid picture Face plays the problem of Caton, improves the fluency of picture broadcasting.

Figure 11 is please referred to, it illustrates the knots of the presentation of video frames device of the live video of the application one embodiment offer Structure block diagram, the presentation of video frames device of the live video can be applied in the live streaming client in terminal shown in Fig. 3.This is straight Broadcast the presentation of video frames device of video, comprising:

Module 1110 is obtained, for obtaining live video stream, live video stream includes live streaming picture data stream and interaction number According to stream, interactive data stream includes at least one of present data flow and barrage data flow, live video stream, present data flow and Barrage data flow has different traffic identifier；

Creation module 1120, for for obtaining the obtained every kind of data flow of module 1110, create a mission thread and One task queue, and it is added to task queue for the data segment for belonging to same live video frame in data flow as a task In；

Processing module 1130, for utilizing mission thread for mission thread corresponding to the task queue there are task Task is handled, processing result is obtained；

Generation module 1140, for being rendered to obtain live video to each processing result that processing module 1130 obtains Frame, live video frame include live streaming picture and interactive data；

Playing module 1150, for playing the live video frame of the generation of generation module 1140.

Optionally, generation module 1140 are also used to:

Each processing result is added in screen buffer using rendering thread；

Notice GPU renders each rendering result, obtains live video frame.

Optionally, when there are at least two mission threads to send simultaneously to rendering thread corresponding to same live video frame When processing result, generation module 1040 is also used to:

At least two processing results for corresponding to same live video frame are received using rendering thread；

Optionally, generation module 1040 are also used to:

The drafting context that the corresponding process of client is broadcast live is assigned to rendering thread；

Optionally, the device further include:

Optionally, generation module 1040 are also used to obtaining the corresponding each rendering result of a live video frame every time When, to GPU send start render instruction, start render instruction be used to indicate GPU start render live video frame it is corresponding each Rendering result obtains live video frame.

Optionally, the device further include:

In conclusion the presentation of video frames device of live video provided by the embodiments of the present application, since what task was seized is The process resource of the mission thread of the task being handled, so, when creating a task queue to each mission thread, can keep away Exempt from the problem of task in different task queue seizes the process resource of same mission thread, so as to avoid due to seizing processing The problem that resource causes the reduced performance of terminal, fever hot, improves the performance of terminal.In addition, due to each mission thread It can handle each task, the picture of each task in video frame rendered completely, is broadcast to improve picture The fluency put.In addition, handling task by multiple tasks thread parallel, the processing progress of task can be accelerated, avoid picture Face plays the problem of Caton, improves the fluency of picture broadcasting.

Figure 12 shows the structural block diagram of the terminal 1200 of one exemplary embodiment of the application offer.The terminal 1200 can To be portable mobile termianl, such as: smart phone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III, dynamic image expert's compression standard audio level 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert's compression standard audio level 4) player, laptop Or desktop computer.Terminal 1200 be also possible to referred to as user equipment, portable terminal, laptop terminal, terminal console etc. other Title.

In general, terminal 1200 includes: processor 1201 and memory 1202.

Processor 1201 may include one or more processing cores, such as 4 core processors, 8 core processors etc..Place Reason device 1201 can use DSP (Digital Signal Processing, Digital Signal Processing), FPGA (Field- Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, may be programmed Logic array) at least one of example, in hardware realize.Processor 1201 also may include primary processor and coprocessor, master Processor is the processor for being handled data in the awake state, also referred to as CPU (Central Processing Unit, central processing unit)；Coprocessor is the low power processor for being handled data in the standby state.? In some embodiments, processor 1201 can be integrated with GPU (Graphics Processing Unit, image processor), GPU is used to be responsible for the rendering and drafting of content to be shown needed for display screen.In some embodiments, processor 1201 can also be wrapped AI (Artificial Intelligence, artificial intelligence) processor is included, the AI processor is for handling related machine learning Calculating operation.

Memory 1202 may include one or more computer readable storage mediums, which can To be non-transient.Memory 1202 may also include high-speed random access memory and nonvolatile memory, such as one Or multiple disk storage equipments, flash memory device.In some embodiments, the non-transient computer in memory 1202 can Storage medium is read for storing at least one instruction, at least one instruction performed by processor 1201 for realizing this Shen Please in embodiment of the method provide Online Video video frame generating method and/or live video presentation of video frames method.

In some embodiments, terminal 1200 is also optional includes: peripheral device interface 1203 and at least one periphery are set It is standby.It can be connected by bus or signal wire between processor 1201, memory 1202 and peripheral device interface 1203.It is each outer Peripheral equipment can be connected by bus, signal wire or circuit board with peripheral device interface 1203.Specifically, peripheral equipment includes: In radio circuit 1204, touch display screen 1205, camera 1206, voicefrequency circuit 1207, positioning component 1208 and power supply 1209 At least one.

Peripheral device interface 1203 can be used for I/O (Input/Output, input/output) is relevant outside at least one Peripheral equipment is connected to processor 1201 and memory 1202.In some embodiments, processor 1201, memory 1202 and periphery Equipment interface 1203 is integrated on same chip or circuit board；In some other embodiments, processor 1201, memory 1202 and peripheral device interface 1203 in any one or two can be realized on individual chip or circuit board, this implementation Example is not limited this.

Radio circuit 1204 is for receiving and emitting RF (Radio Frequency, radio frequency) signal, also referred to as electromagnetic signal. Radio circuit 1204 is communicated by electromagnetic signal with communication network and other communication equipments.Radio circuit 1204 is by telecommunications Number being converted to electromagnetic signal is sent, alternatively, the electromagnetic signal received is converted to electric signal.Optionally, radio circuit 1204 include: antenna system, RF transceiver, one or more amplifiers, tuner, oscillator, digital signal processor, volume solution Code chipset, user identity module card etc..Radio circuit 1204 can by least one wireless communication protocol come with it is other Terminal is communicated.The wireless communication protocol includes but is not limited to: Metropolitan Area Network (MAN), each third generation mobile communication network (2G, 3G, 4G and 5G), WLAN and/or WiFi (Wireless Fidelity, Wireless Fidelity) network.In some embodiments, radio frequency electrical Road 1204 can also include NFC (Near Field Communication, wireless near field communication) related circuit, the application This is not limited.

Display screen 1205 is for showing UI (User Interface, user interface).The UI may include figure, text, Icon, video and its their any combination.When display screen 1205 is touch display screen, display screen 1205 also there is acquisition to exist The ability of the touch signal on the surface or surface of display screen 1205.The touch signal can be used as control signal and be input to place Reason device 1201 is handled.At this point, display screen 1205 can be also used for providing virtual push button and/or dummy keyboard, it is also referred to as soft to press Button and/or soft keyboard.In some embodiments, display screen 1205 can be one, and the front panel of terminal 1200 is arranged；Another In a little embodiments, display screen 1205 can be at least two, be separately positioned on the different surfaces of terminal 1200 or in foldover design； In still other embodiments, display screen 1205 can be flexible display screen, is arranged on the curved surface of terminal 1200 or folds On face.Even, display screen 1205 can also be arranged to non-rectangle irregular figure, namely abnormity screen.Display screen 1205 can be with Using LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, Organic Light Emitting Diode) etc. materials preparation.

CCD camera assembly 1206 is for acquiring image or video.Optionally, CCD camera assembly 1206 includes front camera And rear camera.In general, the front panel of terminal is arranged in front camera, the back side of terminal is arranged in rear camera.? In some embodiments, rear camera at least two is that main camera, depth of field camera, wide-angle camera, focal length are taken the photograph respectively As any one in head, to realize that main camera and the fusion of depth of field camera realize background blurring function, main camera and wide Pan-shot and VR (Virtual Reality, virtual reality) shooting function or other fusions are realized in camera fusion in angle Shooting function.In some embodiments, CCD camera assembly 1206 can also include flash lamp.Flash lamp can be monochromatic temperature flash of light Lamp is also possible to double-colored temperature flash lamp.Double-colored temperature flash lamp refers to the combination of warm light flash lamp and cold light flash lamp, can be used for Light compensation under different-colour.

Voicefrequency circuit 1207 may include microphone and loudspeaker.Microphone is used to acquire the sound wave of user and environment, and It converts sound waves into electric signal and is input to processor 1201 and handled, or be input to radio circuit 1204 to realize that voice is logical Letter.For stereo acquisition or the purpose of noise reduction, microphone can be separately positioned on the different parts of terminal 1200 to be multiple. Microphone can also be array microphone or omnidirectional's acquisition type microphone.Loudspeaker is then used to that processor 1201 or radio frequency will to be come from The electric signal of circuit 1204 is converted to sound wave.Loudspeaker can be traditional wafer speaker, be also possible to piezoelectric ceramics loudspeaking Device.When loudspeaker is piezoelectric ceramic loudspeaker, the audible sound wave of the mankind can be not only converted electrical signals to, can also be incited somebody to action Electric signal is converted to the sound wave that the mankind do not hear to carry out the purposes such as ranging.In some embodiments, voicefrequency circuit 1207 may be used also To include earphone jack.

Positioning component 1208 is used for the current geographic position of positioning terminal 1200, to realize navigation or LBS (Location Based Service, location based service).Positioning component 1208 can be the GPS (Global based on the U.S. Positioning System, global positioning system), the dipper system of China, Russia Gray receive this system or European Union The positioning component of Galileo system.

Power supply 1209 is used to be powered for the various components in terminal 1200.Power supply 1209 can be alternating current, direct current Electricity, disposable battery or rechargeable battery.When power supply 1209 includes rechargeable battery, which can support wired Charging or wireless charging.The rechargeable battery can be also used for supporting fast charge technology.

In some embodiments, terminal 1200 further includes having one or more sensors 1210.One or more sensing Device 1210 includes but is not limited to: acceleration transducer 1211, gyro sensor 1212, pressure sensor 1213, fingerprint sensing Device 1214, optical sensor 1215 and proximity sensor 1216.

Acceleration transducer 1211 can detecte the acceleration in three reference axis of the coordinate system established with terminal 1200 Size.For example, acceleration transducer 1211 can be used for detecting component of the acceleration of gravity in three reference axis.Processor The 1201 acceleration of gravity signals that can be acquired according to acceleration transducer 1211, control touch display screen 1205 with transverse views Or longitudinal view carries out the display of user interface.Acceleration transducer 1211 can be also used for game or the exercise data of user Acquisition.

Gyro sensor 1212 can detecte body direction and the rotational angle of terminal 1200, gyro sensor 1212 Acquisition user can be cooperateed with to act the 3D of terminal 1200 with acceleration transducer 1211.Processor 1201 is according to gyro sensors The data that device 1212 acquires, following function may be implemented: action induction (for example changing UI according to the tilt operation of user) is clapped Image stabilization, game control and inertial navigation when taking the photograph.

The lower layer of side frame and/or touch display screen 1205 in terminal 1200 can be set in pressure sensor 1213.When When the side frame of terminal 1200 is arranged in pressure sensor 1213, user can detecte to the gripping signal of terminal 1200, by Reason device 1201 carries out right-hand man's identification or prompt operation according to the gripping signal that pressure sensor 1213 acquires.Work as pressure sensor 1213 when being arranged in the lower layer of touch display screen 1205, is grasped by processor 1201 according to pressure of the user to touch display screen 1205 Make, realization controls the operability control on the interface UI.Operability control include button control, scroll bar control, At least one of icon control, menu control.

Fingerprint sensor 1214 is used to acquire the fingerprint of user, is collected by processor 1201 according to fingerprint sensor 1214 Fingerprint recognition user identity, alternatively, by fingerprint sensor 1214 according to the identity of collected fingerprint recognition user.Knowing Not Chu the identity of user when being trusted identity, authorize the user to execute relevant sensitive operation by processor 1201, which grasps Make to include solving lock screen, checking encryption information, downloading software, payment and change setting etc..Fingerprint sensor 1214 can be set Set the front, the back side or side of terminal 1200.When being provided with physical button or manufacturer Logo in terminal 1200, fingerprint sensor 1214 can integrate with physical button or manufacturer Logo.

Optical sensor 1215 is for acquiring ambient light intensity.In one embodiment, processor 1201 can be according to light The ambient light intensity that sensor 1215 acquires is learned, the display brightness of touch display screen 1205 is controlled.Specifically, work as ambient light intensity When higher, the display brightness of touch display screen 1205 is turned up；When ambient light intensity is lower, the aobvious of touch display screen 1205 is turned down Show brightness.In another embodiment, the ambient light intensity that processor 1201 can also be acquired according to optical sensor 1215, is moved The acquisition parameters of state adjustment CCD camera assembly 1206.

Proximity sensor 1216, also referred to as range sensor are generally arranged at the front panel of terminal 1200.Proximity sensor 1216 for acquiring the distance between the front of user Yu terminal 1200.In one embodiment, when proximity sensor 1216 is examined When measuring the distance between the front of user and terminal 1200 and gradually becoming smaller, by processor 1201 control touch display screen 1205 from Bright screen state is switched to breath screen state；When proximity sensor 1216 detect the distance between front of user and terminal 1200 by When gradual change is big, touch display screen 1205 is controlled by processor 1201 and is switched to bright screen state from breath screen state.

It, can be with it will be understood by those skilled in the art that the restriction of the not structure paired terminal 1200 of structure shown in Figure 12 Including than illustrating more or fewer components, perhaps combining certain components or being arranged using different components.

The application one embodiment provides a kind of computer readable storage medium, is stored at least in the storage medium One instruction, at least a Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the generation Code collection or instruction set are loaded by the processor and are executed the video frame generating method to realize Online Video as described above, or Person, at least one instruction, an at least Duan Chengxu, the code set or instruction set are loaded and are executed by the processor To realize the presentation of video frames method of live video as described above.

The application one embodiment provides a kind of electronic equipment, and the electronic equipment includes processor and memory, institute It states and is stored at least one instruction in memory, described instruction is loaded by the processor and executed to realize and exist as described above The video frame generating method of line video, alternatively, described instruction loaded by the processor and executed it is as described above straight to realize Broadcast the presentation of video frames method of video.

It should be understood that the video frame generating apparatus of Online Video provided by the above embodiment is carrying out video frame life At the presentation of video frames device of, live video when playing video frame, only illustrate with the division of above-mentioned each functional module It is bright, in practical application, it can according to need and be completed by different functional modules above-mentioned function distribution, i.e., by Online Video The internal structure of video frame generating apparatus is divided into different functional modules, by the inside of the presentation of video frames device of live video Structure is divided into different functional modules to complete all or part of the functions described above.In addition, above-described embodiment provides The video frame generating apparatus of Online Video and the video frame generating method embodiment of Online Video belong to same design, live streaming view The presentation of video frames device of frequency and the presentation of video frames embodiment of the method for live video belong to same design, implement process It is detailed in embodiment of the method, which is not described herein again.

Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..

The above is not to limit the embodiment of the present application, all within the spirit and principle of the embodiment of the present application, institute Any modification, equivalent substitution, improvement and etc. of work, should be included within the protection scope of the embodiment of the present application.

Claims

1. a kind of video frame generating method of Online Video, which is characterized in that in client, which comprises

Video data stream is received, includes different types of sub-data flow in the video data stream, every seed data stream has not Same traffic identifier；

For every seed data stream, a mission thread and a task queue are created, and will be belonged in the sub-data flow same The data segment of one video frame is added in the task queue as a task；

For mission thread corresponding to the task queue there are task, using the mission thread to the task at Reason, obtains processing result；

Each processing result is rendered, video frame is obtained.

2. obtaining video the method according to claim 1, wherein described render each processing result Frame, comprising:

Each processing result is added in screen buffer using the rendering thread；

Successively each processing result in the screen buffer is rendered using the rendering thread, obtains each rendering As a result；

Notice graphics processor GPU renders each rendering result, obtains the video frame.

3. according to the method described in claim 2, it is characterized in that, when there are at least two mission threads simultaneously to the rendering It is described to be added to each processing result using the rendering thread when thread sends the processing result for corresponding to same video frame In screen buffer, comprising:

At least two processing results for corresponding to same video frame are received using the rendering thread；

The processing priority of the corresponding sub-data flow of each processing result is determined using the rendering thread；

It is polled using the rendering thread according to the processing priority, the processing result that each poll is obtained adds It is added in the screen buffer.

4. according to the method described in claim 2, it is characterized in that, the method also includes:

When receiving running background instruction, the rendering thread is notified to stop poll.

5. according to the method described in claim 2, it is characterized in that, described successively slow to the video using the rendering thread The each processing result rushed in area is rendered, and each rendering result is obtained, comprising:

The drafting context of the corresponding process of the client is assigned to the rendering thread；

It is added to the sequence in the screen buffer according to the processing result using the rendering thread, successively from described A processing result is read in screen buffer；

For each processing result, using the rendering thread creation texture, and the texture is added to the drafting or more Wen Zhong renders the processing result, obtains intermediate data, and the intermediate data is added in the texture, is obtained The rendering result.

6. according to the method described in claim 5, it is characterized in that, the method also includes:

When receiving running background instruction, the data texturing in the drafting context is removed using the rendering thread, it will The drafting context is assigned to the process.

7. being obtained described the method according to claim 1, wherein described render each rendering result Video frame, comprising:

When obtaining the corresponding each rendering result of a video frame every time, is sent to graphics processor GPU and starts render instruction, The beginning render instruction is used to indicate the GPU and starts to render the corresponding each rendering result of the video frame, obtains described Video frame.

8. method according to any one of claims 1 to 7, which is characterized in that the method also includes:

It when receiving running background instruction, is sent to graphics processor GPU and stops render instruction, the stopping render instruction being used The rendering result is rendered in the instruction GPU stopping.

9. a kind of presentation of video frames method of live video, which is characterized in that for being broadcast live in client, which comprises

Live video stream is obtained, the live video stream includes live streaming picture data stream and interactive data stream, the interactive data Stream includes at least one of present data flow and barrage data flow, the live video stream, the present data flow and described Barrage data flow has different traffic identifier；

For every kind of data flow, a mission thread and a task queue are created, and will be belonged in the data flow with always The data segment for broadcasting video frame is added in the task queue as a task；

Each processing result is rendered to obtain live video frame, the live video frame includes live streaming picture and interaction number According to；

Play the live video frame.

10. according to the method described in claim 9, it is characterized in that, described rendered to each processing result is broadcast live Video frame, comprising:

Each processing result is added in screen buffer using the rendering thread；

Notice graphics processor GPU renders each rendering result to obtain the live video frame.

11. a kind of video frame generating apparatus of Online Video, which is characterized in that in client, described device to include:

Receiving module includes different types of sub-data flow, every seed in the video data stream for receiving video data stream Data flow has different traffic identifier；

Creation module, every seed data stream for receiving for the receiving module, creates a mission thread and one Task queue, and it is added to the task team for the data segment for belonging to same video frame in the sub-data flow as a task In column；

Processing module, for for mission thread corresponding to the task queue there are task, using the mission thread to institute The task of stating is handled, and each processing result is obtained；

12. device according to claim 10, which is characterized in that the generation module is also used to:

Each processing result is added in screen buffer using the rendering thread；

13. a kind of presentation of video frames device of live video, which is characterized in that for being broadcast live in client, described device includes:

Module is obtained, for obtaining live video stream, the live video stream includes that picture data stream and interactive data stream is broadcast live, The interactive data stream includes at least one of present data flow and barrage data flow, the live video stream, the present Data flow and the barrage data flow have different traffic identifier；

Creation module, every kind of data flow for obtaining for the acquisition module, creates a mission thread and a task Queue, and be added to the data segment for belonging to same video frame in the data flow as a task in the task queue；

Processing module, for for mission thread corresponding to the task queue there are task, using the mission thread to institute The task of stating is handled, and processing result is obtained；

Generation module obtains live video frame for being rendered to each processing result that the processing module obtains, described Live video frame includes live streaming picture and interactive data；

14. a kind of computer readable storage medium, which is characterized in that be stored at least one instruction, extremely in the storage medium A few Duan Chengxu, code set or instruction set, at least one instruction, an at least Duan Chengxu, the code set or instruction Collection is loaded by the processor and is executed the video frame generation side to realize Online Video as described in any of the claims 1 to 8 Method, alternatively, at least one instruction, an at least Duan Chengxu, the code set or instruction set are loaded by the processor And the presentation of video frames method executed to realize the live video as described in claim 9 or 10.

15. a kind of electronic equipment, which is characterized in that the electronic equipment includes processor and memory, is deposited in the memory At least one instruction is contained, described instruction is loaded by the processor and executed as described in any of the claims 1 to 8 to realize The video frame generating method of Online Video, alternatively, described instruction is loaded by the processor and executed to realize such as claim 9 Or the presentation of video frames method of live video described in 10.