CN111654740A - Rendering method and device in video playing process and electronic equipment - Google Patents
Rendering method and device in video playing process and electronic equipment Download PDFInfo
- Publication number
- CN111654740A CN111654740A CN202010590739.8A CN202010590739A CN111654740A CN 111654740 A CN111654740 A CN 111654740A CN 202010590739 A CN202010590739 A CN 202010590739A CN 111654740 A CN111654740 A CN 111654740A
- Authority
- CN
- China
- Prior art keywords
- interface
- frame data
- video
- rendering
- video playing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4318—Generation of visual interfaces for content selection or interaction; Content or additional data rendering by altering the content in the rendering process, e.g. blanking, blurring or masking an image region
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440227—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by decomposing into layers, e.g. base layer and one or more enhancement layers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/458—Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules ; time-related management operations
- H04N21/4586—Content update operation triggered locally, e.g. by comparing the version of software modules in a DVB carousel to the version stored locally
Abstract
The embodiment of the invention discloses a rendering method and device in a video playing process. Therefore, by adopting the embodiment of the invention, when the interface is refreshed but the frame data is not updated, the frame data is refreshed while the interface is refreshed by reading the decoded frame data played by the video, so that the situation that the decoded frame data displayed in the video layer of the interface is not re-rendered is reduced, the problems of blockage or ghost and the like of the picture displayed by the interface can be avoided, the picture quality of the video playing in the interface is improved, and good experience is brought to a user.
Description
Technical Field
The present invention relates to image processing technologies, and in particular, to a method and an apparatus for rendering during video playing, and an electronic device.
Background
Before playing the video, it is necessary to render each frame data required for playing the video. Currently, each frame data decoding and rendering of video playing is completed by an independent unit in a video playing module of a computer system, and after the independent unit decodes and renders frame data required by video playing frame by frame, the frame data after decoding and rendering is sent to a video layer of an interface set by the computer system for frame by frame display, so that a video playing process is realized. Fig. 1 is a schematic structural diagram of a module for rendering frame data required for video playback provided in the prior art, which is a video playback module of a computer system, and includes: a video display control unit, a stream fetching unit and a decoding and rendering unit, wherein,
the video display control unit is used for sending a request for acquiring frame data required by playing the video to the stream taking unit; acquiring decoded and rendered frame data frame by frame from a decoding and rendering unit, and displaying the frame data frame by frame in a video layer of the set interface;
the stream taking unit is used for acquiring frame data required by playing the video frame by frame from the computer storage unit or the computer network according to the request and sending the frame data to the decoding and rendering unit;
and the decoding and rendering unit is used for decoding and rendering the frame data required by the playing video frame by frame and then sending the decoded and rendered frame data to the video display control unit frame by frame.
And displaying the frame data required by the played video frame by frame, namely frame data refreshing. The video display control unit can also refresh the interface loaded with the video layer, which is referred to as interface refresh for short, for example, when an interface change event is triggered, an interface refresh instruction can be obtained. The interface change event comprises: the content in the interface covers partial images in the video layer, or events which can cause the change of the video layer of the interface such as interface zooming, and the like, and interface refreshing and frame data refreshing are two sets of processing mechanisms and are not synchronous.
There is no data update between two frames of data of frame data required by playing video, frame data refreshing is not performed, but an event causing interface change may be triggered in the interval, so as to cause interface refreshing, and at this time, the problem of image displayed on the interface, such as blocking or ghost, occurs, so that the image quality of video playing in the interface is reduced, and the user experience is affected.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method for rendering in a video playing process, where the method can improve the picture quality of video playing in an interface when the interface is refreshed.
The embodiment of the invention also provides a rendering device in the video playing process, and the device can improve the picture quality of video playing in the interface when the interface is refreshed.
In one embodiment, a method of rendering during video playback is provided, comprising:
obtaining an interface refreshing instruction, and reading decoded frame data played by a video;
and rendering an interface according to the interface refreshing related information and the decoding frame data.
Preferably, before the reading of the decoded frame data of the video playback, the method further comprises:
acquiring a video playing instruction, acquiring frame data required by a video playing process frame by frame, decoding to obtain decoded frame data, and storing the decoded frame data;
the reading the decoded frame data of the video playing comprises: reading the stored decoded frame data.
Preferably, the decoded frame data is decoded frame data of a current video playing.
Preferably, the method further comprises:
stopping video playing and emptying the stored decoded frame data;
emptying the rendering in the video layer of the interface.
Preferably, the reading of the decoded frame data of the video playing; rendering an interface according to the interface refresh related information and the decoded frame data includes:
under the condition of obtaining an interface refreshing instruction, blocking a main thread of a video playing interface, and after reading the decoding frame data of video playing by a sub-thread of the main thread of the video playing interface, ending the blocking of the main thread of the video playing interface;
and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
Preferably, the method further comprises:
setting a sub-control in a video layer of the interface to superpose a set window, wherein the window is in transparent presentation or animation presentation on the interface;
the obtaining of the interface refreshing instruction comprises: and obtaining an interface refreshing instruction under the condition that the window is changed.
In another embodiment, an apparatus for rendering during video playing is provided, including: a video presentation control unit and a rendering unit, wherein,
the video display control unit is used for obtaining an interface refreshing instruction;
the rendering unit is used for reading the played decoded frame data; and rendering an interface according to the interface refreshing related information and the decoding frame data.
Preferably, the apparatus further comprises a stream fetching unit and a decoding unit, wherein,
the video display control unit is also used for obtaining a video playing instruction;
the stream taking unit is used for acquiring frame data required by a video playing process frame by frame;
the decoding unit is used for decoding the frame data acquired by the stream taking unit to obtain decoded frame data;
the rendering unit is further configured to store the decoded frame data.
Preferably, the decoded frame data is decoded frame data of a current video playing.
Preferably, the rendering unit is specifically configured to block a main thread of a video playing interface when the interface refresh instruction is obtained, and end the blocking of the main thread of the video playing interface after a sub-thread of the main thread of the video playing interface reads decoded frame data of video playing and obtains interface refresh related information; and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
Preferably, the video presentation control unit is further configured to stop video playing and empty the stored decoded frame data;
the rendering unit is further configured to empty the rendering in the video layer of the interface.
Preferably, the video presentation control unit is specifically configured to set a sub-control in a video layer of the interface to superimpose the set window, where the window is transparently presented or animated on the interface; and obtaining the triggered interface refreshing instruction when the window is changed.
In another embodiment, there is provided an electronic device including: a processor of one or more processing cores, a memory of one or more computer readable storage media, and a computer program stored on the memory and executable on the processor, which when executed, implement the method of rendering during video playback as described in the foregoing embodiments.
In another embodiment, a non-transitory computer readable storage medium is provided that stores instructions that, when executed by a processor, cause the processor to perform a method of rendering during video playback as described in the preceding embodiments.
In another embodiment, a computer program product containing instructions is provided, which when run on a computer causes the computer to perform the method of rendering during video playback as described in the previous embodiments.
As can be seen from the above, in the embodiment of the present invention, when the interface is refreshed, the rendering process is triggered, and the interface is rendered according to the interface refresh related information and the acquired decoded frame data of the video playing. Therefore, by adopting the embodiment of the invention, when the interface is refreshed but the frame data is not updated, the frame data is refreshed while the interface is refreshed by reading the decoded frame data played by the video, so that the situation that the decoded frame data displayed in the video layer of the interface is not re-rendered is reduced, the problems of blockage or ghost and the like of the picture displayed by the interface can be avoided, the picture quality of the video playing in the interface is improved, and good experience is brought to a user.
Drawings
Fig. 1 is a schematic structural diagram of a module for rendering frame data required for video playing according to the prior art;
fig. 2 is a flowchart of a method for rendering during a video playing process according to an embodiment of the present invention;
fig. 3 is a schematic process diagram of multi-core rendering in a video playing process according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating an example of a rendering method during a video playing process according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a superimposed video window provided by the prior art;
fig. 6 is a schematic diagram of a video layer overlay arrangement according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a device for rendering during video playing according to an embodiment of the present invention;
fig. 8 is a schematic diagram illustrating a specific example of a rendering apparatus in a video playing process according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
fig. 10 is a schematic diagram of a specific example of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and examples.
As can be seen from the background art, the main reasons that the image quality of the interface is not high in the video playing process are: the frame data required by video playing is displayed in a video layer of an interface after being decoded and rendered based on a frame time sequence, the decoding and rendering operation of the frame data required by the video playing is triggered when the frame data is refreshed, and the rendering operation of the frame data required by the current video playing cannot be triggered because the frame data is not refreshed when the interface is refreshed. Furthermore, under the conditions that the frame rate of frame data required by the playing of the video is relatively low and the frame interval time is relatively long, frame data refreshing is not performed because no data is updated between two frames, but an event causing interface change may be triggered in the interval, so that the interface is refreshed and the frame data is not refreshed, and the situation that the interface after the interface refreshing covers the frame data in the video layer of the interface which is not rendered and refreshed in time occurs, which may finally cause the problems of image displayed by the interface such as image sticking or image sticking, and reduce the image quality of video playing in the interface.
In order to overcome the problem, the embodiment of the invention triggers a rendering process when the interface is refreshed, and renders the interface according to the interface refreshing related information and the acquired decoding frame data of the video playing. Therefore, when the interface is refreshed but the frame data is not updated, the frame data is refreshed while the interface refreshing is realized by reading the decoded frame data of the video playing, so that the situation that the decoded frame data displayed in the video layer of the interface is not rendered again is reduced, the problems of blockage or ghost and the like of the picture displayed by the interface can be avoided, the picture quality of the video playing in the interface is improved, and good experience is brought to a user.
In the embodiment of the invention, when the interface is refreshed, the interface is rendered according to the interface refreshing related information and the acquired decoding frame data of the video playing, the rendering process of the frame data required by the video playing is separated from the decoding process, the rendering process of the frame data required by the video playing is realized on the interface layer, and the decoding frame data of the video playing is stored, so that the stored decoding frame data of the video playing can be acquired in time.
Furthermore, the invention can also trigger the rendering process when the frame data is refreshed in real time, and the rendering process is also realized at the interface layer, namely, the stored decoded frame data of the video playing is obtained, and the video layer of the interface is rendered according to the obtained decoded frame data of the video playing.
The embodiment of the invention separates the rendering process and the decoding process of the frame data required by video playing, actually controls the rendering process on the interface, and particularly sets the sub-thread implementation of the rendering process under the interface main thread.
Fig. 2 is a flowchart of a method for rendering during video playing according to an embodiment of the present invention, which includes the following specific steps:
Here, interface refreshing refers to an operation of redrawing or updating an interface, such as adding or deleting a visible interface element, performed when an interface element sends a change; the interface element is moved, resized or changed in position; interface elements go from occluded to unoccluded states, and so on. The redrawing or updating operation of the interface sometimes affects the picture change of the video played by the interface including the video layer, and sometimes does not affect the picture change of the video played by the interface including the video layer. When the influence is caused, the process according to fig. 2 needs to be executed, and in the process of rendering the interface once, the related information and the decoded frame data are refreshed according to the interface at the same time; when not affected, the process described in fig. 2 may be performed, or the process described in fig. 2 may not be performed, but only the interface is rendered according to the interface refresh related information. When the interface refreshing event is triggered, the interface control layer in the computer system can obtain an interface refreshing command.
To implement step 201 of fig. 2, the decoded frame data of the video playing may be stored in advance. Before the reading of the decoded frame data of the video playback, the method further comprises:
acquiring a video playing instruction, acquiring frame data required by a video playing process frame by frame, decoding to obtain decoded frame data, and storing the decoded frame data;
the reading the decoded frame data of the video playing comprises: reading the stored decoded frame data.
The embodiment of the invention can be provided with a buffer area for storing the decoding frame data played by the video, and the stored decoding frame data played by the video can be multiple frames or one frame. When the decoded frame data of the multi-frame video playing is stored, the decoded frame data of the first several frames or/and the decoded frame data of the last several frames of the current frame of the video playing may be stored, and when the interface is rendered, any one frame of decoded frame data may also be read from the decoded frame data, or one frame of decoded frame data related to the current video playing frame, such as reading the decoded frame data of the previous frame of the current video playing or reading the decoded frame data of the next frame of the current video playing, may be rendered. In this case, the problem of image display on the interface that is jammed or ghost image, etc. can be overcome, but the video played in the video layer of the interface may have frame skipping or frame returning, which causes the problem of discontinuous playing.
Preferably, in order to overcome the problem that a picture displayed on the interface is jammed or ghost, and the like, and to make the video played in the video layer of the interface continuous, the decoded frame data shown in fig. 2 is the decoded frame data played in the current video, and the decoded frame data played in the current video is also used when the interface is rendered. In this case, the buffer may still store the decoded frame data of the video playback in multiple frames, and the data of the multiple frames includes the decoded frame data of the current video playback to be read. Preferably, in order to reduce the storage space of the buffer, the decoded frame data of the video playing stored in the buffer is a frame, and the frame data is the decoded frame data of the current video playing. And replacing the stored decoded frame data of the video playing once the decoded frame data of the subsequent video playing arrives, and ensuring that the decoded frame data of the current video playing is stored in the buffer area.
In the preferred embodiment, the data frames of the video playing are continuously changed along with the frame sequence of the video playing, each time the decoded frame data of one frame is stored, when a new decoded frame data of one frame is to be played, the stored decoded frame data of the previous frame is replaced by the new decoded frame data of one frame, so as to ensure that the decoded frame data of the current video playing stored in the buffer area is dynamically updated. Specifically, the storing the decoded frame data of the current video playing includes:
and acquiring the multi-frame decoding frame data frame by frame according to the frame sequence of video playing, storing the acquired decoding frame data frame by frame in a buffer area as the decoding frame data of the current video playing, and covering the decoding frame data of the current video playing which is cached in the buffer area. When the decoding frame data of the playing cached in the cache region is read, the decoding frame data of the current video playing is read, so that the storage space of the cache region is saved, the interface frame data of the current video playing is ensured to be adopted during the rendering of a subsequent interface, and the occurrence of frame skipping or frame returning of the video played by the video layer of the interface is reduced.
In this embodiment of the present invention, the method illustrated in fig. 2 further includes:
stopping video playing and emptying the stored decoded frame data;
emptying the rendering in the video layer of the interface.
Specifically, when the video playing is stopped, the decoding frame data stored in the buffer area and the memory resource to be rendered are emptied, so as to save the storage resource of the buffer area and the memory resource used by the device during rendering, and prepare for the subsequent device to perform video playing in the video layer of the interface.
The rendering of the video playing in the embodiment of the present invention adopts a synchronous refresh rendering mode, which needs to be kept consistent with interface refresh by interface control refresh, and in order to maximize the rendering on the interface, a multi-core rendering mode is adopted, a rendering thread is set, the rendering thread is used as a sub-thread of a main thread of the video playing interface, the main thread of the video playing interface is blocked in the rendering process, and the rendering is performed by using the sub-thread mode, that is, the rendering is performed by using the set rendering thread. In order to avoid the change of rendering related information caused by the influence of new changes of the interface brought by animation or other events, the main thread of the video playing interface needs to be blocked in the rendering process, and the rendering thread reads the decoding frame data of the video playing and acquires the current interface refreshing related information. The method specifically comprises the following steps: under the condition of obtaining an interface refreshing instruction, blocking a main thread of a video playing interface, reading decoded frame data of video playing by a sub-thread of the main thread of the video playing interface and obtaining interface refreshing related information, and ending the blocking of the main thread of the video playing interface; and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
Before the sub-thread performs interface rendering, the main thread of the video playing interface is blocked, the decoding frame data of the video playing is ensured to be up-to-date and can be acquired by the sub-thread, when the sub-thread reads the decoding frame data of the video playing and acquires interface refreshing related information, the main thread of the video playing interface is stopped being blocked, the main thread of the video playing interface is continuously executed, and monitoring and corresponding control of the video playing interface are achieved.
When the video playing interface main thread executes, animation or other events appearing in the interface are monitored, and the events may cause a new interface rendering process, or may not affect the interface change, without triggering the interface rendering process.
Fig. 3 is a schematic process diagram of multi-core rendering in a video playing process, shown in the figure, including a video playing interface main thread and a rendering thread, where the rendering thread is a sub-thread of the video playing interface main thread, and a detailed description is given by taking an example of processing an interface refresh event, where the detailed steps include:
in this step, the video playing interface is blocked and executed under the control of a processor of the device, or after the main thread of the video playing interface obtains an interface refreshing instruction, the rendering thread is notified, and the main thread is unblocked by the rendering thread;
in this step, the main thread of the video playing interface is stopped and executed under the control of a processor of the device, or the main thread is stopped after the sub-thread reads the decoded frame data of the video playing and the interface refreshing related information;
in this step, if there is still decoded frame data of the previous video playing in the rendering thread before drawing new decoded frame data, the decoded frame data of the previous video playing is refreshed by using the read decoded frame data of the video playing;
it should be noted that step 307 may be executed after step 305 and before step 306;
308, rendering the interface by the rendering thread according to the drawn new decoding frame data and the interface refreshing related information;
The execution of step 309 and the execution of steps 307-308 in FIG. 3 may be separate, and they do not affect each other and have no sequential points.
Step 308 in fig. 3 specifically includes: rendering the drawn new decoding frame data in a video layer of the interface by the rendering thread; meanwhile, the rendering thread renders the interface according to the acquired interface refreshing related information, and finally the whole rendering process is completed. It should be noted that, the rendering thread may not execute step 307, and that in step 308, the rendering thread renders the interface according to the read decoded frame data and the interface refresh related information, specifically, the rendering thread renders the read decoded frame data in the video layer of the interface, and meanwhile, the rendering thread renders the interface according to the obtained interface refresh related information.
The process of fig. 3 is an example of processing an interface refresh event, and details a process of multi-core rendering in a video playing process. Further, the process of fig. 3 may also be applied to a data frame refresh event, and a multi-core rendering interface is adopted, in which case, step 301 is changed to: monitoring a data frame refreshing event by a main thread of a video playing interface; step 302 changes to: a video playing main thread obtains a data frame refreshing instruction; the subsequent steps are the same as the process of fig. 3.
When the process of fig. 3 is applied to a data frame refresh event, if there is no interface change involved in the process, the interface refresh related information is not changed, so that the decoded frame data of the video playback can be only relied upon when rendering the interface. Of course, even if interface change is not involved in the process, the interface refreshing related information can be obtained in the manner of fig. 3, and the interface refreshing related information and the decoded frame data of the video playing are read when the interface is rendered.
In general, a main thread of a video playing interface controls a video playing process, and when decoded frame data of current video playing changes or an interface refreshing instruction is obtained, rendering updating is triggered, the main thread of the video playing interface is blocked, and the video playing interface jumps to a rendering thread; after the main thread of the video playing interface is stopped being blocked, the main thread of the video playing interface is continuously executed, and triggering response, animation processing and the like of other events of the interface are carried out.
When the main thread of the video playing interface is blocked, the rendering thread starts processing of the decoded frame data of a new frame or a plurality of frames, namely reading the decoded frame data of the video playing and obtaining the interface refreshing related information, and after the processing is finished, the main thread of the video playing interface is stopped to be blocked, so that the main thread of the video playing interface is continuously executed. The rendering thread starts to perform drawing of new decoded frame data, the new decoded data are drawn according to the obtained decoded frame data of the video playing, or if the decoded frame data exist in the rendering thread before, the obtained decoded frame data of the video playing are adopted to refresh the previous decoded frame data; and the rendering thread refreshes the interface according to the drawn new decoding frame data and the interface refreshing related information. Specifically, the rendering thread renders the drawn new decoded frame data in the video layer of the interface; meanwhile, the rendering thread renders the interface according to the acquired interface refreshing related information, and finally the whole rendering process is completed.
The embodiment of the invention ensures the synchronization of interface refreshing and frame data refreshing in the video playing process, realizes the control of video playing and the rendering control in the interface layer, needs to render on the interface again when the frame data refreshing caused by the change of the decoded frame data required by video playing or the interface refreshing triggered by the operation of amplifying or dragging the interface by a user and the like, and is matched with the decoded frame data storage strategy of the played video in the rendering process, so that the video layer of the interface can be refreshed in time, and the problem of ghost can be effectively solved.
Furthermore, in the embodiment of the present invention, the control and rendering control of the video playing is implemented in the interface layer, and the control and rendering control of the video playing can be implemented by the set video presentation control unit, where the video presentation control unit controls the interface to perform the video playing and the rendering on the interface, and the whole control process is as shown in fig. 4.
Fig. 4 is a schematic diagram of an example of a rendering method in a video playing process according to an embodiment of the present invention, which specifically includes:
in this step, the video display control unit can control the video playing of the interface;
in this step, the video presentation control unit may initiate a request for fetching a stream to a device or a streaming media service, and control the stream fetching unit to obtain frame data required by the video playing process frame by frame;
in the step, after the stream taking unit acquires the frame data, the frame data is sent to a decoding unit for decoding, and the decoding unit decodes the frame data to obtain decoded frame data;
here, after the decoding unit completes decoding, the video presentation control unit may be notified of the related data, and the video presentation control unit controls the rendering unit to store the decoded frame data of the video playing; the decoding unit may also be configured to transmit the decoded frame data to the rendering unit after the decoding unit decodes the decoded frame data, and the rendering unit stores the decoded frame data after obtaining the decoded frame data, so that the decoded frame data can be read by a subsequent rendering unit when rendering an interface. After step 405 is executed, step 408 is executed;
in this step, the video display control unit can obtain an interface refreshing instruction after recognizing an interface refreshing event triggered by a user;
in this step, the video presentation control unit may notify the rendering unit to execute the rendering process, and jump to step 408, and the rendering unit continues to execute the rendering process;
in this step, under the condition that the decoded frame data has an update but an interface refresh instruction is not obtained (i.e. after 401 and 405 are executed, 408 is executed), after the decoded frame data is stored, the rendering unit may render the video layer of the interface according to the latest decoded frame data, or may refresh the rendering interface according to the latest decoded frame data and the acquired interface; under the condition of obtaining an interface refreshing instruction (namely after the step of executing 407 and 406, the step of executing 408), after receiving the notification of the video display control unit, the rendering unit refreshes the interface according to the read decoded frame data played by the video and the obtained interface;
in this step, the rendering interface is implemented by a rendering unit, and the step of rendering the interface according to the read decoded frame data of the video playing and the obtained interface refresh related information may be performed in a specific process: the rendering unit reads the decoded frame data of the video playing and renders the decoded frame data of the video playing to the video layer of the interface; meanwhile, interface refreshing related information is also acquired (under the condition that the interface refreshing related information is updated or is not updated), and the interface is rendered according to the interface refreshing related information.
As can be seen from the process of fig. 4, the video playing, the data frame refreshing and the interface refreshing can be controlled and realized by the set video display control unit. The processes of steps 401 to 404 in fig. 4 are the same as those in the background art, but in step 405 and thereafter, the method for storing decoded frame data of video playback, the method for refreshing data frames, and the method for refreshing interfaces are the key points of the present application, and can be controlled and implemented by the video presentation control unit.
Furthermore, the embodiment of the invention can calibrate the set area on the interface or display the transparent effect or animation effect on the interface in a mode of overlapping windows. Generally, when an interface is calibrated for a set area, one mode is to perform rendering of lines, frames or pictures on a video layer of a current video playing, and this mode needs to perform rendering of the video layer of the interface on decoded frame data of the current video playing, and the drawing process is complex and difficult to implement; another way is as shown in fig. 5, and fig. 5 is a schematic view of an overlay video window provided in the prior art, where a transparent layer 102 is added before a video layer 101, and an area-of-interest window 103 is marked in the transparent layer 102, where the area-of-interest window 103 is a sub-window of the transparent layer 102, so as to achieve the purpose of marking an area-of-interest and an animation effect. In this way, the transparent layer 102 needs to monitor the information of the video layer 101, including the size and position change, so as to adjust the layer information itself, otherwise, the transparent layer may be misaligned with the video layer 101, and the problem of delay in event monitoring and dynamic adjustment may occur, and when the transparent layer 102 moves or changes the size of the window, the problem of jitter in the calibration area may occur, and the implementation is complicated. Under the circumstance, when the background technology plays the video in the video layer of the interface, because the interface refreshing and the frame data refreshing are two sets of processing mechanisms, the refreshing can interfere with each other, when the interface is applied to some aspects such as video overlapping, the overlapping of the transparent layer can not be effectively realized in the picture displayed by the interface, and the problems of asynchronous moving of the transparent layer and the like are easy to occur. Such as: the problem that the region of interest of a user cannot be displayed in a transparent layer overlapping mode when a video layer of an interface is played.
By adopting the embodiment of the invention, the problems can be solved. The method provided by the embodiment of the invention also comprises the following steps:
setting a sub-control in a video layer of the interface to superpose a set window, wherein the window is in transparent presentation or animation presentation on the interface;
the obtaining of the interface refreshing instruction comprises: and triggering an interface refreshing instruction under the condition that the window is changed.
Therefore, interface refreshing can be triggered, and transparent effect, animation effect and the like can be realized simply by adopting a window overlapping mode.
By adopting the above method, the calibrated window 502, where the calibrated window 502 is an area of interest, may be directly superimposed on the video layer 101 of the interface, that is, the calibrated window 502 is superimposed as a sub-control of the video layer 101 of the interface, as shown in fig. 6, fig. 6 is a schematic diagram of video layer superimposition setting provided in the embodiment of the present invention, the calibrated window 502 is superimposed on the video layer 101 of the interface, and the calibrated window 502 may present a transparent effect or an animation effect on the video layer 101 of the interface. Under the condition that the calibrated window 502 is changed, interface refreshing is triggered to carry out an interface refreshing process, so that the calibrated window 502 can be presented on the video layer 101 of the interface without distortion, and the problems of asynchronization and ineffective superposition are reduced. Therefore, the transparent effect, the animation effect and the like can be realized by simply adopting a superposed window mode.
That is to say, after the video layer of the interface overlaps the window, when the window is changed, the interface refresh event according to the embodiment of the present invention is triggered, so that the interface is rendered again according to the decoded frame data of the played video and according to the interface refresh related information, and the problems of window jitter and the like do not occur.
An embodiment of the present invention further provides a device for rendering during a video playing process, as shown in fig. 7, including: a video presentation control unit and a rendering unit, wherein,
the video display control unit is used for obtaining an interface refreshing instruction;
the rendering unit is used for reading the played decoded frame data; and rendering an interface according to the interface refreshing related information and the decoding frame data.
Specifically, the video presentation control unit may be configured to obtain an interface refresh instruction, and send a rendering control instruction to the rendering unit;
the rendering unit may be configured to obtain the rendering control instruction from the video presentation control unit, and read played decoded frame data; and rendering an interface according to the interface refreshing related information and the decoding frame data.
Therefore, the device provided by the embodiment of the invention can render the interface according to the interface refreshing related information and the acquired decoding frame data of the video playing when the interface is refreshed. When the interface is refreshed but the frame data is not updated, the frame data is refreshed while the interface refreshing is realized by reading the decoded frame data played by the video, so that the condition that the decoded frame data displayed in the video layer of the interface is not re-rendered is reduced, and the problems of jamming, ghost and the like of the picture displayed by the interface can be avoided.
As shown in fig. 8, fig. 8 is a schematic diagram of a specific example of the rendering apparatus in the video playing process according to the embodiment of the present invention, and preferably, the apparatus further includes a stream fetching unit and a decoding unit, wherein,
the video display control unit is also used for obtaining a video playing instruction;
the stream taking unit is used for acquiring frame data required by a video playing process frame by frame;
the decoding unit is used for decoding the frame data acquired by the stream taking unit to obtain decoded frame data;
the rendering unit is further configured to store the decoded frame data.
Specifically, the video presentation control unit may be further configured to obtain a video playing instruction, and send a control instruction for obtaining frame data required by a video playing process to the stream fetching unit;
the stream fetching unit may be configured to obtain, from the video presentation control unit, the control instruction for obtaining frame data required by the video playing process, obtain, frame by frame, frame data required by the video playing process, and send the frame data to the decoding unit;
the decoding unit may be configured to decode frame data obtained from the stream fetching unit to obtain decoded frame data, and send the decoded frame data to the rendering unit under the control of the video presentation control unit;
the rendering unit may be further configured to obtain the decoded frame data, and store the decoded frame data.
Here, the rendering unit stores the decoded frame data, may store one or more frames of decoded frame data, may store, when storing decoded frame data of a multi-frame video playback, previous frames of decoded frame data and/or next frames of decoded frame data of a current frame of the video playback, and may also read any one frame of decoded frame data from the decoded frame data when the subsequent rendering unit is rendering an interface, or may read one frame of decoded frame data related to the current video playback frame, such as reading previous frames of decoded frame data of the current video playback or reading next frames of decoded frame data of the current video playback, and perform rendering. In this case, the problem of image display on the interface that is jammed or ghost image, etc. can be overcome, but the video played in the video layer of the interface may have frame skipping or frame returning, which causes the problem of discontinuous playing.
Preferably, the decoded frame data is decoded frame data of a current video playing.
Specifically, the rendering unit may further include a buffering subunit and a rendering subunit, wherein,
the buffer subunit is configured to store the decoded frame data;
the rendering subunit is configured to read the currently played decoded frame data cached by the caching subunit, and render an interface according to interface refresh related information and the currently played decoded frame data.
In order to overcome the problems of image jamming or image sticking and the like of the interface display, and to make the video played in the video layer of the interface continuous, the decoded frame data is the decoded frame data of the current video playing, and the decoded frame data is also used in the interface rendering. In this case, the decoded frame data of the video playback stored in the buffer subunit may still be multiple frames, and the multiple frames of data include the decoded frame data of the current video playback to be read. Preferably, in order to reduce the storage space of the buffer, the decoded frame data of the video playing stored by the buffer subunit is a frame, and the frame data is the decoded frame data of the current video playing. And replacing the stored decoded frame data of the video playing once the decoded frame data of the subsequent video playing arrives, and ensuring that the decoded frame data of the current video playing is stored in the buffer area.
Preferably, the rendering unit is specifically configured to block a main thread of a video playing interface when the interface refresh instruction is obtained, and end the blocking of the main thread of the video playing interface after a sub-thread of the main thread of the video playing interface reads decoded frame data of video playing and obtains interface refresh related information; and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
In the embodiment of the present invention, preferably, the video presentation control unit is further configured to stop video playing and empty the stored decoded frame data;
the rendering unit is further configured to empty the rendering in the video layer of the interface.
In the embodiment of the present invention, preferably, the video presentation control unit is specifically configured to set a sub-control in a video layer of the interface to superimpose the set window, where the window is transparently presented or animated on the interface; and obtaining an interface refreshing instruction under the condition that the window is changed.
An embodiment of the present invention further provides an electronic device, as shown in fig. 9, fig. 9 is a schematic structural diagram of an exemplary electronic device provided in an embodiment of the present invention, where a device for implementing a method in an embodiment of the present application may be integrated. Specifically, the method comprises the following steps:
the electronic device 20 may include a processor 810 of one or more processing cores, a memory 830 of one or more computer-readable storage media, and a computer program stored on the memory 830 and executable on the processor 810. When the program of the memory is executed, the method for rendering in the video playing process described in the above embodiment may be implemented.
Fig. 10 is a schematic diagram of a specific example of an electronic device according to an embodiment of the present invention, as shown in the figure, the electronic device 20 may further include: a communication module 820, configured to perform communication connection (wired or wireless) with other entities, and acquire video playing data on an interface; the memory 830 of the computer-readable storage medium may store data and information, a portion of which, when executed by the processor 810, causes the processor 810 to perform the methods as described in the previous embodiments.
In addition, the electronic device 20 may further include a display module 840, and when the processor 810 executes the method according to the foregoing embodiments, an interface is displayed on the display module 840. The electronic device 20 may further include: a magnetic disk medium 850 that can be used to store video playback data; the buffer medium 860 may be used to store at least decoded frame data of video playback, interface refresh related information, and the like.
Also provided in another embodiment is a non-transitory computer readable storage medium that may store instructions, a portion of which, when executed by a processor, cause the processor to perform a method as described in the preceding embodiments.
There is also provided in yet another embodiment a computer program product containing instructions which, when run on a computer, may cause the computer to perform the method as described in the preceding embodiments.
The method and the device solve the problem of interface ghost caused by asynchronous interface refreshing and frame data refreshing in the video playing process and the problem that the transparent overlay window of the video layer of the interface cannot be effectively finished, and realize the synchronization of the interface refreshing and the frame data refreshing and the multi-core rendering with maximized performance; under the condition that the interface changes but no video playing decoding frame data changes, performing a video playing frame data storage strategy on the interface to provide stored video playing decoding frame data during interface rendering; and an additional window can be automatically and conveniently superposed on the video layer of the interface through the rendering and refreshing of the interface so as to achieve transparent video superposition.
The embodiment of the invention solves the problems that the image of the interface has ghost in the existing rendering process under the conditions of lower frame rate and longer frame interval time, and the problem of window overlapping is not easy to carry out because the decoding rendering and the interface refreshing of frame data are two systems. The embodiment of the invention further adopts a multi-core rendering mode, so that the rendering efficiency reaches the same performance index as the existing rendering.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (11)
1. A method for rendering during video playing, comprising:
obtaining an interface refreshing instruction, and reading decoded frame data played by a video;
and rendering an interface according to the interface refreshing related information and the decoding frame data.
2. The method of claim 1, wherein prior to said reading decoded frame data of a video playback, the method further comprises:
acquiring a video playing instruction, acquiring frame data required by a video playing process frame by frame, decoding to obtain decoded frame data, and storing the decoded frame data;
the reading the decoded frame data of the video playing comprises: reading the stored decoded frame data.
3. The method of claim 1 or 2, wherein the decoded frame data is decoded frame data of a current video playback.
4. The method of claim 2, wherein the method further comprises:
stopping video playing and emptying the stored decoded frame data;
emptying the rendering in the video layer of the interface.
5. The method of claim 1 or 2, wherein the reading of decoded frame data of a video playback; rendering an interface according to the interface refresh related information and the decoded frame data includes:
under the condition of obtaining an interface refreshing instruction, a main thread of a video playing interface is blocked, and after a sub-thread of the main thread of the video playing interface reads decoded frame data of video playing and obtains interface refreshing related information, the main thread of the video playing interface is stopped being blocked;
and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
6. The method of claim 1 or 2, wherein the method further comprises:
setting a sub-control in a video layer of the interface to superpose a set window, wherein the window is in transparent presentation or animation presentation on the interface;
the obtaining of the interface refreshing instruction comprises: and obtaining an interface refreshing instruction under the condition that the window is changed.
7. An apparatus for rendering during video playback, comprising: a video presentation control unit and a rendering unit, wherein,
the video display control unit is used for obtaining an interface refreshing instruction;
the rendering unit is used for reading the played decoded frame data; and rendering an interface according to the interface refreshing related information and the decoding frame data.
8. The apparatus of claim 7, wherein the apparatus further comprises a stream fetching unit and a decoding unit, wherein,
the video display control unit is also used for obtaining a video playing instruction;
the stream taking unit is used for acquiring frame data required by a video playing process frame by frame;
the decoding unit is used for decoding the frame data acquired by the stream taking unit to obtain decoded frame data;
the rendering unit is further configured to store the decoded frame data.
9. The apparatus according to claim 7 or 8, wherein the decoded frame data is decoded frame data of a current video play.
10. The apparatus of claim 7 or 8,
the rendering unit is specifically used for blocking a main thread of a video playing interface under the condition that an interface refreshing instruction is obtained, and after the sub-thread of the main thread of the video playing interface reads the decoding frame data of video playing and obtains interface refreshing related information, the video playing interface main thread is stopped being blocked; and refreshing the relevant information and the decoding frame data rendering interface by the sub thread according to the interface.
11. An electronic device, comprising: a processor of one or more processing cores, a memory of one or more computer readable storage media, and a computer program stored on the memory and executable on the processor, which when executed, implement the method of rendering during video playback as recited in claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010590739.8A CN111654740B (en) | 2020-06-24 | 2020-06-24 | Rendering method and device in video playing process and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010590739.8A CN111654740B (en) | 2020-06-24 | 2020-06-24 | Rendering method and device in video playing process and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111654740A true CN111654740A (en) | 2020-09-11 |
| CN111654740B CN111654740B (en) | 2022-12-16 |
Family
ID=72351833
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010590739.8A Active CN111654740B (en) | 2020-06-24 | 2020-06-24 | Rendering method and device in video playing process and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111654740B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360708A (en) * | 2021-05-31 | 2021-09-07 | 北京达佳互联信息技术有限公司 | Video playing method and device, electronic equipment and storage medium |
| CN114115688A (en) * | 2021-11-23 | 2022-03-01 | 北京鲸鲮信息系统技术有限公司 | Method and device for suppressing display jitter, electronic equipment and storage medium |
| CN113360708B (en) * | 2021-05-31 | 2024-05-10 | 北京达佳互联信息技术有限公司 | Video playing method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130057519A1 (en) * | 2011-09-01 | 2013-03-07 | Sharp Laboratories Of America, Inc. | Display refresh system |
| CN109547838A (en) * | 2018-12-06 | 2019-03-29 | 青岛海信传媒网络技术有限公司 | The processing method and processing device of video window |
| CN111163345A (en) * | 2018-11-07 | 2020-05-15 | 杭州海康威视系统技术有限公司 | Image rendering method and device |
-
2020
- 2020-06-24 CN CN202010590739.8A patent/CN111654740B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130057519A1 (en) * | 2011-09-01 | 2013-03-07 | Sharp Laboratories Of America, Inc. | Display refresh system |
| CN111163345A (en) * | 2018-11-07 | 2020-05-15 | 杭州海康威视系统技术有限公司 | Image rendering method and device |
| CN109547838A (en) * | 2018-12-06 | 2019-03-29 | 青岛海信传媒网络技术有限公司 | The processing method and processing device of video window |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113360708A (en) * | 2021-05-31 | 2021-09-07 | 北京达佳互联信息技术有限公司 | Video playing method and device, electronic equipment and storage medium |
| CN113360708B (en) * | 2021-05-31 | 2024-05-10 | 北京达佳互联信息技术有限公司 | Video playing method and device, electronic equipment and storage medium |
| CN114115688A (en) * | 2021-11-23 | 2022-03-01 | 北京鲸鲮信息系统技术有限公司 | Method and device for suppressing display jitter, electronic equipment and storage medium |
| CN114115688B (en) * | 2021-11-23 | 2023-08-01 | 北京字节跳动网络技术有限公司 | Method and device for inhibiting display jitter, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111654740B (en) | 2022-12-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109547838B (en) | Video window processing method and device | |
| CN108574806B (en) | Video playing method and device | |
| EP4178188A1 (en) | Landscape interaction method and apparatus, electronic device, and storage medium | |
| US8713472B2 (en) | Display apparatus, a method for a display control, and program | |
| JP6867501B2 (en) | Predicted bitrate selection for 360 video streaming | |
| WO2021244666A1 (en) | Video playing control method and apparatus, and computer device and storage medium | |
| CN111526404B (en) | Single frame playing method, video playing device, monitoring equipment and storage medium | |
| US9466089B2 (en) | Apparatus and method for combining video frame and graphics frame | |
| US20220249945A1 (en) | Picture update method and apparatus, computer device, and storage medium | |
| CN111654740B (en) | Rendering method and device in video playing process and electronic equipment | |
| KR20220143442A (en) | Method and apparatus for timed and event triggered updates in a scene | |
| KR101404489B1 (en) | System and method for implementing real-time animation using multi-thread | |
| CN108521604B (en) | Multi-screen display method and device for redirecting video | |
| CN112804410A (en) | Multi-display-screen synchronous display method and device, video processing equipment and storage medium | |
| US20200401496A1 (en) | Method and device of displaying video comments, computing device, and readable storage medium | |
| US20230137492A1 (en) | Landscape interaction method and apparatus, electronic device, and storage medium | |
| CN115361579A (en) | Video transmitting and displaying method and device, electronic equipment and storage medium | |
| US20220223098A1 (en) | Frame replay for variable rate refresh display | |
| CN113542765A (en) | Media data jumping continuous playing method and display equipment | |
| CN113709574A (en) | Video screenshot method and device, electronic equipment and computer-readable storage medium | |
| EP4262216A1 (en) | Encoding method, electronic device, communication system, storage medium, and program product | |
| CN113342516A (en) | Data processing method and device, terminal and storage medium | |
| JP6226616B2 (en) | Video playback apparatus and video playback method | |
| US8730398B2 (en) | Video output apparatus and control method therefor, and non-transitory recording (storing) medium that records program | |
| EP4124051A1 (en) | Method for controlling video playing, apparatus, and computer device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |