WO2022188753A1

WO2022188753A1 - Video frame caching method, and device

Info

Publication number: WO2022188753A1
Application number: PCT/CN2022/079575
Authority: WO
Inventors: 罗小伟; 郭春磊; 李�荣
Original assignee: 展讯通信（上海）有限公司
Priority date: 2021-03-08
Filing date: 2022-03-07
Publication date: 2022-09-15
Also published as: CN113015003A; CN113015003B

Abstract

The present application relates to the technical field of video display, and in particular to a video frame caching method, and a device. The method comprises: determining a shared cache, wherein the shared cache comprises a plurality of frame caches, and a state identifier is set for each frame cache; writing reconstructed frames, which are decoded and output, into frame caches for which a writing-available identifier is set, and if the reconstructed frames have reference frame features, setting a reference frame identifier for corresponding frame caches; determining, from among cached reconstructed frames, a frame to be displayed, and setting a to-be-displayed identifier for the frame cache where the frame to be displayed is located; and displaying the frame to be displayed that is in the frame cache for which the to-be-displayed identifier is set, and if there no longer is a reference frame feature after the frame to be displayed is displayed, setting a writing-available identifier for the corresponding frame cache. By means of the solution in the embodiments of the present invention, a video output delay can be decreased while relatively little system memory is occupied.

Description

Video frame buffer method and device

This application claims the priority of the Chinese patent application with the application number 202110249758.9 and the application title "Video Frame Buffering Method and Apparatus" filed with the China Patent Office on March 08, 2021, the entire contents of which are incorporated into this application by reference.

technical field

The present application relates to the technical field of video display, and in particular, to a video frame buffering method and device.

Background technique

In the audio and video playback scenario, the processing of the audio and video streams involves the decoding of the video streams and the display of image frames in addition to the demultiplexing of the audio and video streams. That is, the video stream is first reconstructed into image frame data through video decoding, and then the image frame is rendered to the screen through the image display process. Among them, in order to support the normal scheduling of the video stream decoding and image frame display process, the image frame data is usually stored in the frame buffer of the system memory. However, in the current image frame data caching method, there may be decoded reconstructed frames that need to be copied and occupy more memory; or the decoded reconstructed frames need to be delayed output because they need to be used as reference frames for subsequent video frames, resulting in video Caton, etc.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present invention provide a video frame buffering method and device, which can not only avoid the copy operation of the reconstructed frame, but also reduce the delay caused by the need to use the reconstructed frame as a reference frame. Therefore, the solutions of the embodiments of the present invention can reduce the video output delay while occupying a small amount of system memory.

In a first aspect, an embodiment of the present invention provides a video frame buffering method, including: determining a shared buffer, where the shared buffer includes a plurality of frame buffers, and each frame buffer is provided with a status identifier; The frame is written into a frame buffer provided with a writable identifier, and if the reconstructed frame has a reference frame feature, a reference frame identifier is set for the corresponding frame buffer; the frame to be displayed is determined from the reconstructed frame that has been buffered, and the The frame buffer where the frame to be displayed is located is set with a to-be-displayed flag; the to-be-displayed frame in the frame buffer where the to-be-displayed flag is set is displayed, and if the to-be-displayed frame no longer has the reference frame feature after being displayed, the corresponding Framebuffer settings writable flag.

Optionally, the number of frame buffers included in the shared buffer is determined according to the number of reference frames and the number of buffered frames of the video stream to be played.

Optionally, each of the frame buffers is provided with a status identifier, including: each of the frame buffers is provided with a counter, and the value of the counter is used as a status identifier of the corresponding frame buffer. When the state of the reconstructed frame buffered in the frame buffer changes, the corresponding counter value changes.

Optionally, the method further includes: creating a frame buffer queue, where the frame buffer queue is used to store the status identification information of each frame buffer in the shared buffer; writing the reconstructed frame output from the decoding is set with a writable mark. The frame buffer, including: according to the state identification information of the respective frame buffers stored in the frame buffer queue, determine the frame buffer that is provided with the writable mark, wherein, the frame buffer that is provided with the writable mark is used for Write the reconstructed frame of the decoded output.

Optionally, determining the frame to be displayed from the cached reconstructed frame includes: if the reconstructed frame output by the current decoding only has the display feature, then determining the reconstructed frame output by the current decoding as the frame to be displayed, and writing it in the writable frame. On the basis of the input identification, the corresponding frame buffer is set with the identification to be displayed; if the reconstructed frame outputted by the current decoding has both the reference frame feature and the display feature, the reconstructed frame outputted by the current decoding is determined as the frame to be displayed, and the reference frame identification On the basis of the corresponding frame buffer, the to-be-displayed flag is set. Wherein, if the reconstructed frame output by the current decoding has both the reference frame feature and the display feature, the reconstructed frame output by the current decoding may be determined as the frame to be displayed when a preset display condition is satisfied.

Optionally, determining the frame to be displayed from the reconstructed frame that has been buffered, further comprising: in addition to the reconstructed frame output by the current decoding, if the reconstructed frame in at least one frame buffer that has been set with a reference frame identifier no longer has The reference frame features, and has the display feature and has not yet been displayed, then the corresponding reconstructed frame is determined as the frame to be displayed, and the reference frame identifier of the corresponding frame buffer is set as the to-be-displayed identifier; in addition to the reconstructed frame output by the current decoding, if If the reconstructed frame in the at least one frame buffer already set with the reference frame identifier no longer has the reference frame identifier, and has been displayed or has no display feature, the reference frame identifier of the corresponding frame buffer is set as a writable identifier.

Optionally, the method further includes: creating a to-be-displayed queue, where the to-be-displayed queue is used to store to-be-displayed identification information of the to-be-displayed frame. Wherein, only the to-be-displayed identifications of the to-be-displayed frames having the display feature and the to-be-displayed frame having both the display feature and the reference feature are different.

Optionally, the to-be-displayed queue is also used to store timestamp information and number information of the to-be-displayed frame; displaying the to-be-displayed frame includes: according to the to-be-displayed frame timestamp information and number information. , determine the display order of the to-be-displayed frames, and display the to-be-displayed frames according to the display order of the to-be-displayed frames.

Optionally, if the to-be-displayed frame no longer has the reference frame feature after being displayed, setting a writable flag to the corresponding frame buffer, including: if the to-be-displayed frame only has the display feature, then after the to-be-displayed frame is displayed , set the to-be-displayed identification as a writable identification; if the to-be-displayed frame has both display features and reference features, after the to-be-displayed frame is displayed, the to-be-displayed identification is set as the reference frame identification, and then the reference frame identification Set to the ID to be written.

In a second aspect, an embodiment of the present invention provides a terminal device, including: a determination module configured to determine a shared buffer, where the shared buffer includes a plurality of frame buffers, and each frame buffer is provided with a status identifier; a decoding module , for writing the decoded output reconstructed frame into a frame buffer provided with a writable identifier, and if the reconstructed frame has a reference frame feature, then a reference frame identifier is set for the corresponding frame buffer; The frame to be displayed is determined in the reconstructed frame, and the frame buffer where the frame to be displayed is located is set with a to-be-displayed mark; After the frame is displayed, it no longer has the reference frame feature, then the writable flag is set for the corresponding frame buffer.

In a third aspect, an embodiment of the present invention provides a terminal device, including: at least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores a program that can be executed by the processor Program instructions to be invoked by the processor to execute to perform the method of the first aspect or any possible embodiment of the first aspect. .

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, wherein when the program runs, a device on which the computer-readable storage medium is located is controlled to execute the first The method of an aspect or any possible embodiment of the first aspect. .

In the solution of the embodiment of the present invention, the video decoding scheduling and the image display scheduling share the frame buffer, and the video decoding scheduling and the image display scheduling do not need to copy and reconstruct the frame data frequently during the decoding and display process, which saves the system cache space and improves the system performance. Moreover, the solution of the embodiment of the present invention sets a status flag for the reconstructed frame output by decoding, and the reference frame output by decoding can be displayed in real time, and can continue to be used as the reference frame for subsequent video frame decoding after display, thereby reducing the phenomenon of video frame delay and stalling.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a video frame buffering method provided by the related art;

2 is a flowchart of another video frame buffering method provided by the related art;

3 is a flowchart of a video frame buffering method provided by an embodiment of the present invention;

4 is a flowchart of another video frame buffering method provided by an embodiment of the present invention;

5 is a schematic structural diagram of a terminal device provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of another terminal device provided by an embodiment of the present invention.

Detailed ways

In order to better understand the technical solutions of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be clear that the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of this application.

FIG. 1 is a flowchart of a video frame buffering method provided in the related art. As shown in Figure 1, in the video playback scenario, the system schedules create video decoding schedules and image display schedules. Video decoding schedules and image display schedules maintain their internal frame buffers. For example, the frame buffers maintained by video decoding schedules are used to store frames. (0), Frame(1)...Frame(N) reconstructed frames, and the frame buffer maintained by the image display schedule is used for Frame(0), Frame(1)...Frame(M) frames to be displayed. As shown in FIG. 1 , in the video decoding scheduling process, the video decoding engine decodes the video code stream and outputs the reconstructed frame of the video frame image. The video decoding engine writes the output reconstructed frames into the frame buffer it maintains. During the image display scheduling process, the image display engine copies the reconstructed frames to be displayed and buffers them into the frame buffer maintained by the image display engine. The image display engine displays the copied reconstructed frames according to the display order. In the method shown in FIG. 1 , each video frame to be displayed needs to be copied, and the process of copying data is relatively time-consuming, which greatly reduces the performance of the entire system and increases the system bandwidth.

FIG. 2 is a flowchart of another video frame buffering method provided by the related art. As shown in Figure 2, the system scheduling creates video decoding scheduling and image display scheduling, and the frame buffer maintained by the video decoding scheduling is used to store Frame(0), Frame(1)...Frame(N) reconstructed frames. is released after the reconstructed frame is displayed. Since the reconstructed frame will be released after being displayed, and since most of the reconstructed frames need to be used as reference frames for the subsequent decoding process, there will be reconstructed frames that are decoded first and displayed later in the frame buffer. For example, the P1 frame in the video stream I ₀ P ₁ B ₂ P ₃ , the P ₁ frame is decoded before the B ₂ frame, but the P ₁ frame is after the B ₂ frame when _displayed . It can be seen that, for the reconstructed frame that cannot be output and displayed immediately after decoding, it needs to be temporarily stored in the frame buffer until the reconstructed frame is no longer used as a reference frame for the subsequent decoding process. Compared with the method shown in Figure 1, the method shown in Figure 2 does not need to copy the reconstructed frame data during the image display scheduling process, but the decoded reconstructed frame may be retained for a long time, which may cause a large delay or The stuttering phenomenon affects the video playback effect.

In order to solve the related art, due to the occupation of system memory caused by data copying and the delayed and stuck phenomenon caused by temporary storage of reference frames for a long time, the embodiments of the present invention provide a video frame buffering solution. In this solution, the video decoding scheduling and the image display scheduling share the frame buffer, and by setting the status flags for each frame buffer included in the shared buffer, the real-time display of the reconstructed frame can be realized while maintaining the reference frame function of the reconstructed frame. Thereby, the display delay can be reduced.

FIG. 3 is a flowchart of a video frame buffering method provided by an embodiment of the present invention. As shown in Figure 3, in the video playback scenario, the system schedules to create video decoding schedules and image display schedules, and allocates 0, 1, 2...N frame buffers as shared buffers, which can be used by video decoding scheduling and image display. Display scheduling is co-scheduling, and the shared buffer is used to store Frame(0), Frame(1)...Frame(N) reconstructed frames. Different from FIG. 1 and FIG. 2 , in the shared buffer shown in FIG. 3 , each frame buffer is provided with a state identifier, and the state of the reconstructed frame in the corresponding frame buffer can be determined through the state identifier. Optionally, when the state of the reconstructed frame stored in the frame buffer changes, the state identifier corresponding to the frame buffer is also changed. Optionally, the method according to the embodiment of the present invention may further include: creating a frame buffer queue, where the frame buffer queue is used to store address information and state identification information of each frame buffer. According to the address information and status identification information of each frame buffer in the frame buffer queue, the reconstructed frame to be processed can be called. For example, upon initialization, each framebuffer in the shared buffer is set with a writable flag. The video decoding scheduling can randomly select a frame buffer with a writable flag set as the storage location of the reconstructed frame through the frame buffer queue. When the reconstructed frames in one or more frame buffers are determined to be frames to be displayed, the corresponding frame buffers set the to-be-displayed flags. After the frame to be displayed is displayed, a displayed flag can be set for the corresponding frame buffer. Wherein, if the video frame that has been displayed is no longer used as the reference frame of other subsequent video frames, a writable flag may be set for it. If a video frame that has been displayed still needs to be used as a reference frame for other subsequent video frames, a reference frame identifier can be reserved for it. Specifically, based on the shared buffer shown in FIG. 3 , the video decoding engine decodes the video stream, and outputs the reconstructed frame of the video frame image. The video decoding engine writes the decoded output reconstructed frame into the frame buffer set with the writable flag. Optionally, if the written reconstructed frame has the reference frame feature, the reference frame identifier is set for the corresponding frame buffer. During the image display scheduling process, the image display engine determines the frame to be displayed from the reconstructed frames that have been buffered, and sets the to-be-displayed flag in the frame buffer where the frame to be displayed is located. The image display engine reads and displays the reconstructed frame from the frame buffer with the to-be-displayed flag set. Among them, according to the display characteristics and reference frame characteristics of the reconstructed frames, the reconstructed frames can be divided into: reconstructed frames only used for display, reconstructed frames used as both reference frames and display, reconstructed frames only used as reference frames, and neither The reference frame is in turn not used for the reconstructed frame for display. Optionally, the image display scheduling process renders and displays two types of reconstructed frames that are only used for display, and are both used as reference frames and used for display.

Based on the description of FIG. 3 , the processing flow of the video frame buffering method according to the embodiment of the present invention, as shown in FIG. 4 , includes:

101. Determine a shared buffer, where the shared buffer includes a plurality of frame buffers, and each frame buffer is provided with a status identifier. Optionally, a counter may be set for each frame buffer, and the value of the counter is used as a state identifier of the corresponding frame buffer. When the state of the reconstructed frame buffered in the frame buffer changes, the value of the corresponding counter also changes correspondingly. Optionally, in the initial state, each frame buffer is set with a writable flag. For example, the initial value of each counter may take 0. When the value of the counter is 0, it indicates that the corresponding frame buffer is in a writable state.

In some embodiments, after the shared buffer is determined, a frame buffer queue may also be created based on the shared buffer, where the frame buffer queue is used to store address information and state identification information of each frame buffer included in the shared buffer. The video decoding schedule may determine the address of the frame buffer with the writable flag according to the frame buffer queue.

In some embodiments, the number of frame buffers included in the shared buffer may be determined according to the number of reference frames and the number of buffered frames of the video stream to be played. In some embodiments, the number of reference frames is determined as ref_num according to the characteristics of the video stream to be played. Determine the number of buffered frames M according to the number of image frame buffers supported by the image display scheduling. The number of frame buffers included in the frame buffer queue may be: ref_num+M.

102. Write the decoded output reconstructed frame into a frame buffer set with a writable flag, and if the currently written reconstructed frame has a reference frame feature, set a reference frame flag for the corresponding frame buffer.

In some embodiments, the video decoding scheduler may determine the address information of the frame buffer set with the writable identifier according to the status identifier information of each frame buffer stored in the frame buffer queue. The video decoding scheduler can write the reconstructed frame output by the current decoding into the frame buffer set with the writable flag. Wherein, if the reconstructed frame output by the current decoding has the reference frame feature, the writable identifier of the corresponding frame buffer is set as the reference frame identifier.

In some examples, the initial value of each counter may take 0. When the value of the counter is 0, it indicates that the corresponding frame buffer is in a writable state. The video decoding scheduling can write the reconstructed frame output by the current decoding into the frame buffer whose counter value is 0. If the reconstructed frame currently decoded and output has the reference frame feature, the counter for the frame buffer used to store the current reconstructed frame is +1, and the status identification information of the corresponding frame buffer in the frame buffer queue also corresponds to +1.

The fact that the current frame has the reference frame feature means that the current frame is a reference frame for decoding subsequent video frames. For example, when decoding a _Px video frame, the video decoding schedule determines that the _Px video frame is a reference frame of the subsequent _Py video frame, and the video decoding schedule adds 1 to the frame buffered counter corresponding to the _Px video frame. Among them, P _x represents the video frame currently being decoded, and P _y represents any video frame whose decoding time is after P _x . Optionally, P _x and P _y can be two adjacent video frames in decoding order. _Px and _Py may also be spaced apart by multiple other video frames in decoding order. In addition, P _x and P _y may be video frames of the same type, or may be video frames of different types, which will not be described one by one here.

Optionally, the video decoding scheduler may also maintain a reference frame queue. When a reconstructed frame has the reference frame feature, the video decoding scheduler may store the address information and state identification information of the corresponding reconstructed frame in the reference frame queue. Optionally, when any reference frame no longer has the reference frame feature, relevant information such as the address of the reconstructed frame that no longer has the reference frame queue may be deleted from the reference frame queue.

103. Determine the frame to be displayed from the reconstructed frames that have been buffered, and set a to-be-displayed flag in the frame buffer where the frame to be displayed is located. Optionally, the frames to be displayed include reconstructed frames having only display features and both display features and reference features.

When the video decoding scheduling decodes the current frame, if the reconstructed frame output by the current decoding only has the display feature, the video decoding scheduling can determine the reconstructed frame output by the current decoding as the frame to be displayed, and increase the counter of the corresponding reconstructed frame by 1. . If the reconstructed frame output by the current decoding has both the reference frame feature and the display feature, the reconstructed frame output by the current decoding is determined as the frame to be displayed, and the corresponding frame buffer is set on the basis of the reference frame identifier. The counter value is 1 based on +1. Optionally, if the reconstructed frame currently decoded and output has both the reference frame feature and the display feature, it may be determined as the frame to be displayed when a preset condition is met. For example, when the difference between its time stamp and the time stamp or number of the last displayed reconstructed frame is smaller than a certain threshold, it is determined as a frame to be displayed. Of course, for the reconstructed frame output by the current decoding, when it is determined that it has both the reference frame feature and the display feature, it is determined as the frame to be displayed.

Optionally, in this embodiment of the present invention, a queue to be displayed may also be created, and the queue to be displayed is used to store address information and identification information of a frame to be displayed. The image display scheduling can schedule and display each frame to be displayed according to the queue to be displayed.

Further, in addition to the reconstructed frame output by the current decoding, if the reconstructed frame in the at least one frame buffer that has been set with the reference frame identifier no longer has the reference frame feature, and has the display feature and has not yet been displayed, the corresponding reconstructed frame can be The frame to be displayed is determined, and the reference frame identifier of the corresponding frame buffer is set as the identifier to be displayed. Optionally, setting the reference frame identifier of the corresponding frame buffer as the to-be-displayed identifier may be: when the reconstructed frame in the at least one frame buffer is determined as the to-be-displayed frame, the counter of the at least one frame buffer is set in the original reference frame. The frame ID is +1 based on 1, then its value is 2. When the reconstructed frame in the at least one frame buffer is added to the queue to be displayed, and it is determined that it is a special reference frame, the corresponding counter takes a value of -1, and the counter takes a value of 1 at this time. Among them, the special reference frame can be considered as: the original reference frame, at this time, it no longer has the reference frame feature, and has the display feature and is not displayed. Optionally, a special reference frame identifier may also be set for the special reference frame. The special reference frame identifier may not be reflected by the value of the counter, for example, the corresponding reconstructed frame may be marked with the special reference frame identifier in the queue to be displayed.

Further, in addition to the reconstructed frame output by the current decoding, if the reconstructed frame in at least one frame buffer that has been set with the reference frame identifier no longer has the reference frame feature, and has been displayed or does not have the display feature, the corresponding frame buffer The reference frame flag is set to writable flag. For example, the counter of the at least one frame buffer may take a value of -1, and the value of the counter after -1 may take a value of 0, that is, a writable state. In a specific example, the video frame currently being decoded is a _Px video frame, and the _Pz video frame is a reference frame of the _Px video frame. After the P _x video frame is decoded, the P _z video frame is no longer a reference frame of other video frames, and the P _z video frame no longer has the reference frame feature. And if the P _z video frame has been displayed or the P _z video frame is a frame not to be displayed, the counter of the frame buffer where the P _z video frame is located can be set to -1. Wherein, if the counter value of the frame buffer where the P _z video frame is located is 1, the value after -1 is 0, indicating that the frame buffer where the P _z video frame is located can write a new reconstructed frame.

Optionally, setting the to-be-displayed flag in the frame buffer where the frame to be displayed is located may be the counter of the frame buffer corresponding to the to-be-displayed frame +1. Wherein, if the determined frame to be displayed only has the display feature, the to-be-displayed flag is set for the frame buffer where the to-be-displayed frame is located on the basis of the writable flag. If the to-be-displayed frame has both the display feature and the reference feature, the to-be-displayed identifier is set for the frame buffer where the to-be-displayed frame is located on the basis of the reference frame identifier. Among them, when the special reference frame is determined to be the frame to be displayed, the counter is +1 based on the reference frame identification, that is, the value is 2; when it is added to the queue to be displayed, the counter is -1, that is, the value is 1 .

104. Display the to-be-displayed frame in the frame buffer set with the to-be-displayed flag. If the to-be-displayed frame no longer has the reference frame feature after being displayed, set a writable flag to the corresponding frame buffer.

The image display scheduler stores the frame buffer address and counter value of the frame to be displayed in the queue to be displayed. Optionally, timestamp information and serial number information of frames to be displayed may also be stored in the queue to be displayed. The image display scheduling may determine the display order of the to-be-displayed frames according to the timestamp information and the serial number information of the to-be-displayed frames, and render and display the to-be-displayed frames according to the display order of the to-be-displayed frames. Optionally, the image display schedule may be displayed according to the current time stamp distance from far to near, and displayed in the sequence of the numbers of the frames to be displayed. Optionally, after the to-be-displayed frame is displayed by the image display schedule, its counter is set to -1. Wherein, if the to-be-displayed frame only has display features, after the to-be-displayed frame is displayed, the value of its counter-1 becomes 0, and the corresponding frame buffer is in a writable state. If the frame to be displayed has both the display feature and the reference frame feature, after the frame to be displayed is displayed, the value of its counter-1 becomes 1. For a special reference frame, the value of counter-1 becomes 0 after it is displayed, and the corresponding frame buffer is in a writable state. After each frame to be displayed in the to-be-displayed queue is displayed by the image display schedule, the display queue can be cleared.

In a specific example, the system schedule creates a video decoding schedule and an image display schedule, and allocates a shared buffer, the shared buffer includes 10 frame buffers, each frame buffer is set with a counter, and the initial value of the counter is 0. The video decoding scheduling creates a frame buffer queue, and the frame buffer queue stores the addresses and counter values of the 10 frame buffers. The video decoding scheduling determines the frame buffer whose counter value is 0 according to the frame buffer queue, and writes the reconstructed frame currently decoded and output into the frame buffer whose counter value is 0 according to its corresponding address. Wherein, if the reconstructed frame output by the current decoding is the reference frame of the subsequent decoded video frame, the frame buffer counter where the current reconstructed frame is located is +1. If, when decoding the current frame, the shared buffer has already stored reconstructed frames, and one or several of the reconstructed frames that have been stored are no longer reference frames, but they still have display features and are not displayed, they will be marked as special Refer to the frame ID, add it to the queue to be displayed, and increase its counter by 1. If the shared buffer has already stored reconstructed frames, and one or several of the reconstructed frames already stored are no longer reference frames, and they have no display feature and have been displayed, set their corresponding counters to -1, that is, their corresponding counters are set to -1. The counter returns to 0, indicating that a new reconstructed frame can be written.

If the reconstructed frame output by the current decoding is not only the reference frame of the subsequent decoded video frame (that is, has the reference frame feature) but also is determined to be the frame to be displayed, the image display scheduler can add it to the to-be-displayed queue and add the frame where it is located. The cached counter is +1, and its counter value is 2 at this time. If the reconstructed frame output by the current decoding does not have the reference frame feature, but has the display feature and is determined to be the frame to be displayed, the image display scheduler can add it to the to-be-displayed queue and add 1 to the counter of the frame buffer where it is located. When the counter value is 1.

Optionally, in addition to the reconstructed frame currently output by decoding, the image display scheduler can also determine the frame to be displayed from other reconstructed frames that have been buffered, add the determined frame to be displayed to the queue to be displayed, and buffer the frame where it is located. The counter + 1, such as the above-mentioned reconstructed frame marked with a special reference frame identifier. After that, if the frame to be displayed has a special reference frame identification, its counter is -1. That is, the counter of each to-be-displayed frame in the to-be-displayed queue takes a value of 1 or 2.

The image display schedules each reconstructed frame in the queue to be displayed to be displayed in sequence. For example, each reconstructed frame in the queue of to-be-displayed frames is displayed in sequence in order of distance from the current time from far to near and the numbers are consecutive. And after the reconstructed frame in the frame queue to be displayed is displayed, the counter of its frame buffer is -1. Optionally, for the frame buffer whose counter -1 returns to 0, the stored reconstructed frame can be deleted to support writing a new reconstructed frame. For a frame buffer whose counter is 1 after -1, the stored reconstructed frame can be used as a reference frame for subsequent video frame decoding.

It can be seen that in the solution of the embodiment of the present invention, the video decoding scheduling and the image display scheduling share the frame buffer, and the video decoding scheduling and the image display scheduling do not need to copy data frequently between the decoding buffer and the display buffer, which saves the system buffer space and improves the system performance. Moreover, the solution of the embodiment of the present invention sets a status flag for the reconstructed frame output by decoding, and the reference frame output by decoding can be displayed in real time, and can continue to be used as the reference frame for subsequent video frame decoding after display, thereby reducing the phenomenon of video frame delay and stalling.

The shared buffer mechanism provided by the embodiments of the present invention can reduce the display delay of video frames with as few frame buffer requirements as possible, and this solution can avoid system bandwidth increase, decoding time-consuming and system functions caused by copying data in memory. The disadvantages such as increased power consumption ensure the system performance and improve the video playback experience.

Corresponding to the above video frame buffering method, an embodiment of the present invention further provides a terminal device. Those skilled in the art can understand that these terminal devices can be configured by using commercially available hardware components through the steps taught in this solution.

FIG. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present invention. As shown in FIG. 5 , the terminal device includes: a determination module 201, configured to determine a shared buffer, where the shared buffer includes a plurality of frame buffers, and each of the frame buffers is provided with a status identifier. The decoding module 202 is configured to write the decoded output reconstructed frame into a frame buffer provided with a writable flag, and if the reconstructed frame has a reference frame feature, set a reference frame flag to the corresponding frame buffer. The display module 203 is configured to determine the frame to be displayed from the reconstructed frame that has been buffered, and to set the to-be-displayed flag in the frame buffer where the to-be-displayed frame is located; The frame is displayed, and if the frame to be displayed no longer has the reference frame feature after being displayed, a writable flag is set for the corresponding frame buffer.

The terminal device in this embodiment of the present invention may execute the methods in the embodiments shown in FIG. 3 and FIG. 4 . For parts that are not described in detail in this embodiment, reference may be made to the related descriptions of the embodiments shown in FIG. 3 and FIG. 4 . For the execution process and technical effects of the technical solution, refer to the descriptions in the embodiments shown in FIG. 3 and FIG. 4 , which will not be repeated here.

It should be understood that the division of each module of the terminal device shown in FIG. 5 is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in hardware. For example, the determination module 201 may be a separately established processing element, or may be integrated in a certain chip of the electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, and can also be implemented independently. In the implementation process, each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more specific integrated circuits (Application Specific Integrated Circuit; hereinafter referred to as: ASIC), or, one or more microprocessors Digital Singnal Processor (hereinafter referred to as: DSP), or, one or more Field Programmable Gate Array (Field Programmable Gate Array; hereinafter referred to as: FPGA), etc. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (System-On-a-Chip; hereinafter referred to as: SOC).

FIG. 6 is a schematic structural diagram of another terminal device provided by an embodiment of the present invention. As shown in FIG. 6, the terminal device is represented in the form of a general-purpose computing device. Components of the terminal device may include, but are not limited to, one or more processors 310 , a communication interface 320 , a memory 330 , a communication bus 340 connecting different system components (including the memory 330 , the communication interface 320 and the processing unit 310 ).

Communication bus 340 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (hereinafter referred to as: ISA) bus, Micro Channel Architecture (hereinafter referred to as: MAC) bus, enhanced ISA bus, video electronics Standards Association (Video Electronics Standards Association; hereinafter referred to as: VESA) local bus and Peripheral Component Interconnection (Peripheral Component Interconnection; hereinafter referred to as: PCI) bus.

Electronic devices typically include various computer system readable media. These media can be any available media that can be accessed by the electronic device, including both volatile and nonvolatile media, removable and non-removable media.

The memory 330 may include a computer system-readable medium in the form of volatile memory, such as random access memory (Random Access Memory; hereinafter referred to as: RAM) and/or cache memory. The electronic device may further include other removable/non-removable, volatile/non-volatile computer system storage media. The memory 330 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments shown in FIGS. 3 and 4 of the present specification.

A program/utility having a set (at least one) of program modules that may be stored in memory 330, such program modules including, but not limited to, an operating system, one or more application programs, other program modules, and program data , each or some combination of these examples may include an implementation of a network environment. Program modules typically perform the functions and/or methods of the embodiments described in this specification.

The processor 310 executes various functional applications and data processing by running the programs stored in the memory 330 , for example, implementing the video frame buffering method provided by the embodiments shown in FIGS. 3 to 4 of this specification.

In a specific implementation, the present application further provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in the various embodiments provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (English: read-only memory, abbreviated as: ROM) or a random access memory (English: random access memory, abbreviated as: RAM) and the like.

In a specific implementation, an embodiment of the present invention further provides a computer program product, where the computer program product includes executable instructions, and when the executable instructions are executed on a computer, causes the computer to execute part or some of the above method embodiments. all steps.

In the embodiments of the present invention, "at least one" refers to one or more, and "multiple" refers to two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can indicate the existence of A alone, the existence of A and B at the same time, and the existence of B alone. where A and B can be singular or plural. The character "/" generally indicates that the associated objects are an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c may be single or multiple.

Those of ordinary skill in the art can realize that the units and algorithm steps described in the embodiments disclosed herein can be implemented by a combination of electronic hardware, computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided by the present invention, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM for short), random access memory (RAM for short), magnetic disk or CD, etc. that can store program codes medium.

The above are only specific embodiments of the present invention. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included within the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

A video frame buffering method, comprising:

determining a shared buffer, the shared buffer includes a plurality of frame buffers, and each of the frame buffers is provided with a status identifier;

Write the decoded output reconstructed frame into a frame buffer that is provided with a writable identifier, and if the reconstructed frame has a reference frame feature, then set a reference frame identifier to the corresponding frame buffer;

Determine the frame to be displayed from the reconstructed frame that has been buffered, and set a to-be-displayed flag for the frame buffer where the frame to be displayed is located;

The to-be-displayed frame in the frame buffer set with the to-be-displayed flag is displayed, and if the to-be-displayed frame no longer has the reference frame feature after being displayed, a writable flag is set to the corresponding frame buffer.
The method according to claim 1, wherein the number of frame buffers included in the shared buffer is determined according to the number of reference frames and the number of buffered frames of the video stream to be played.
The method according to claim 1 or 2, wherein each of the frame buffers is provided with a status identifier, comprising:

Each of the frame buffers is provided with a counter, and the value of the counter is used as a state identifier of the corresponding frame buffer.
The method according to claim 1, wherein the method further comprises: creating a frame buffer queue, and the frame buffer queue is used to store the status identification information of each frame buffer in the shared buffer;

Write the decoded output reconstructed frame to the framebuffer with writable flag, including:

According to the state identification information of the respective frame buffers stored in the frame buffer queue, determine the frame buffers provided with the writable flags, wherein the frame buffers provided with the writeable flags are used for the reconstruction of the write decoding output frame.
The method according to claim 1, wherein determining the frame to be displayed from the reconstructed frames that have been buffered comprises:

If the reconstructed frame output by the current decoding only has the display feature, then the reconstructed frame output by the current decoding is determined as the frame to be displayed, and the corresponding frame buffer is set on the basis of the writable mark. The mark to be displayed is set;

If the reconstructed frame output by the current decoding has both the reference frame feature and the display feature, the reconstructed frame output by the current decoding is determined as the frame to be displayed, and the to-be-displayed flag is set for the corresponding frame buffer based on the reference frame flag.
The method according to claim 5, wherein determining the frame to be displayed from the reconstructed frames that have been buffered, further comprising:

In addition to the reconstructed frame currently decoded and output, if the reconstructed frame in at least one frame buffer that has been set with the reference frame identifier no longer has the reference frame feature, and has the display feature and has not yet been displayed, the corresponding reconstructed frame is determined to be pending Display the frame, and set the reference frame ID of the corresponding frame buffer as the ID to be displayed;

In addition to the reconstructed frame output by the current decoding, if the reconstructed frame in at least one frame buffer that has been set with a reference frame identifier no longer has a reference frame identifier, and has been displayed or has no display feature, the reference frame of the corresponding frame buffer The ID is set to writable ID.
The method according to claim 1, 5 or 6, wherein the method further comprises:

A queue to be displayed is created, and the queue to be displayed is used to store the to-be-displayed identification information of the to-be-displayed frame.
The method according to claim 7, wherein the queue to be displayed is further used to store timestamp information and number information of the to-be-displayed frame; displaying the to-be-displayed frame includes:

The display order of the to-be-displayed frames is determined according to the timestamp information and the serial number information of the to-be-displayed frames, and the to-be-displayed frames are displayed according to the display order of the to-be-displayed frames.
The method according to claim 8, wherein if the to-be-displayed frame no longer has the reference frame feature after being displayed, setting a writable flag to the corresponding frame buffer, comprising:

If the to-be-displayed frame only has display features, then after the to-be-displayed frame is displayed, the to-be-displayed identification is set as a writable identification;

If the frame to be displayed has both the display feature and the reference feature, then after the frame to be displayed is displayed, the marker to be displayed is set as the reference frame marker, and then the reference frame marker is set as the marker to be written.
A terminal device, characterized in that it includes:

a determining module, configured to determine a shared buffer, the shared buffer includes a plurality of frame buffers, and each of the frame buffers is provided with a state identifier;

A decoding module for writing the decoded output reconstructed frame into a frame buffer that is provided with a writable identifier, and if the reconstructed frame has a reference frame feature, then a reference frame identifier is set to the corresponding frame buffer;

A display module, configured to determine a frame to be displayed from the reconstructed frame that has been cached, and set a to-be-displayed mark in the frame buffer where the to-be-displayed frame is located; set the to-be-displayed frame in the frame buffer where the to-be-displayed mark is set The display is performed, and if the frame to be displayed no longer has the reference frame feature after being displayed, a writable flag is set for the corresponding frame buffer.
A terminal device, characterized in that it includes:

at least one processor; and

at least one memory communicatively coupled to the processor, wherein:

The memory stores program instructions executable by the processor, which are invoked by the processor to perform a method as claimed in any one of claims 1 to 9.
A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein when the program is run, a device where the computer-readable storage medium is located is controlled to execute any one of claims 1 to 9 the method described.