CN116939215A

CN116939215A - Video frame coding transmitting method and device, storage medium, product and electronic equipment

Info

Publication number: CN116939215A
Application number: CN202310773785.5A
Authority: CN
Inventors: 章万国
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2023-06-27
Filing date: 2023-06-27
Publication date: 2023-10-24

Abstract

The application discloses a video frame coding sending method, a device, a storage medium, a product and electronic equipment, wherein the method comprises the following steps: if the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed is used as a reference frame, a confirmation frame identification corresponding to the reference frame is sent to the sending end, the sending end takes the video frame to be confirmed corresponding to the confirmation frame identification as the reference frame, coding processing is carried out on the uncoded video frame in the video source based on the reference frame to obtain video coding data, the video coding data is sent to the receiving end, and decoding processing is carried out on the video coding data based on the reference frame by the receiving end to obtain the video frame of the video source.

Description

Video frame coding transmitting method and device, storage medium, product and electronic equipment

Technical Field

The present application relates to the field of computer technologies, and in particular, to a video frame coding transmission method and apparatus, a storage medium, a product, and an electronic device.

Background

The data transmission among different devices can be realized through wireless communication modes such as the Internet, the Internet can meet the data transmission modes that a transmitting end uploads data and a receiving end downloads data at present, and can meet the real-time data transmission mode that the transmitting end directly transmits the data to the receiving end through the Internet, and the faster network propagation speed can even enable video data among different devices to be transmitted in real time, for example, a user can finish online video communication, video live broadcast and the like through the Internet. In the prior art, a transmitting end can encode a video frame and then transmit the encoded video frame to a receiving end, the receiving end can obtain the video frame after decoding to realize video playing, wherein the encoded data stream of the video frame has a forward neighbor dependency relationship, if the encoded data of any video frame has defects and packet loss, the subsequent video frame cannot be normally decoded, the receiving end cannot normally play the video until the transmitting end is required to retransmit the video, and the video playing is seriously blocked.

Disclosure of Invention

The embodiment of the application provides a video frame coding and transmitting method, a device, a storage medium, a product and electronic equipment, which can prevent the follow-up video frame from being unable to be decoded and played normally due to packet loss and the like by taking a complete video frame obtained by a receiving end as a reference frame and carrying out coding and decoding processing on the video frame by the reference frame, thereby improving the fluency of video playing. The technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a video frame coding transmission method, applied to a transmitting end, where the method includes:

if the confirmation frame identification sent by the receiving end is received, taking a video frame to be confirmed corresponding to the confirmation frame identification as a reference frame; the video frame to be confirmed is a video frame which can be decoded by the receiving end;

coding the uncoded video frames in the video source based on the reference frames to obtain video coding data;

and sending the video coding data to the receiving end.

In a second aspect, an embodiment of the present application provides a video frame coding transmission method, applied to a receiving end, where the method includes:

if a decodable video frame to be confirmed sent by a sending end is obtained, the video frame to be confirmed is used as a reference frame;

Transmitting a confirmation frame identifier corresponding to the reference frame to the transmitting end, so that the transmitting end encodes an uncoded video frame in a video source based on the reference frame to obtain video encoding data;

and receiving the video coding data sent by the sending end, and decoding the video coding data based on the reference frame to obtain the video frame of the video source.

In a third aspect, an embodiment of the present application provides a video frame coding transmitting apparatus, including:

the frame identification receiving module is used for taking a video frame to be confirmed corresponding to the confirmation frame identification as a reference frame if the confirmation frame identification sent by the receiving end is received; the video frame to be confirmed is a video frame which can be decoded by the receiving end;

the coding processing module is used for coding the uncoded video frames in the video source based on the reference frames to obtain video coding data;

and the coded data transmitting module is used for transmitting the video coded data to the receiving end.

In a fourth aspect, an embodiment of the present application provides a video frame coding receiving apparatus, including:

the reference frame acquisition module is used for taking the video frame to be confirmed as a reference frame if the decodable video frame to be confirmed sent by the sending end is acquired;

The frame identification sending module is used for sending the confirmation frame identification corresponding to the reference frame to the sending end so that the sending end can encode the uncoded video frame in the video source based on the reference frame to obtain video encoding data;

and the decoding processing module is used for receiving the video coding data sent by the sending end, and decoding the video coding data based on the reference frame to obtain the video frame of the video source.

In a fifth aspect, embodiments of the present application provide a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the above-described method steps.

In a sixth aspect, embodiments of the present application provide a computer program product storing a plurality of instructions adapted to be loaded by a processor and to perform the above-described method steps.

In a seventh aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the above-mentioned method steps.

In one or more embodiments of the present application, if a receiving end obtains a decodable video frame to be confirmed sent by a sending end, the receiving end uses the video frame to be confirmed as a reference frame, a confirmation frame identifier corresponding to the reference frame is sent to the sending end, the sending end uses the video frame to be confirmed corresponding to the confirmation frame identifier as the reference frame, the video frame is subjected to coding processing on an uncoded video frame in a video source based on the reference frame to obtain video coding data, the video coding data is sent to the receiving end, and the receiving end performs decoding processing on the video coding data based on the reference frame to obtain the video frame of the video source. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved.

Drawings

In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is an exemplary schematic diagram of video frame transmission according to an embodiment of the present application;

fig. 2 is a schematic flow chart of a video frame coding transmission method according to an embodiment of the present application;

fig. 3 is a schematic flow chart of a video frame coding transmission method according to an embodiment of the present application;

FIG. 4 is an exemplary diagram of a reference frame replacement update provided by an embodiment of the present application;

fig. 5 is a schematic flow chart of a video frame coding transmission method according to an embodiment of the present application;

fig. 6 is a flowchart of a video frame coding transmission method according to an embodiment of the present application;

fig. 7 is a schematic flow chart of a video frame coding transmission method according to an embodiment of the present application;

fig. 8 is a flowchart of a video frame coding transmission method according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of a video frame coding transmitting device according to an embodiment of the present application;

fig. 10 is a schematic structural diagram of a video frame coding transmitting device according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Fig. 12 is a schematic structural diagram of a video frame coding receiving device according to an embodiment of the present application;

Fig. 13 is a schematic structural diagram of a video frame coding receiving device according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

Referring to fig. 1, an exemplary schematic diagram of video frame transmission is provided for an embodiment of the present application, and the video frame coding transmission method provided in the embodiment of the present application may be implemented by a computer program and may be run on a video frame coding transmission device and a video frame coding reception device based on von neumann system. The computer program may be integrated in the application or may run as a stand-alone tool class application. The video frame coding transmitting device can be a transmitting end, a module for realizing the video frame coding transmitting method in the transmitting end, and the video frame coding receiving device can be a receiving end, a module for realizing the video frame coding transmitting method in the receiving end. The receiving end and the sending end can be mobile phones, tablet computers, wearable equipment, servers and other terminal equipment, and the receiving end and the sending end can be connected in a wireless mode such as the Internet, so that the transmission of video frame related data is realized. The sending end can sequentially encode the video frames in the video source according to the time sequence, so as to obtain video encoding data corresponding to the video frames, the sending end can transmit the video encoding data to the receiving end in a wireless mode such as the Internet, the receiving end can decode the video encoding data after receiving the video encoding data, so as to obtain the video frames of the video source, and play the video frames, so that real-time video communication between the sending end and the receiving end is realized.

The video source can be video data stored locally or in the cloud end of the sending end, and can also be video data collected through a camera of the sending end, and it can be understood that the sending end can be a server of a video website, the receiving end can check the video data on the video website on line through accessing the sending end, the sending end can determine the video source to be checked by the receiving end from the local storage or the cloud end storage, then the video frames in the video source are subjected to coding processing frame by frame to obtain video coding data and sent to the receiving end, the receiving end decompresses the video coding to obtain the video frames of the video source, and the video frames are subjected to playing processing on the receiving end, so that the purpose that a user checks the video data on the sending end on the receiving end is achieved.

Similarly, the sending end can also be terminal equipment such as a mobile phone held by other users, the other users and the users can carry out video communication through the sending end and the receiving end, the other users adopt cameras of the sending end to collect video data as video sources, the sending end carries out coding processing on video frames in the video sources frame by frame to obtain video coding data and sends the video coding data to the receiving end, the receiving end decompresses the video coding to obtain video frames of the video sources, and play processing is carried out on the video frames on the receiving end, so that the users can view the video data collected by the cameras of the sending end on the receiving end in real time. It can be understood that when other users and users can perform video communication with the receiving end through the transmitting end, the receiving end can also use the camera of the receiving end to collect video data and transmit the video data to the transmitting end, and the steps of the transmitting end for transmitting the video frames to the receiving end are the same.

The sending end may perform coding processing on the video frame based on the reference frame, as shown in fig. 1, where the video frame a, the video frame B, the video frame C, and the like are all video frames in the video source, the sending end may use the video frame a as the reference frame, and then perform coding processing on the video frame subsequent to the video frame a by using the reference frame, for example, the sending end may perform coding processing on the video frame B based on the reference frame to obtain video coding data of the video frame B, where the video coding data is data for recording differences between the video frame B and the video frame a, and after the receiving end obtains the video coding data of the video frame B, perform decoding processing on the video coding data, where the decoding processing is to combine the video frame a and the data for combining the differences between the video frame B and the video frame a to obtain the video frame B. Because the coding process of the transmitting end and the decoding process of the receiving end both depend on the reference frame, the reference frame needs to be a complete video frame which can be obtained by both the transmitting end and the receiving end, and it can be understood that the transmitting end can easily obtain the complete video frame A from the video source, i.e. the video frame A can be used as the reference frame, and the subsequent video frames can be successfully coded. In order to ensure that the reference frame is a complete video frame in both the transmitting end and the receiving end, the reference frame in the embodiment of the application is a video frame which can be decoded by the receiving end, that is, the receiving end receives a piece of coded data to be confirmed transmitted by the transmitting end, the coded data to be confirmed is complete and can be completely decoded by the receiving end, the receiving end can obtain a video frame to be confirmed after decoding the coded data to be confirmed, the video frame to be confirmed is taken as the reference frame, the coded data to be confirmed is the video coded data of the reference frame transmitted to the receiving end by the transmitting end, and the video frame to be confirmed is the video frame corresponding to the coded data to be confirmed.

The video frame coding transmission method provided by the application is described in detail below with reference to specific embodiments.

Referring to fig. 2, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 2, the embodiment of the present application describes a video frame coding transmission method with reference to a transmitting end and a receiving end, and the method may include the following steps S102 to S112.

S102, if the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed is used as a reference frame.

Specifically, before the receiving end receives the decodable video frame to be confirmed, the transmitting end can encode the video frame based on the video frame of the previous frame because of no reference frame, generate the encoded data to be confirmed, and then transmit the encoded data to be confirmed to the receiving end. For example, the transmitting end may perform encoding processing on the second video frame based on the first video frame, so as to obtain encoded data to be confirmed corresponding to the second video frame, where the first video frame is a previous video frame of the second video frame in the video source. It can be understood that if the first video frame is the first frame in the video source, no other video frame is before the first video frame, and the first video frame cannot be encoded based on the video frame of the previous frame, the receiving end may directly encode the first video frame to obtain the encoded data to be confirmed corresponding to the first video frame, where the encoded data to be confirmed corresponding to the first video frame directly includes the data of the first video frame, and if the receiving end may successfully decode the encoded data to be confirmed corresponding to the first video frame, the first video frame may be directly obtained.

The receiving end can receive the coded data to be confirmed sent by the sending end, but the receiving end cannot necessarily receive all the coded data to be confirmed due to the reasons of network signal fluctuation, poor network signal quality and the like, and decoding processing cannot be completed due to the reasons of defects, packet loss and the like of the received coded data to be confirmed, so that a complete video frame cannot be obtained. When the receiving end obtains the first coded data to be confirmed which can be successfully decoded, the coded data to be confirmed is decoded to obtain a video frame to be confirmed, and it can be understood that the video frame to be confirmed is a complete video frame as the receiving end can successfully decode the coded data to be confirmed, and is also the first complete video frame obtained from the transmitting end by the receiving end. If the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed can be used as a reference frame.

S104, the receiving end sends the identification of the confirmation frame corresponding to the reference frame to the sending end.

Specifically, each video frame has a corresponding frame identifier, the frame identifiers are used for distinguishing different video frames, the frame identifiers can be used for marking the video frames when the sending end encodes the video frames, can be corresponding time stamps of the video frames in the video source or can be sequencing of the video frames in the video source, and after the receiving end successfully decodes the coded data to be confirmed, the video frames to be confirmed corresponding to the coded data to be confirmed and the frame identifiers corresponding to the video frames to be confirmed can be obtained. The receiving end can send the confirmation frame identifier corresponding to the reference frame to the sending end, so that the sending end carries out coding processing on the uncoded video frame in the video source based on the reference frame to obtain video coding data.

S106, the transmitting end takes the video frame to be confirmed corresponding to the confirmation frame identification as a reference frame.

Specifically, if the transmitting end receives the acknowledgement frame identifier sent by the receiving end, the video frame to be acknowledged corresponding to the acknowledgement frame identifier can be obtained in the video frame buffer, and the video frame to be acknowledged can be used as a reference frame.

It can be understood that, in order to prevent the acknowledgement frame identifier from being lost in the process that the receiving end sends the acknowledgement frame identifier to the sending end, the sending end may send the identifier receiving reply message to the receiving end after receiving the acknowledgement frame identifier, where the identifier receiving reply message is used to inform the receiving end that the receiving end successfully receives the acknowledgement frame identifier and takes the video frame to be acknowledged corresponding to the acknowledgement frame identifier as a reference frame, and similarly, the receiving end may take the video frame to be acknowledged as the reference frame after receiving the identifier receiving reply message sent by the sending end.

S108, the transmitting end carries out coding processing on the uncoded video frames in the video source based on the reference frames to obtain video coding data.

Specifically, the transmitting end may perform coding processing on an uncoded video frame in the video source based on the reference frame to obtain video coding data, where the uncoded video frame is a video frame in the video source that has not been subjected to coding processing by the transmitting end and has not yet been transmitted to the receiving end, and the transmitting end may sequentially perform coding processing on the video frames in the video source according to a time sequence. For example, the transmitting end may confirm a third video frame in the video source, where the third video frame is an uncoded video frame in the video source, and the transmitting end may encode the third video frame based on the reference frame, so as to obtain video encoded data of the third video frame.

S110, the transmitting end transmits the video coding data to the receiving end.

Specifically, the transmitting end may transmit the video encoded data to the receiving end. It can be understood that the transmitting end may sequentially encode the video frames in the video source according to the time sequence to obtain video encoded data, and transmit the video encoded data after the encoding process to the receiving end according to the time sequence.

S112, the receiving end decodes the video coding data based on the reference frame to obtain a video frame of the video source.

Specifically, the receiving end can decode the video frame encoded data based on the reference frame, so as to obtain the video frame of the video source, and because the transmitting end sequentially transmits the video encoded data according to the time sequence, the receiving end can sequentially decode the video encoded data according to the time sequence of receiving the video encoded data, so as to obtain the continuous and correctly ordered video frame. After the receiving end obtains the video frame of the video source, the video frame can be played.

In the embodiment of the application, if the receiving end acquires the decodable video frame to be confirmed sent by the sending end, the receiving end takes the video frame to be confirmed as a reference frame, a confirmation frame identification corresponding to the reference frame is sent to the sending end, the sending end takes the video frame to be confirmed corresponding to the confirmation frame identification as the reference frame, the video frame not coded in the video source is coded based on the reference frame to obtain video coding data, the video coding data is sent to the receiving end, and the receiving end decodes the video coding data based on the reference frame to obtain the video frame of the video source. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved.

Referring to fig. 3, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 3, the embodiment of the present application describes a video frame coding transmission method with reference to a transmitting end and a receiving end, and the method may include the following steps S202 to S222.

S202, the transmitting end encodes the second video frame based on the first video frame to obtain the encoded data to be confirmed.

Specifically, before the receiving end receives the decodable video frame to be confirmed, because there is no reference frame, the transmitting end can encode the video frame based on the previous frame video frame, and the transmitting end can encode the second video frame based on the first video frame, so as to obtain the encoded data to be confirmed corresponding to the second video frame, wherein the first video frame is the previous frame of the second video frame in the video source.

Optionally, the transmitting end may determine video frame difference data between the second video frame and the first video frame, and encode the second video frame based on the video frame difference data, so as to obtain encoded data to be confirmed of the second video frame, where the encoded data to be confirmed is data recording differences between the second video frame and the first video frame.

It can be understood that if the first video frame is the first frame in the video source, no other video frame is before the first video frame, and the first video frame cannot be encoded based on the video frame of the previous frame, the receiving end may directly encode the first video frame to obtain the encoded data to be confirmed corresponding to the first video frame, where the encoded data to be confirmed corresponding to the first video frame directly includes the data of the first video frame, and if the receiving end may successfully decode the encoded data to be confirmed corresponding to the first video frame, the first video frame may be directly obtained.

S204, the sending end stores the second video frame into a video frame buffer memory.

Specifically, since the receiving end does not acquire the decodable video frame yet, the coded data to be confirmed sent by the sending end may not be successfully decoded by the receiving end, so when no reference frame exists, the sending end can save the video frame into the video frame buffer after coding the video frame, thereby avoiding the loss of the coded data to be confirmed sent before due to the too weak network signal and improving the fault tolerance. Therefore, after the sending end performs coding processing on the second video frame to obtain to-be-confirmed coded data of the second video frame, the second video frame can be stored in the video frame buffer.

Optionally, if the sending end does not receive the acknowledgement frame identifier sent by the receiving end within the preset waiting period, which means that the sending end does not receive the acknowledgement frame identifier sent by the receiving end for a long time, possibly because the receiving end does not receive complete and decodable coded data to be acknowledged due to weak network signals and other reasons, the sending end may prompt a user or related staff to check the network signals, and may also retransmit the coded data to be acknowledged corresponding to the video frame in the video frame buffer. The preset waiting period may be an initial setting of the video frame coding transmitting device or the video frame coding receiving device, or may be an initial setting of a user or a related staff, for example, may be 2 seconds.

If the sending end does not receive the acknowledgement frame identifier sent by the receiving end within the preset waiting time period, the sending end can output sending failure prompt information, where the sending failure prompt information is used to prompt the user or related staff that the receiving end does not obtain the decodable video frame for a long time, and there may be situations such as weak network signal, etc., where the user or related staff is required to check the network signal or attempt to connect with the receiving end again. The sending failure prompt information can be displayed on a display screen of the sending end in a popup window mode, and can also be played by a loudspeaker of the sending end in a voice mode.

If the sending end does not receive the confirmation frame identification sent by the receiving end in the preset waiting time period, the sending end carries out coding processing again on the video frames in the video frame buffer to obtain coded data to be confirmed, and then the coded data to be confirmed is sent to the receiving end. And re-encoding the second video frame in the video frame buffer to obtain the encoded data to be confirmed of the second video frame, and then transmitting the encoded data to be confirmed to a receiving end. It can be understood that, after the sending end performs coding processing on the second video frame to obtain the coded data to be confirmed, the sending end not only can save the second video frame to the video frame buffer, but also can save the coded data to be confirmed of the second video frame to the video frame buffer, and if the sending end does not receive the identification of the confirmed frame sent by the receiving end in the preset waiting period, the sending end can directly send the coded data to be confirmed in the video frame buffer to the receiving end.

Optionally, if the Round-Trip Time (RTT) of the signal is large, the video frame buffer of the transmitting end is small, and the like, each video frame is stored before the reference frame is not available, which occupies an excessive video frame buffer space, so the transmitting end may store the second video frame according to the buffer Time interval, that is, if the Time interval between the second video frame and the video frame stored in the video frame buffer before the second video frame is greater than or equal to the buffer Time interval, the second video frame is stored in the video frame buffer. The buffer time interval may be set initially for the video frame coding transmitting device, or may be set by a user or a related staff.

S206, the sending end sends the coded data to be confirmed to the receiving end.

Specifically, the transmitting end performs coding processing on an uncoded video frame in the video source to obtain coded data to be confirmed, and then the transmitting end can transmit the coded data to be confirmed to the receiving end. It can be understood that before the acknowledgement frame identifier sent by the receiving end is not received, the sending end may sequentially perform coding processing on the uncoded video frames in the video source according to the time sequence to obtain coded data to be acknowledged, and then sequentially send the coded data to be acknowledged to the receiving end according to the time sequence.

And S208, if the receiving end successfully completes the decoding processing of the coded data to be confirmed, acquiring a video frame to be confirmed in the coded data to be confirmed.

Specifically, the receiving end may receive the coded data to be confirmed sent by the sending end, but the receiving end may not necessarily receive all the coded data to be confirmed due to reasons of network signal fluctuation, poor network signal quality, and the like, or may not complete decoding processing due to reasons of defects, packet loss, and the like of the received coded data to be confirmed, so that a complete video frame may not be obtained. When the receiving end obtains the first coded data to be confirmed which can be successfully decoded, the coded data to be confirmed is decoded to obtain a video frame to be confirmed, and it can be understood that the video frame to be confirmed is a complete video frame as the receiving end can successfully decode the coded data to be confirmed, and is also the first complete video frame obtained from the transmitting end by the receiving end.

It can be understood that if the receiving end cannot complete the decoding process of the coded data to be confirmed, that is, the coded data to be confirmed has defects, packet loss and other conditions, the receiving end can discard the coded data to be confirmed. And then decoding the next coded data to be confirmed.

S210, if the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed is taken as a reference frame.

Specifically, the video frame to be confirmed is a complete video frame, and is also the first complete video frame obtained by the receiving end from the sending end, and the video frame to be confirmed is a complete video frame stored in the receiving end and the sending end at the same time, so if the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed can be used as a reference frame.

S212, the receiving end sends the identification of the acknowledgement frame corresponding to the reference frame to the sending end.

Optionally, in order to avoid the loss of the acknowledgement frame identifier in the process of sending from the receiving end to the sending end due to the network signal difference, the receiving end may send the switching frame identifier to the sending end in a redundant feedback manner, that is, at least two acknowledgement frame identifiers may be sent to the sending end at the same time, so as to avoid the loss of the acknowledgement frame identifier.

S214, the transmitting end takes the video frame to be confirmed corresponding to the confirmation frame identification as a reference frame.

S216, the sending end releases the video frames with the time stamp smaller than the time stamp corresponding to the reference frame in the video frame buffer.

Specifically, the sender receives the identification of the acknowledgement frame, and after taking the video frame to be acknowledged as the reference frame, the timestamp corresponding to the reference frame can be obtained, the timestamp corresponding to the reference frame is the timestamp corresponding to the reference frame in the video source, then the release processing is performed on the video frames in the video frame buffer, the timestamp of which is smaller than the timestamp corresponding to the reference frame, namely the release processing is performed on the video frames except the reference frame, namely the video frames in the video frame buffer, the timestamp of which is smaller than the timestamp corresponding to the reference frame, are deleted and discarded.

S218, the transmitting end performs coding processing on the uncoded video frames in the video source based on the reference frames to obtain video coding data.

Specifically, the transmitting end may perform coding processing on an uncoded video frame in the video source based on the reference frame to obtain video coding data, where the uncoded video frame is a video frame in the video source that has not been subjected to coding processing by the transmitting end and has not yet been transmitted to the receiving end, and the transmitting end may sequentially perform coding processing on the video frames in the video source according to a time sequence.

Optionally, the transmitting end may determine a third video frame in the video source, where the third video frame is an uncoded video frame in the video source, and then determine video frame difference data between the third video frame and the reference frame, and encode the third video frame based on the video frame difference data to obtain video encoded data of the third video frame. The video coding data of the third video frame is data for recording the difference between the third video frame and the reference frame, and after the receiving end obtains the video coding data of the third video frame, the receiving end can decode the video coding data by combining the reference frame to obtain the third video frame.

S220, the transmitting end transmits the video coding data to the receiving end.

S222, the receiving end decodes the video coding data based on the reference frame to obtain a video frame of the video source.

It can be understood that after the reference frame is confirmed, the receiving end encodes the subsequent uncoded video frame of the video source based on the reference frame, and because the receiving end encodes the uncoded video frame based on the video frame difference data between the uncoded video frame and the reference frame, so as to obtain the video encoded data of the uncoded video frame, it can be understood that the longer the time interval between the uncoded video frame and the reference frame is, the larger the difference between the uncoded video frame and the reference frame is, the larger the video encoded data of the uncoded video frame is, and the larger the data transmission code rate between the transmitting end and the receiving end is, and the larger the network transmission pressure is. Therefore, the receiving end and the transmitting end can perform replacement update processing on the reference frame every other switching time period, so that the fluency of video playing is further improved, wherein the switching time period can be the initial setting of the video frame coding transmitting device or the video frame coding receiving device, and can also be set for a user or related staff, for example, the switching time period can be 1 second.

Optionally, the receiving end may determine a switching video frame of the switching period of the video frame interval to be confirmed, acquire a switching frame identifier corresponding to the switching video frame, replace the video frame to be confirmed with the switching video frame, confirm the video frame to be confirmed as a reference frame, and send the switching frame identifier to the sending end. If the transmitting end receives the switching frame identification transmitted by the receiving end, the transmitting end acquires the switching video frame corresponding to the switching frame identification, and then adopts the switching video frame to replace the video frame to be confirmed, and confirms the video frame to be the reference frame.

Optionally, in order to avoid the loss of the switching frame identifier in the process of sending the switching frame identifier from the receiving end to the sending end due to the network signal difference, the receiving end may send the switching frame identifier to the sending end in a redundant feedback manner, that is, at least two switching frame identifiers may be sent to the sending end at the same time, so as to avoid the loss of the switching frame identifier.

Referring to fig. 4 together, an exemplary schematic diagram of a reference frame replacement update is provided in an embodiment of the present application, as shown in fig. 4, a video frame a may be a video frame to be confirmed, a transmitting end and a receiving end may confirm the video frame a as a reference frame, that is, the transmitting end may perform coding processing on an uncoded video frame after the video frame a in a video source based on the video frame a, for example, perform coding processing on a video frame B, a video frame C, etc. based on the video frame a to obtain video coded data, and the receiving end may perform decoding processing on the video coded data based on the video frame a after receiving the video coded data, thereby obtaining a video frame after the video frame a in the video source such as the video frame B, the video frame C, etc. If the receiving end determines the switching video frame of the switching time period between the video frame to be confirmed and the video frame to be confirmed, as shown in fig. 4, the switching time period is between the video frame D and the video frame a, so that the video frame D is the switching video frame, the receiving end can acquire the switching frame identifier of the video frame D, the receiving end confirms that the video frame D replaces the video frame a as a reference frame, and sends the switching frame identifier to the sending end. After the sending end receives the switching frame identifier, the switching video frame corresponding to the switching frame identifier, namely the video frame D, can be obtained based on the switching frame identifier, and the sending end confirms that the video frame D replaces the video frame a as a reference frame. The sending end still carries out coding processing on uncoded video frames after the reference frames in the video source based on the reference frames, namely carries out coding processing on uncoded video frames after the video frames D in the video source such as the video frames E based on the video frames D, so as to obtain video coding data, then sends the video coding data to the receiving end, and after receiving the video coding data, the receiving end still carries out decoding processing on the video coding data based on the reference frames, namely carries out decoding processing on the video coding data based on the video frames D.

In the embodiment of the application, the transmitting end encodes the second video frame based on the first video frame to obtain the coded data to be confirmed, and the second video frame is stored in the video frame buffer, so that the coded data to be confirmed, which is transmitted before, is prevented from being lost due to the fact that the network signal is too weak, and the fault tolerance rate is improved. The method comprises the steps that a sending end sends coded data to be confirmed to a receiving end, if the receiving end successfully completes decoding processing of the coded data to be confirmed, a video frame to be confirmed in the coded data to be confirmed is obtained, if the receiving end obtains a decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed is taken as a reference frame, a confirmation frame identification corresponding to the reference frame is sent to the sending end, the sending end takes the video frame to be confirmed corresponding to the confirmation frame identification as the reference frame, release processing is carried out on the video frame with a time stamp smaller than a time stamp corresponding to the reference frame in a video frame buffer, and the video frame buffer is released to reduce the memory pressure of the sending end. The transmitting end encodes the uncoded video frames in the video source based on the reference frames to obtain video encoded data, the video encoded data is transmitted to the receiving end, and the receiving end decodes the video encoded data based on the reference frames to obtain the video frames of the video source. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved. And the receiving end and the transmitting end can replace and update the reference frame every other switching time period, so that the overlarge data transmission code rate between the transmitting end and the receiving end is avoided, the network transmission pressure is reduced, and the fluency of video playing is further improved.

Referring to fig. 5, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 5, an embodiment of the present application describes a video frame coding transmission method based on a transmitting end side, and the method may include the following steps S302 to S306.

S302, if the confirmation frame identification sent by the receiving end is received, the video frame to be confirmed corresponding to the confirmation frame identification is used as a reference frame.

Specifically, before the receiving end receives the decodable video frame to be confirmed, the transmitting end can encode the video frame based on the video frame of the previous frame because of no reference frame, generate the encoded data to be confirmed, and then transmit the encoded data to be confirmed to the receiving end. The receiving end can receive the coded data to be confirmed sent by the sending end, but the receiving end cannot necessarily receive all the coded data to be confirmed due to the reasons of network signal fluctuation, poor network signal quality and the like, and decoding processing cannot be completed due to the reasons of defects, packet loss and the like of the received coded data to be confirmed, so that a complete video frame cannot be obtained. When the receiving end obtains the first coded data to be confirmed which can be successfully decoded, the coded data to be confirmed is decoded to obtain a video frame to be confirmed, and it can be understood that the video frame to be confirmed is a complete video frame as the receiving end can successfully decode the coded data to be confirmed, and is also the first complete video frame obtained from the transmitting end by the receiving end. If the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed can be used as a reference frame.

Each video frame has a corresponding frame identifier, the frame identifiers are used for distinguishing different video frames, the frame identifiers can be marked on the video frames when the sending end encodes the video frames, can be corresponding time stamps of the video frames in the video source or can be sequencing of the video frames in the video source, and after the receiving end successfully decodes the encoded data to be confirmed, the video frames to be confirmed corresponding to the encoded data to be confirmed and the frame identifiers corresponding to the video frames to be confirmed can be obtained. The receiving end can send the confirmation frame identifier corresponding to the reference frame to the sending end, so that the sending end carries out coding processing on the uncoded video frame in the video source based on the reference frame to obtain video coding data. If the transmitting end receives the acknowledgement frame identifier sent by the receiving end, the video frame to be acknowledged corresponding to the acknowledgement frame identifier can be obtained in the video frame buffer, and the video frame to be acknowledged can be used as a reference frame.

S304, coding the uncoded video frames in the video source based on the reference frames to obtain video coding data.

And S306, transmitting the video coding data to a receiving end.

In the embodiment of the application, if the confirmation frame identifier sent by the receiving end is received, a video frame to be confirmed corresponding to the confirmation frame identifier is taken as a reference frame, wherein the video frame to be confirmed is a video frame which can be decoded by the receiving end, the video frame which is not coded in a video source is coded based on the reference frame, video coding data are obtained, and the video coding data are sent to the receiving end. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved.

Referring to fig. 6, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 6, an embodiment of the present application describes a video frame coding transmission method based on a transmitting side, and the method may include the following steps S402 to S412.

S402, encoding the second video frame based on the first video frame to obtain the encoded data to be confirmed.

S404, the second video frame is stored in a video frame buffer, and the coded data to be confirmed is sent to the receiving end.

Specifically, since the receiving end does not acquire the decodable video frame yet, the coded data to be confirmed sent by the sending end may not be successfully decoded by the receiving end, so when no reference frame exists, the sending end can save the video frame into the video frame buffer after coding the video frame, thereby avoiding the loss of the coded data to be confirmed sent before due to the too weak network signal and improving the fault tolerance. Therefore, after the sending end performs coding processing on the second video frame to obtain to-be-confirmed coded data of the second video frame, the second video frame can be stored in the video frame buffer. The sending end performs coding processing on the uncoded video frames in the video source to obtain coded data to be confirmed, and then the sending end can send the coded data to be confirmed to the receiving end. It can be understood that before the acknowledgement frame identifier sent by the receiving end is not received, the sending end may sequentially perform coding processing on the uncoded video frames in the video source according to the time sequence to obtain coded data to be acknowledged, and then sequentially send the coded data to be acknowledged to the receiving end according to the time sequence.

Optionally, if the signal RTT is larger, the video frame buffer of the transmitting end is smaller, and the like, each video frame is stored before the reference frame is absent, which occupies an excessive video frame buffer space, so the transmitting end may store the second video frame according to a buffer time interval, that is, if the time interval between the second video frame and the video frame stored in the video frame buffer before the second video frame is greater than or equal to the buffer time interval, the second video frame is stored in the video frame buffer. The buffer time interval may be set initially for the video frame coding transmitting device, or may be set by a user or a related staff.

And S406, if the confirmation frame identification sent by the receiving end is received, taking the video frame to be confirmed corresponding to the confirmation frame identification as a reference frame.

Specifically, the video frame to be confirmed is a complete video frame, and is also the first complete video frame obtained by the receiving end from the sending end, and the video frame to be confirmed is a complete video frame stored in the receiving end and the sending end at the same time, so if the receiving end obtains the decodable video frame to be confirmed sent by the sending end, the video frame to be confirmed can be used as a reference frame. The receiving end can send the confirmation frame identifier corresponding to the reference frame to the sending end, so that the sending end carries out coding processing on the uncoded video frame in the video source based on the reference frame to obtain video coding data. If the transmitting end receives the acknowledgement frame identifier sent by the receiving end, the video frame to be acknowledged corresponding to the acknowledgement frame identifier can be obtained in the video frame buffer, and the video frame to be acknowledged can be used as a reference frame.

S408, releasing the video frames with the time stamps smaller than the time stamps corresponding to the reference frames in the video frame buffer.

Specifically, the sender receives the identification of the acknowledgement frame, and after taking the video frame to be acknowledged as the reference frame, the timestamp corresponding to the reference frame can be obtained, the timestamp corresponding to the reference frame is the timestamp corresponding to the reference frame in the video source, then the release processing is performed on the video frames in the video frame buffer, the timestamp of which is smaller than the timestamp corresponding to the reference frame, namely the release processing is performed on the video frames except the reference frame, and the video frames in the video frame buffer, the timestamp of which is smaller than the timestamp corresponding to the reference frame, are deleted and discarded.

S410, coding the uncoded video frames in the video source based on the reference frames to obtain video coding data.

And S412, the video coding data is sent to a receiving end.

The transmitting end may transmit the video encoded data to the receiving end. It can be understood that the transmitting end may sequentially encode the video frames in the video source according to the time sequence to obtain video encoded data, and transmit the video encoded data after the encoding process to the receiving end according to the time sequence.

It can be understood that after the reference frame is confirmed, the receiving end encodes the subsequent uncoded video frame of the video source based on the reference frame, and because the receiving end encodes the uncoded video frame based on the video frame difference data between the uncoded video frame and the reference frame, so as to obtain the video encoded data of the uncoded video frame, it can be understood that the longer the time interval between the uncoded video frame and the reference frame is, the larger the difference between the uncoded video frame and the reference frame is, the larger the video encoded data of the uncoded video frame is, and the larger the data transmission code rate between the transmitting end and the receiving end is, and the larger the network transmission pressure is. Therefore, the receiving end and the transmitting end can perform replacement and update processing on the reference frame at intervals of a switching time period, so that the fluency of video playing is further improved, wherein the switching time period can be the initial setting of the video frame coding transmitting device or the video frame coding receiving device, and can also be the setting of a user or related staff.

In the embodiment of the application, the second video frame is encoded based on the first video frame to obtain the encoded data to be confirmed, and the second video frame is stored in the video frame buffer, so that the encoded data to be confirmed, which is transmitted before, is prevented from being lost due to the fact that the network signal is too weak, and the fault tolerance rate is improved. And if the confirmation frame identifier sent by the receiving end is received, taking the video frame to be confirmed corresponding to the confirmation frame identifier as a reference frame, wherein the video frame to be confirmed is a video frame which can be decoded by the receiving end, releasing the video frame with the time stamp smaller than the time stamp corresponding to the reference frame in the video frame buffer, and releasing the video frame buffer to reduce the memory pressure of the sending end. And carrying out coding processing on the uncoded video frames in the video source based on the reference frames to obtain video coding data, and sending the video coding data to the receiving end. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved. And the reference frames can be replaced and updated every other switching time period, so that the overlarge data transmission code rate between the sending end and the receiving end is avoided, the network transmission pressure is reduced, and the fluency of video playing is further improved.

Referring to fig. 7, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 7, the embodiment of the present application describes a method for transmitting video frame codes based on a receiving end side, and the method may include the following steps S502 to S506.

S502, if a decodable video frame to be confirmed sent by a sending end is obtained, the video frame to be confirmed is taken as a reference frame.

S504, the identification of the confirmation frame corresponding to the reference frame is sent to the sending end.

S506, receiving the video coding data sent by the sending end, and decoding the video coding data based on the reference frame to obtain a video frame of the video source.

Specifically, the receiving end may receive the video encoded data sent by the sending end, and may perform decoding processing on the video frame encoded data based on the reference frame, so as to obtain a video frame of the video source. After the receiving end obtains the video frame of the video source, the video frame can be played.

In the embodiment of the application, if a decodable video frame to be confirmed sent by a sending end is obtained, the video frame to be confirmed is used as a reference frame, a confirmation frame identifier corresponding to the reference frame is sent to the sending end, so that the sending end encodes an uncoded video frame in a video source based on the reference frame to obtain video encoded data, the video encoded data sent by the sending end is received, and the video encoded data is decoded based on the reference frame to obtain the video frame of the video source. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved.

Referring to fig. 8, a flowchart of a video frame coding transmission method is provided in an embodiment of the present application. As shown in fig. 8, the embodiment of the present application describes a video frame coding transmission method based on a receiving end side, and the method may include the following steps S602 to S610.

S602, receiving coded data to be confirmed sent by a sending end.

Specifically, before the receiving end receives the decodable video frame to be confirmed, the transmitting end can encode the video frame based on the previous frame video frame to generate the encoded data to be confirmed because of no reference frame, and then the transmitting end can transmit the encoded data to be confirmed to the receiving end. The receiving end can receive the coded data to be confirmed sent by the sending end.

S604, if the decoding processing of the coded data to be confirmed is successfully completed, obtaining a video frame to be confirmed in the coded data to be confirmed.

S606, if the decodable video frame to be confirmed sent by the sending end is obtained, the video frame to be confirmed is taken as a reference frame.

And S608, transmitting the identification of the acknowledgement frame corresponding to the reference frame to the transmitting end.

S610, receiving video coding data sent by a sending end, and decoding the video coding data based on a reference frame to obtain a video frame of a video source.

In the embodiment of the application, the to-be-confirmed encoded data sent by the sending end is received, if the decoding processing of the to-be-confirmed encoded data is successfully completed, the to-be-confirmed video frame in the to-be-confirmed encoded data is obtained, if the receiving end obtains the decodable to-be-confirmed video frame sent by the sending end, the to-be-confirmed video frame is used as a reference frame, and the confirmation frame identification corresponding to the reference frame is sent to the sending end, so that the sending end carries out the encoding processing on the uncoded video frame in the video source based on the reference frame to obtain the video encoded data, the video encoded data sent by the sending end is received, and the receiving end carries out the decoding processing on the video encoded data based on the reference frame to obtain the video frame of the video source. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved. And the reference frames can be replaced and updated every other switching time period, so that the overlarge data transmission code rate between the sending end and the receiving end is avoided, the network transmission pressure is reduced, and the fluency of video playing is further improved.

The following describes in detail a video frame coding transmitting apparatus according to an embodiment of the present application with reference to fig. 9 to fig. 10. It should be noted that, the video frame coding transmitting apparatus in fig. 9 to fig. 10 is used to execute the method of the embodiment shown in fig. 5 and fig. 6, and for convenience of explanation, only the portion relevant to the embodiment of the present application is shown, and specific technical details are not disclosed, please refer to the embodiment shown in fig. 5 and fig. 6 of the present application.

Referring to fig. 9, a schematic diagram of a video frame coding transmitting apparatus according to an exemplary embodiment of the present application is shown. The video frame coding transmission means may be implemented as all or part of the apparatus by software, hardware or a combination of both. The apparatus 1 comprises a frame identification receiving module 11, an encoding processing module 12 and an encoded data transmitting module 13.

The frame identifier receiving module 11 is configured to, if receiving an acknowledgement frame identifier sent by a receiving end, take a video frame to be acknowledged corresponding to the acknowledgement frame identifier as a reference frame; the video frame to be confirmed is a video frame which can be decoded by the receiving end;

the encoding processing module 12 is configured to perform encoding processing on an uncoded video frame in the video source based on the reference frame, so as to obtain video encoding data;

And the coded data transmitting module 13 is used for transmitting the video coded data to the receiving end.

In this embodiment, if the acknowledgement frame identifier sent by the receiving end is received, a video frame to be acknowledged corresponding to the acknowledgement frame identifier is used as a reference frame, where the video frame to be acknowledged is a video frame that can be decoded by the receiving end, encoding processing is performed on an uncoded video frame in a video source based on the reference frame, so as to obtain video encoded data, and the video encoded data is sent to the receiving end. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved.

Referring to fig. 10, a schematic diagram of a video frame coding transmitting apparatus according to an exemplary embodiment of the present application is shown. The video frame coding transmission means may be implemented as all or part of the apparatus by software, hardware or a combination of both. The device 1 comprises an acknowledgement data transmitting module 14, a frame identification receiving module 11, a buffer releasing module 15, a reference frame switching module 16, an encoding processing module 12 and an encoding data transmitting module 13.

The acknowledgement data sending module 14 is configured to perform encoding processing on a second video frame based on a first video frame, to obtain encoded data to be acknowledged, where the first video frame is a video frame preceding the second video frame in a video source;

and storing the second video frame to the video frame buffer, and sending the coded data to be confirmed to a receiving end.

the buffer release module 15 is configured to release the video frame in the video frame buffer, where the time stamp of the video frame is smaller than the time stamp corresponding to the reference frame;

the reference frame switching module 16 is configured to acquire a switching video frame corresponding to the switching frame identifier if the switching frame identifier sent by the receiving end is received;

and replacing the video frame to be confirmed with the switching video frame, and confirming the video frame to be confirmed as the reference frame.

Optionally, the encoding processing module 12 is specifically configured to determine a third video frame in the video source, where the third video frame is an uncoded video frame in the video source;

determining video frame difference data between the third video frame and the reference frame;

and carrying out coding processing on the third video frame based on the video frame difference data to obtain video coding data of the third video frame.

In this embodiment, the second video frame is encoded based on the first video frame to obtain the encoded data to be confirmed, and the second video frame is stored in the video frame buffer, so that the encoded data to be confirmed, which is sent before, is prevented from being lost due to the fact that the network signal is too weak, and the fault tolerance rate is improved. And if the confirmation frame identifier sent by the receiving end is received, taking the video frame to be confirmed corresponding to the confirmation frame identifier as a reference frame, wherein the video frame to be confirmed is a video frame which can be decoded by the receiving end, releasing the video frame with the time stamp smaller than the time stamp corresponding to the reference frame in the video frame buffer, and releasing the video frame buffer to reduce the memory pressure of the sending end. And carrying out coding processing on the uncoded video frames in the video source based on the reference frames to obtain video coding data, and sending the video coding data to the receiving end. The complete video frame obtained by the receiving end is used as the reference frame, and the reference frame is used for encoding and decoding the video frame, so that the problem that the subsequent video frame cannot be normally decoded and played due to packet loss and the like is avoided, and the fluency of video playing is improved. And the reference frames can be replaced and updated every other switching time period, so that the overlarge data transmission code rate between the sending end and the receiving end is avoided, the network transmission pressure is reduced, and the fluency of video playing is further improved.

Referring to fig. 11, a block diagram of an electronic device according to an exemplary embodiment of the present application is shown. The electronic device of the present application may include one or more of the following components: processor 110, memory 120, input device 130, output device 140, and bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected by a bus 150.

Processor 110 may include one or more processing cores. The processor 110 connects various parts within the overall electronic device using various interfaces and lines, performs various functions of the terminal 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and invoking data stored in the memory 120. Alternatively, the processor 110 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 110 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user page, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 110 and may be implemented solely by a single communication chip.

The Memory 120 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (ROM). Optionally, the memory 120 includes a Non-transitory computer readable medium (Non-Transitory Computer-Readable Storage Medium). Memory 120 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, which may be an Android (Android) system, including an Android system-based deep development system, an IOS system developed by apple corporation, including an IOS system-based deep development system, or other systems, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like.

Memory 120 may be divided into an operating system space in which the operating system runs and a user space in which native and third party applications run. In order to ensure that different third party application programs can achieve better operation effects, the operating system allocates corresponding system resources for the different third party application programs. However, the requirements of different application scenarios in the same third party application program on system resources are different, for example, under the local resource loading scenario, the third party application program has higher requirement on the disk reading speed; in the animation rendering scene, the third party application program has higher requirements on the GPU performance. The operating system and the third party application program are mutually independent, and the operating system often cannot timely sense the current application scene of the third party application program, so that the operating system cannot perform targeted system resource adaptation according to the specific application scene of the third party application program.

In order to enable the operating system to distinguish specific application scenes of the third-party application program, data communication between the third-party application program and the operating system needs to be communicated, so that the operating system can acquire current scene information of the third-party application program at any time, and targeted system resource adaptation is performed based on the current scene.

The input device 130 is configured to receive input instructions or data, and the input device 130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output instructions or data, and the output device 140 includes, but is not limited to, a display device, a speaker, and the like. In one example, the input device 130 and the output device 140 may be combined, and the input device 130 and the output device 140 are touch display screens.

The touch display screen may be designed as a full screen, a curved screen, or a contoured screen. The touch display screen may also be designed as a combination of a full screen and a curved screen, and the combination of a special-shaped screen and a curved screen, which is not limited in the embodiment of the present application.

In addition, those skilled in the art will appreciate that the configuration of the electronic device shown in the above-described figures does not constitute a limitation of the electronic device, and the electronic device may include more or less components than illustrated, or may combine certain components, or may have a different arrangement of components. For example, the electronic device further includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a wireless fidelity (Wireless Fidelity, wiFi) module, a power supply, and a bluetooth module, which are not described herein.

In the electronic device shown in fig. 11, the processor 110 may be configured to invoke the video frame coding transmission application stored in the memory 120, and specifically perform the following operations:

and sending the video coding data to the receiving end.

In one embodiment, before executing the to-be-acknowledged video frame corresponding to the acknowledgement frame identifier as the reference frame if the acknowledgement frame identifier sent by the receiving end is received, the processor 110 further executes the following operations:

encoding a second video frame based on a first video frame to obtain encoded data to be confirmed, wherein the first video frame is a previous video frame of the second video frame in a video source;

In one embodiment, the processor 110 further performs the following operations after performing the identification of the corresponding video frame to be acknowledged as a reference frame:

And releasing the video frames with the time stamp smaller than the time stamp corresponding to the reference frame in the video frame buffer.

In one embodiment, after performing the identifying the video frame to be acknowledged corresponding to the acknowledgement frame as a reference frame, the processor 110 further performs the following operations:

if the switching frame identification sent by the receiving end is received, acquiring a switching video frame corresponding to the switching frame identification;

In one embodiment, the processor 110, when performing encoding processing on an uncoded video frame in the video source based on the reference frame to obtain video encoded data, specifically performs the following operations:

determining a third video frame in a video source, the third video frame being an uncoded video frame in the video source;

The following describes in detail a video frame coding transmitting apparatus according to an embodiment of the present application with reference to fig. 12 to 13. It should be noted that, the video frame coding transmitting apparatus in fig. 12 to fig. 13 is used to execute the method of the embodiment shown in fig. 7 and fig. 8, and for convenience of explanation, only the portion relevant to the embodiment of the present application is shown, and specific technical details are not disclosed, please refer to the embodiment shown in fig. 7 and fig. 8 of the present application.

Referring to fig. 12, a schematic diagram of a video frame coding receiving device according to an exemplary embodiment of the present application is shown. The video frame coding reception apparatus may be implemented as all or part of the apparatus by software, hardware, or a combination of both. The apparatus 2 comprises a reference frame acquisition module 21, a frame identification transmission module 22 and a decoding processing module 23.

The reference frame obtaining module 21 is configured to, if a decodable video frame to be confirmed sent by the sending end is obtained, take the video frame to be confirmed as a reference frame;

the frame identifier sending module 22 is configured to send a confirmation frame identifier corresponding to the reference frame to the sending end, so that the sending end performs encoding processing on an uncoded video frame in a video source based on the reference frame to obtain video encoded data;

And the decoding processing module 23 is configured to receive the video encoded data sent by the sending end, and perform decoding processing on the video encoded data based on the reference frame, so as to obtain a video frame of the video source.

Referring to fig. 13, a schematic diagram of a video frame coding receiving device according to an exemplary embodiment of the present application is shown. The video frame coding reception apparatus may be implemented as all or part of the apparatus by software, hardware, or a combination of both. The apparatus 2 comprises an acknowledgement data receiving module 24, a reference frame acquisition module 21, a frame identification transmitting module 22, a handover identification transmitting module 25 and a decoding processing module 23.

A confirmation data receiving module 24, configured to receive the coded data to be confirmed sent by the sending end;

if the decoding processing of the coded data to be confirmed is successfully completed, acquiring a video frame to be confirmed in the coded data to be confirmed;

and if the decoding processing of the coded data to be confirmed cannot be completed, discarding the coded data to be confirmed.

a switching identifier sending module 25, configured to determine a switching video frame of the switching period of time interval between the video frames to be confirmed, and obtain a switching frame identifier corresponding to the switching video frame;

replacing the video frame to be confirmed with the switching video frame, and confirming the video frame to be confirmed as the reference frame;

and sending the switching frame identification to the sending end.

Optionally, the switch identifier sending module 25 is specifically configured to send the switch frame identifier to the sending end by using a redundant feedback manner.

Referring to fig. 14, a block diagram of an electronic device according to an exemplary embodiment of the present application is shown. The electronic device of the present application may include one or more of the following components: processor 210, memory 220, input device 230, output device 240, and bus 250. The processor 210, memory 220, input device 230, and output device 240 may be connected by a bus 250.

Processor 210 may include one or more processing cores. The processor 210 utilizes various interfaces and lines to connect various portions of the overall electronic device, perform various functions of the electronic device 200, and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 220, and invoking data stored in the memory 220. Alternatively, the processor 210 may be implemented in at least one hardware form of DSP, FPGA, PLA. The processor 210 may integrate one or a combination of several of a CPU, GPU, modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for being responsible for rendering and drawing of display content; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 210 and may be implemented solely by a single communication chip.

The memory 220 may include RAM or ROM. Optionally, the memory 220 includes a non-transitory computer readable medium. Memory 220 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 220 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, which may be an Android (Android) system, including an Android system-based deep development system, an IOS system developed by apple corporation, including an IOS system-based deep development system, or other systems, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The storage data area may also store data created by the electronic device in use, such as phonebooks, audiovisual data, chat log data, and the like.

Memory 220 may be divided into an operating system space in which the operating system runs and a user space in which native and third party applications run. In order to ensure that different third party application programs can achieve better operation effects, the operating system allocates corresponding system resources for the different third party application programs. However, the requirements of different application scenarios in the same third party application program on system resources are different, for example, under the local resource loading scenario, the third party application program has higher requirement on the disk reading speed; in the animation rendering scene, the third party application program has higher requirements on the GPU performance. The operating system and the third party application program are mutually independent, and the operating system often cannot timely sense the current application scene of the third party application program, so that the operating system cannot perform targeted system resource adaptation according to the specific application scene of the third party application program.

The input device 230 is configured to receive input instructions or data, and the input device 230 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 240 is used to output instructions or data, and the output device 240 includes, but is not limited to, a display device, a speaker, and the like. In one example, the input device 230 and the output device 240 may be combined, and the input device 230 and the output device 240 are touch display screens.

In addition, those skilled in the art will appreciate that the configuration of the electronic device shown in the above-described figures does not constitute a limitation of the electronic device, and the electronic device may include more or less components than illustrated, or may combine certain components, or may have a different arrangement of components. For example, the electronic device further includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a WiFi module, a power supply, and a bluetooth module, which are not described herein.

In the electronic device shown in fig. 14, the processor 210 may be configured to invoke the video frame coding transmission application program stored in the memory 220, and specifically perform the following operations:

In one embodiment, before executing the step of obtaining the decodable video frame sent by the sending end and confirming the decodable video frame as the reference frame, the processor 210 further executes the following operations:

receiving coded data to be confirmed sent by a sending end;

In one embodiment, after performing the transmission of the acknowledgement frame identifier corresponding to the reference frame to the transmitting end, the processor 210 further performs the following operations:

determining a switching video frame of a switching time period at intervals of the video frame to be confirmed, and acquiring a switching frame identifier corresponding to the switching video frame;

and sending the switching frame identification to the sending end.

In one embodiment, the processor 210, when executing the transmission of the switch frame identifier to the transmitting end, specifically executes the following operations:

and transmitting the switching frame identification to the transmitting end by adopting a redundant feedback mode.

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, or the like.

The foregoing disclosure is illustrative of the present application and is not to be construed as limiting the scope of the application, which is defined by the appended claims.

It should be noted that, information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals according to the embodiments of the present disclosure are all authorized by the user or are fully authorized by the parties, and the collection, use, and processing of relevant data is required to comply with relevant laws and regulations and standards of relevant countries and regions. For example, references in this specification to a video source, video frames of a video source, acknowledgement frame identification, etc. are all acquired with sufficient authorization.

Claims

1. A video frame coding transmission method applied to a transmitting end, the method comprising:

and sending the video coding data to the receiving end.

2. The method of claim 1, wherein if the acknowledgement frame identifier sent by the receiving end is received, before taking the video frame to be acknowledged corresponding to the acknowledgement frame identifier as the reference frame, the method further comprises:

3. The method of claim 2, the method further comprising, after using the acknowledgement frame identifying the corresponding video frame to be acknowledged as a reference frame:

4. The method of claim 1, after the identifying the corresponding video frame to be acknowledged as a reference frame, the method further comprising:

5. The method of claim 1, wherein the encoding the uncoded video frames in the video source based on the reference frames to obtain video encoded data comprises:

6. A video frame coding transmission method applied to a receiving end, the method comprising:

7. The method of claim 6, wherein if the decodable video frame sent by the sending end is obtained, before the decodable video frame is confirmed as the reference frame, further comprising:

receiving coded data to be confirmed sent by a sending end;

8. The method of claim 6, after the sending the acknowledgement frame identifier corresponding to the reference frame to the sending end, the method further comprises:

and sending the switching frame identification to the sending end.

9. The method of claim 8, the sending the handover frame identifier to the sender, comprising:

10. A video frame coding transmission device, the device comprising:

11. A video frame coding reception apparatus, the apparatus comprising:

12. A computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps of any one of claims 1 to 5 and 6 to 9.

13. A computer program product storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps of any of claims 1 to 5 and 6 to 9.

14. An electronic device includes: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to perform the method steps of any of claims 1-5 and 6-9.