WO2019134499A1

WO2019134499A1 - Method and device for labeling video frames in real time

Info

Publication number: WO2019134499A1
Application number: PCT/CN2018/121730
Authority: WO
Inventors: 张晓恬; 胡军; 潘思霁; 徐健钢; 尉苗苗
Original assignee: 亮风台（上海）信息科技有限公司
Priority date: 2018-01-05
Filing date: 2018-12-18
Publication date: 2019-07-11
Also published as: CN108401190A; CN108401190B

Abstract

The present application aims to provide a method and device for labeling video frames in real time. The method specifically comprises: sending a video stream to a second user equipment; receiving second frame related information of a second video frame intercepted by the second user equipment in the video stream; determining a first video frame corresponding to the second video frame in the video stream according to the second frame related information; receiving labeling operation information of the second video frame by the second user equipment; and displaying a corresponding labeling operation on the first video frame in real time according to the labeling operation information. According to the present application, labeling information is directly displayed in overlapping fashion on a non-coded or non-decoded video frame image sent by a video sender, and because the labeled video frame is not subjected to operations such as coding and decoding, the definition is high. Furthermore, this solution can also achieve real-time display of the label, so that the practicability is good, the interactivity is high, and the user experience and the broadband utilization rate are improved.

Description

Method and device for real-time annotation of video frames

This case requires the priority of CN 201810011908.0 and CN 201810409977.7

Technical field

The present application relates to the field of computers, and in particular, to a technique for real-time annotation of video frames.

Background technique

Nowadays, the universal video streaming means, the sender of the video stream encodes the video according to the encoding protocol and sends it to the video receiver through the network, and the receiver receives the corresponding video, and decodes the video, and takes a screenshot of the decoded video, and The image is marked on the screen, and the labeled image is encoded and sent to the video sender through the network, or the labeled image is sent to the server, and the video sender receives the image added by the server. The decoded image with the annotation information received by the sender has a certain loss of clarity, and the image with the annotation is transmitted from the receiver to the video sender through the network. The speed of the process depends on the transmission of the current network. The rate, and thus the transmission speed, is affected, and there is a delay phenomenon, which is not conducive to real-time interaction between the two parties.

Summary of the invention

It is an object of the present application to provide a method and apparatus for real-time annotation of video frames.

According to an aspect of the present application, a method for real-time annotation of a video frame on a first user equipment side is provided, the method comprising:

Sending a video stream to the second user equipment;

Receiving second frame related information of the second video frame intercepted by the second user equipment in the video stream;

Determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Receiving, by the second user equipment, labeling operation information of the second video frame;

And correspondingly presenting a corresponding labeling operation on the first video frame according to the labeling operation information.

According to another aspect of the present application, a method for real-time annotation of a video frame on a second user equipment side is provided, the method comprising:

Receiving a video stream sent by the first user equipment;

Transmitting second frame related information of the intercepted second video frame to the first user equipment according to a screenshot operation of the user in the video stream;

Obtaining, by the user, the labeling operation information of the second video frame;

Sending the labeling operation information to the first user equipment.

According to still another aspect of the present application, a method for real-time annotation of a video frame on a third user equipment side is provided, the method comprising:

Receiving a video stream that is sent by the first user equipment to the second user equipment and the third user equipment;

Determining, according to the second frame related information, a third video frame corresponding to the second video frame in the video stream;

And correspondingly presenting a corresponding labeling operation on the third video frame according to the labeling operation information.

According to still another aspect of the present application, a method for real-time annotation of a video frame on a network device side is provided, the method comprising:

Receiving and forwarding a video stream sent by the first user equipment to the second user equipment;

Forwarding the second frame related information to the first user equipment;

Forwarding the labeling operation information to the first user equipment.

According to an aspect of the present application, a method for real-time annotation of a video frame is provided, wherein the method includes:

The first user equipment sends a video stream to the second user equipment;

Receiving, by the second user equipment, the video stream, and sending, according to a screenshot operation of the user in the video stream, second frame related information of the intercepted second video frame to the first user equipment;

Receiving, by the first user equipment, the second frame related information, and determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Obtaining, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the first user equipment;

The first user equipment receives the labeling operation information, and presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information.

According to another aspect of the present application, a method for real-time annotation of a video frame is provided, wherein the method includes:

The first user equipment sends a video stream to the network device;

Receiving, by the network device, the video stream, and forwarding the video stream to a second user equipment;

Receiving, by the second user equipment, the video stream, and sending, according to a screenshot operation of the user in the video stream, second frame related information of the intercepted second video frame to the network device;

Receiving, by the network device, the second frame related information, and forwarding the second frame related information to the first user equipment;

Obtaining, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the network device;

The network device receives the labeling operation information of the second video frame by the second user equipment, and forwards the labeling operation information to the first user equipment;

According to still another aspect of the present application, a method for real-time annotation of a video frame is provided, wherein the method includes:

The first user equipment sends a video stream to the second user equipment and the third user equipment;

Transmitting, by the second user equipment, the second frame related information of the intercepted second video frame to the first user equipment and the third user equipment according to a screenshot operation of the user in the video stream;

And acquiring, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the first user equipment and the third user equipment;

Receiving, by the first user equipment, the second frame related information, determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream, and receiving the labeling operation information And displaying a corresponding labeling operation in real time on the first video frame according to the labeling operation information;

Receiving, by the third user equipment, the video stream, receiving second frame related information of the second video frame, receiving the labeling operation information, determining, according to the second frame related information, the video stream and the The third video frame corresponding to the second video frame presents a corresponding labeling operation in real time on the third video frame according to the labeling operation information.

The first user equipment sends a video stream to the network device;

Receiving, by the network device, the video stream, and forwarding the video stream to the second user equipment and the third user equipment;

Receiving, by the network device, the second frame related information, and forwarding the second frame related information to the first user equipment and the third user equipment;

The network device receives the labeling operation information of the second video frame by the second user equipment, and forwards the labeling operation information to the first user equipment and the third user equipment;

The first user equipment receives the labeling operation information, and presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information;

According to an aspect of the present application, a first user equipment for real-time annotation of a video frame is provided, the device comprising:

a video sending module, configured to send a video stream to the second user equipment;

a frame information receiving module, configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream;

a video frame determining module, configured to determine, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

An annotation receiving module, configured to receive the labeling operation information of the second video frame by the second user equipment;

And an annotation presentation module, configured to present a corresponding annotation operation on the first video frame in real time according to the annotation operation information.

According to another aspect of the present application, a second user equipment for real-time annotation of a video frame is provided, the apparatus comprising:

a video receiving module, configured to receive a video stream sent by the first user equipment;

a frame information determining module, configured to send second frame related information of the intercepted second video frame to the first user equipment according to a screenshot operation of the user in the video stream;

An annotation obtaining module, configured to acquire the labeling operation information of the second video frame by the user;

And an annotation sending module, configured to send the labeling operation information to the first user equipment.

According to still another aspect of the present application, a third user equipment for real-time annotation of a video frame is provided, the device comprising:

a third video receiving module, configured to receive a video stream that is sent by the first user equipment to the second user equipment and the third user equipment;

a third frame information receiving module, configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream;

a third video frame determining module, configured to determine, according to the second frame related information, a third video frame corresponding to the second video frame in the video stream;

a third label receiving module, configured to receive the labeling operation information of the second video frame by the second user equipment;

The third rendering module is configured to present a corresponding labeling operation on the third video frame in real time according to the labeling operation information.

According to still another aspect of the present application, a network device for real-time annotation of a video frame is provided, the device comprising:

a video forwarding module, configured to receive and forward a video stream sent by the first user equipment to the second user equipment;

a frame information forwarding module, configured to forward the second frame related information to the first user equipment;

And an annotation forwarding module, configured to forward the labeling operation information to the first user equipment.

In accordance with an aspect of the present application, a system for real-time annotation of video frames is provided, the system comprising a first user device as described above and a second user device as described above.

In accordance with another aspect of the present application, a system for real-time annotation of video frames is also provided, including a first user device as described above, a second user device as described above, and a network device as described above.

According to an aspect of the present application, there is provided a system for real-time annotation of a video frame, the system comprising a first user device as described above, a second user device as described above, and a third user as described above device.

According to an aspect of the present application, there is provided a system for real-time annotation of a video frame, the system comprising a first user device as described above, a second user device as described above, a third user as described above Equipment and network equipment as described above.

According to an aspect of the present application, there is provided a computer readable medium comprising instructions that, when executed, cause a system to:

Sending a video stream to the second user equipment;

According to another aspect of the present application, there is also provided a computer readable medium comprising instructions that, when executed, cause a system to:

Receiving a video stream sent by the first user equipment;

Sending the labeling operation information to the first user equipment.

According to still another aspect of the present application, there is also provided a computer readable medium comprising instructions that, when executed, cause a system to:

Forwarding the second frame related information to the first user equipment;

Forwarding the labeling operation information to the first user equipment.

Compared with the prior art, the present application determines a video sender's unencoded video frame image according to the video receiver's screenshot and the corresponding video frame related information by buffering a certain video frame on the video sender, and the video is The annotation information of the receiver on the screenshot is transmitted to the video sender in real time. The annotation is displayed on the video frame image corresponding to the video sender in real time, so the sender can observe the labeling process of the video receiver in real time, and the resolution is high because the labeled video frame is not encoded and so on; further, the scheme It also enables real-time display of annotations, with good practicability, strong interactivity, and improved user experience and broadband utilization. Moreover, the video sender can send the video frame to the video receiver after determining the uncoded video frame, and the video receiver can also mark the high quality video frame, thereby greatly improving the user experience.

DRAWINGS

Other features, objects, and advantages of the present application will become more apparent from the detailed description of the accompanying drawings.

1 shows a system topology diagram for real-time annotation of video frames in accordance with an embodiment of the present application;

2 shows a flow chart of a method for real-time annotation of video frames at a first user equipment end according to an aspect of the present application;

3 is a flow chart showing a method for real-time annotation of video frames on a second user equipment side according to another aspect of the present application;

4 is a flowchart of a method for real-time annotation of a video frame at a third user equipment end according to still another aspect of the present application;

FIG. 5 is a flowchart of a method for real-time annotation of video frames on a network device side according to still another aspect of the present application; FIG.

6 shows a system method diagram for real-time annotation of video frames in accordance with an aspect of the present application;

7 shows a system method diagram for real-time annotation of video frames in accordance with another aspect of the present application;

8 shows a schematic diagram of a first user equipment for real-time annotation of video frames in accordance with an aspect of the present application;

9 shows a schematic diagram of a second user equipment for real-time annotation of video frames in accordance with another aspect of the present application;

10 shows a schematic diagram of a third user equipment for real-time annotation of video frames in accordance with another aspect of the present application;

11 shows a schematic diagram of a network device for real-time annotation of video frames in accordance with still another aspect of the present application;

FIG. 12 illustrates an exemplary system that can be used to implement various embodiments described in this application.

The same or similar reference numerals in the drawings denote the same or similar components.

Detailed ways

The present application is further described in detail below with reference to the accompanying drawings.

In a typical configuration of the present application, the terminal, the device of the service network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium.

Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, A magnetic tape cartridge, magnetic tape storage or other magnetic storage device or any other non-transportable medium that can be used to store information that can be accessed by a computing device.

The device referred to in the present application includes but is not limited to a user equipment, a network device, or a device formed by integrating a user equipment and a network device through a network. The user equipment includes, but is not limited to, any mobile electronic product that can perform human-computer interaction with the user (for example, human-computer interaction through a touchpad), such as a smart phone, a tablet computer, etc., and the mobile electronic product can be operated by any operation. System, such as android operating system, iOS operating system, etc. The network device includes an electronic device capable of automatically performing numerical calculation and information processing according to an instruction set or stored in advance, and the hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), and programmable logic. Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Digital Signal Processors (DSPs), embedded devices, and more. The network device includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a plurality of servers; wherein the cloud is composed of a large number of computers or network servers based on Cloud Computing. Among them, cloud computing is a kind of distributed computing, a virtual supercomputer composed of a group of loosely coupled computers. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), and the like. Preferably, the device may also be a program running on the user equipment, the network device, or the user equipment and the network device, the network device, the touch terminal, or the network device and the touch terminal integrated through the network.

Of course, those skilled in the art should understand that the above-mentioned devices are only examples, and other existing or future devices may be applicable to the present application, and are also included in the scope of the present application, and are hereby incorporated by reference. this.

In the description of the present application, the meaning of "plurality" is two or more unless specifically defined otherwise.

FIG. 1 shows a typical scenario of the present application. The first user equipment receives the annotation information sent by the second user equipment and performs local storage while performing video communication with the second user equipment and the third user equipment. The uncoded video frame presents the annotation information in real time. The process may be performed by the first user equipment and the second user equipment, or may be completed by the first user equipment, the second user equipment, and the network equipment, and may also be performed by the first user equipment, the second user equipment, and the first user equipment. The three user equipments are completed, and may also be completed by the first user equipment, the second user equipment, the third user equipment, and the network equipment. Here, the first user equipment, the second user equipment, and the third user equipment are any electronic devices that can record and send video, such as smart glasses, mobile phones, tablets, notebooks, smart watches, etc., where the first user equipment is The following embodiments are described by using the smart glasses, the second user equipment, and the third user equipment as a tablet. Those skilled in the art should understand that the embodiments are also applicable to other user equipments such as mobile phones, notebooks, and smart watches.

2 illustrates a method for real-time annotation of a video frame at a first user equipment end according to an aspect of the present application, wherein the method includes step S11, step S12, step S13, step S14, and step S15. In step S11, the first user equipment end sends a video stream to the second user equipment; in step S12, the first user equipment receives the second video frame intercepted by the second user equipment in the video stream. Two-frame related information; in step S13, the first user equipment determines, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream; in step S14, the first The user equipment receives the labeling operation information of the second video frame by the second user equipment; in step S15, the first user equipment presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information. .

Specifically, in step S11, the first user equipment end sends a video stream to the second user equipment. For example, the first user equipment establishes a communication connection with the second user equipment through a wired or wireless network, and the first user equipment encodes the video stream to the second user equipment by using a video communication manner.

In step S12, the first user equipment receives second frame related information of the second video frame intercepted by the second user equipment in the video stream. For example, the second user equipment determines the second frame related information of the video frame corresponding to the screen capture screen based on the screen capture operation of the second user, and then the first user equipment receives the second frame related to the second video frame sent by the second user equipment. Information, wherein the second frame related information includes, but is not limited to, second video frame identification information, a second video frame encoding start time, a second video frame decoding end time, and a second video frame codec and transmission total duration information. .

In step S13, the first user equipment determines, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream. For example, the first user equipment locally stores a period of time or a certain number of transmitted unencoded video frames, and the first user equipment stores the uncoded locally according to the second frame related information sent by the second user equipment. In the video frame, the uncoded first video frame corresponding to the screen capture is determined.

In step S14, the first user equipment receives the labeling operation information of the second video frame by the second user equipment. For example, the second user equipment generates corresponding labeling operation information based on the labeling operation of the second user, and sends the labeling operation information to the first user equipment in real time, and the first user receives the labeling operation information.

In step S15, the first user equipment presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information. For example, the first user equipment displays a corresponding labeling operation in real time on the first video frame based on the received labeling operation information, such as displaying the first video frame in the form of a small window in the current interface, and corresponding to the first video frame in the current interface. The location presents a corresponding annotation operation at a rate of one frame every 50 ms.

For example, user A holds smart glasses, user B holds tablet B, and smart glasses and tablet B establish video communication via wired or wireless network. Smart glasses encode the currently collected picture and send it to tablet B, and cache. A length of time or a certain number of video frames. After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture image, and sends the second frame related information corresponding to the second video frame to the smart glasses. The second frame related information includes, but is not limited to, a second video frame identification information, a second video frame encoding start time, a second video frame decoding end time, and a second video frame codec and transmission total duration information. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. The tablet computer B generates the real-time labeling operation information according to the labeling operation of the user B, and sends the labeling operation information to the smart glasses in real time. After receiving the labeling operation information, the smart glasses display the corresponding unencoded in the preset area of the smart glasses. The first video frame, and the corresponding labeling operation is presented in real time at a position corresponding to the first video frame.

It should be understood by those skilled in the art that the content of the second frame related information in the foregoing embodiment is only an example, and the content of other second frame related information that appears in the prior art or in the future, if applicable to the present application, also belongs to The scope of protection of this application is hereby incorporated by reference.

In some embodiments, the method further includes step S16 (not shown). In step S16, the first user equipment stores the video frame in the video stream; wherein, in step S13, the first user equipment determines, according to the second frame related information, the stored video frame from the first The first video frame corresponding to the two video frames. For example, the first user equipment sends the video stream to the second user equipment, and locally stores a period of time or a certain number of uncoded video frames, wherein a period of time or a certain number may be a preset fixed value, The threshold may be dynamically adjusted according to the network condition or the transmission rate; then, the first user equipment determines the corresponding uncompiled in the locally stored video frame based on the second frame related information of the second video frame sent by the second user equipment. The decoded first video frame. In other embodiments, the stored video frame meets, but is not limited to, at least one of the following: a time interval between a transmission time of the stored video frame and a current time is less than or equal to a video frame storage duration threshold; the stored The cumulative number of video frames is less than or equal to a predetermined number of video frame storage thresholds.

The smart glasses send the collected images to the tablet B and store them for a period of time or a certain number of uncoded video frames locally, wherein the duration or the number may be a fixed value of the system or manually preset, such as A certain duration or a certain number of video frame thresholds obtained by statistical analysis of big data; a period of time or a certain number of video frames may also be a video frame threshold dynamically adjusted according to network conditions or transmission rates. The duration or number threshold of the dynamic adjustment may be determined according to the codec and the total duration information of the video frame, such as calculating the total length of the codec and the current video frame, and using the duration as a unit duration or within the duration. The number of transmitted video frames is taken as a unit number, and the dynamic video frame duration or number threshold is set with reference to the current unit duration or unit number. Here, the set predetermined or dynamic video frame storage duration threshold should be greater than or equal to one unit duration. Similarly, the set predetermined or dynamic video frame storage threshold should be greater than or equal to one unit number. Then, the smart glasses determine a corresponding uncoded first video frame in the stored video according to the second frame related information of the second video frame sent by the tablet B, wherein the stored video frame is sent and current. The interval of time is less than or equal to the video frame storage duration threshold, or the accumulated number of stored video frames is less than or equal to a predetermined video frame storage threshold.

In some embodiments, the method further includes step S17 (not shown). In step S17, the first user equipment acquires codec and total transmission duration information of the video frame in the video stream, and adjusts the video frame storage duration threshold or the video frame according to the codec and total transmission duration information. The number of storage thresholds. For example, the first user equipment records the encoding start time of each video frame, and after encoding, sends the video frame to the second user equipment, and the second user equipment receives and records each video frame decoding end time; subsequently, the second user equipment The video frame decoding end time is sent to the first user equipment, and the first user equipment calculates the codec and total transmission duration information of the current video frame based on the encoding start time and the decoding end time, or the second user equipment is based on the encoding start time and The decoding end time calculates the codec and total transmission duration information of the current video frame, and sends the codec and the total transmission duration information to the first user equipment. The first user equipment adjusts the video frame storage duration threshold or the video frame storage threshold according to the codec and the total transmission duration information. If the duration information is used as a unit time reference, setting a certain multiple of the video frame duration is The video frame stores a duration threshold; for example, the number of video frames that can be sent in the duration information is calculated according to the duration information and the rate at which the first user equipment sends the video frame, and the number is used as a unit number to set a certain multiple of the video frame. The number is used as a threshold for the number of video frames stored.

For example, the smart glasses record the i-th video frame encoding start time as T _si , and after encoding, send the video frame to the tablet B, and the tablet computer B receives and records the video frame decoding end time as T _ei . Subsequently, the tablet B sends the video frame decoding end time T _ei to the smart glasses, and the smart glasses calculate the decoding end time T _ei according to the received i-th video frame and the encoding start time T _si recorded locally _. The total length of codec and transmission of the video frame is T _i =T _ei -T _si ; the smart glasses can send the encoding start time T _si along with the video frame to the tablet B, and the tablet ethyl calculates the video frame at the encoding end time T _ei The total length of codec and transmission T _i =T _ei -T _si , and the total length of time T _{i of the} codec and transmission is returned to the smart glasses.

The smart glasses determine the duration of the video frame in the 1.3T _i time dynamically saved by the smart glasses according to the big data statistics according to the encoding and decoding of the i-th video frame and the total transmission time T _i . Or, dynamically adjust the magnification according to the network transmission rate, for example, setting the buffer duration threshold to (1+k)T _i , where k is a threshold adjusted according to network fluctuation, and if the network fluctuation is large, setting k to 0.5, the network When the fluctuation is small, set k to 0.2 or the like. As another example, the glasses according to the codec and transmit the i-th video frame a total duration T _i, and the frequency f of the video frame currently the glasses transmitted, calculates a the number N of video frames in the _i transport length T = T _i *f, and further determines that the threshold of the number of saved video frames is 1.3N, where N is the value obtained by taking the rounding method. Further, the smart glasses can dynamically adjust the magnification according to the current network transmission rate, such as setting the buffer number threshold (1+k)N, where k is a threshold adjusted according to network fluctuations, and if the network fluctuation is large, k is set. When it is 0.5 and the network fluctuation is small, set k to 0.2 or the like.

It should be understood by those skilled in the art that the content of the storage duration threshold and/or the storage threshold in the foregoing embodiment is only an example, and other storage duration thresholds and/or storage thresholds appearing in the prior art or in the future, if applicable, This application is also intended to be within the scope of the present application, and is hereby incorporated by reference.

In some embodiments, in step S11, the first user equipment sends the video stream and the frame identification information of the transmitted video frame in the video stream to the second user equipment; wherein, in step S13, the first user equipment is configured according to The second frame related information determines a first video frame corresponding to the second video frame in the video stream, where frame identification information of the first video frame corresponds to the second frame related information. The frame identification information of the video frame may be a codec time corresponding to the video frame, or may be a number corresponding to the video. In some embodiments, in step S11, the first user equipment performs encoding processing on a plurality of video frames to be transmitted, and sends frame corresponding information of the corresponding video stream and the transmitted video frame in the video stream to the second User equipment. For example, the first user equipment performs encoding processing on a plurality of video frames to be transmitted, and acquires an encoding start time of the plurality of video frames to be transmitted, and sends the multiple video frames and their encoding start time to the second. User equipment. In some embodiments, the frame identification information of the transmitted video frame in the video stream includes encoding start time information of the transmitted video frame.

For example, the smart glasses record the encoding start time of each video frame, and after encoding, send the video frame and the encoding start time of the transmitted video frame to the tablet computer B, wherein the transmitted video frame includes the video that is to be sent after the current encoding is completed. Frames and transmitted video frames. Here, the smart glasses may be sent at a certain time or at a certain interval of a certain video frame, and the coding start time of the transmitted video frame is sent to the tablet B, or the coding start time of the first video frame may be directly Video frames are sent to tablet B at the same time. The tablet computer operates the screen capture operation of the user B, determines the video frame corresponding to the screen capture screen, and sends the second frame related information of the corresponding second video frame to the smart glasses, wherein the second frame related information and the second frame identification information Correspondingly, including but not limited to at least one of the following: an encoding start time of the second video frame, a second video frame decoding end time, a second video frame codec and transmission total duration information, a second video frame corresponding number or Images, etc. The smart glasses receive the second frame related information, and determine correspondingly stored uncoded first video frames according to the second frame related information, such as according to the encoding start time of the second video frame, the second video frame decoding end time, Determining a coding start time of the uncoded first video frame corresponding to the second video frame, and determining a corresponding first video frame, and determining, by the second video frame, corresponding to the second video frame, The number directly determines the first video frame of the same number, and also determines the corresponding first video frame in the stored uncoded video frame by image recognition of the second video frame.

It should be understood by those skilled in the art that the content of the frame identification information in the foregoing embodiment is only an example, and the content of other frame identification information that appears in the prior art or in the future, if applicable to the present application, also belongs to the protection of the present application. The scope is hereby incorporated by reference.

In some embodiments, the method further includes step S18 (not shown). In step S18, the first user equipment presents the first video frame; wherein, in step S15, the first user equipment superimposes a corresponding labeling operation on the first video frame according to the labeling operation information. For example, the first user equipment determines the first video frame that has not been coded, and displays the first video frame in a preset position in the current interface or in a small window; subsequently, the first user equipment receives the annotation according to the real-time reception. The operation information is superimposed on the corresponding position of the first video frame to present a corresponding labeling operation.

For example, the smart glasses determine a corresponding uncoded first video frame according to the second frame related information sent by the tablet B, and display the first video frame at a preset position of the interface of the smart glasses. Then, the smart glasses receive the real-time annotation operation sent by the tablet B. The smart glasses determine the corresponding position of the annotation operation in the currently displayed first video frame, and present the current annotation operation in real time at the corresponding location.

In some embodiments, the method further includes step S19 (not shown). In step S19, the first user equipment sends the first video frame to the second user equipment as a preferred frame for presenting the labeling operation. For example, the first user equipment determines the first video frame that has not been coded, and sends the first video frame to the second user equipment for the second user equipment to present the first video frame of higher quality.

For example, the smart glasses determine a corresponding unencoded first video frame according to the second frame related information sent by the tablet B, and send the first video frame as a preferred frame to the tablet B, such as by lossless compression. Sending the first video frame to tablet B, or sending the first video frame to tablet B through lossy lossy compression, the lossy compression process guarantees the quality of the video frame locally buffered than the tablet end High enough. Tablet B receives the first video frame and presents the first video frame.

In some embodiments, in step S11, the first user equipment sends a video stream to the second user equipment and the third user equipment. For example, a communication connection is established between the first user equipment, the second user equipment, and the third user equipment, where the first user equipment is the current video frame sender, and the second user equipment and the third user equipment are the current video frame receiving. The first user equipment sends a video stream to the second user equipment and the third user equipment through the communication connection.

For example, user A holds smart glasses, user B holds tablet B, user C holds tablet C, smart glasses and tablet B, and tablet C establishes video communication through wired or wireless network, and smart glasses will be currently collected. The picture is encoded and sent to each tablet B and buffered for a period of time or a certain number of video frames. After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture image, and sends the second frame related information corresponding to the second video frame to the smart glasses and The tablet C, wherein the second frame related information includes but is not limited to: the second video frame identification information, the second video frame encoding start time, the second video frame decoding end time, and the second video frame codec and transmission total duration Information, etc. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. The tablet computer B generates real-time labeling operation information according to the labeling operation of the user B, and sends the labeling operation information to the smart glasses and the tablet computer C in real time. After receiving the labeling operation information, the smart glasses display the corresponding areas in the preset area of the smart glasses. The first video frame that is not coded, and the corresponding labeling operation is presented in real time at the position corresponding to the first video frame. Similarly, the tablet propyl receives the second frame related information and the labeled operation information, and finds the corresponding third video frame in the locally cached codec video library according to the second frame related information. And presenting a corresponding labeling operation in the third video frame based on the third video frame and the labeling operation information.

In some embodiments, the method further includes step S010 (not shown). In step S010, the first user equipment sends the first video frame as a preferred frame for presenting the labeling operation to the second user equipment and/or the third user equipment. For example, the first user equipment determines a corresponding first video frame in the locally cached video frame according to the second frame related information, and sends the first video frame to the second user equipment and/or the third user equipment. After the second user equipment and/or the third user equipment receives the uncoded first video frame, the first video frame is presented, and the second user and/or the third user may perform an annotation operation based on the first video frame.

For example, after the smart glasses determine the unencoded first video frame corresponding to the second video frame, the uncoded first video frame is sent to the tablet B through lossless compression or high quality compression. Tablet PC C, wherein Tablet PC B and Tablet PC C determine whether to obtain the first video frame according to the quality of the current communication network connection, or select the transmission mode of the first video frame according to the quality of the current communication network connection, such as good network quality When using lossless compression, high-quality compression is used when the network quality is poor.

In some embodiments, the first user equipment sends the first video frame and the second frame related information to the second user equipment and/or the third user equipment, where The first video frame is used as a preferred frame to present the annotation operation in the second user device or the third user device.

For example, after the smart glasses determine the first video frame that has not been coded, the first video frame and the second frame related information corresponding to the first video frame are sent to the tablet B and the tablet C. In some embodiments, the tablet ethyl unit performs a plurality of screen capture operations on the operation of the user B, and the tablet computer B determines the screen capture operation corresponding to the first video frame according to the second frame related information, for example, according to the second frame screenshot time. A screen capture operation corresponding to a video frame, and the second frame related information is presented in the first video frame. The tablet computer C receives the second frame related information and the first video frame, and presents the second frame related information in the window in which the first video frame is presented while presenting the first video frame.

FIG. 3 illustrates a method for real-time annotation of a video frame at a second user equipment end according to another aspect of the present application, wherein the method includes step S21, step S22, step S23, and step S24. In step S21, the second user equipment receives the video stream sent by the first user equipment; in step S22, the second user equipment sends the first user equipment to the first user equipment according to the user's screenshot operation in the video stream. Obtaining the second frame related information of the second video frame; in step S23, the second user equipment acquires the labeling operation information of the second video frame by the user; in step S24, the second user equipment The first user equipment sends the labeling operation information. For example, the second user equipment receives and presents the video stream sent by the first user equipment; and the second user equipment determines, according to the screen capture operation of the second user, the second video frame corresponding to the current screen capture, and the second video frame is The second frame related information is sent to the first user equipment. Then, the second user equipment generates the labeling operation information based on the labeling operation of the second user, and sends the labeling operation information to the first user equipment.

For example, user B holds tablet B, user A holds smart glasses, and tablet B and smart glasses communicate video over wired or wireless networks. The tablet computer B receives and presents the video stream sent by the smart glasses, and determines the second video frame corresponding to the screen capture screen according to the screen capture operation of the user B. Then, the tablet B sends the second frame related information corresponding to the second video frame to the smart glasses, and the smart glasses receive the second frame related information and determine a corresponding first video frame based on the second frame related information. The tablet ethyl unit generates the corresponding labeling operation information in the labeling operation of the user B, and sends the labeling operation information to the smart glasses in real time. The smart glasses present a first video frame at a preset position of the interface according to the first video frame and the labeled operation information, and present a corresponding labeling operation in real time in the corresponding position in the first video frame.

In some embodiments, in step S21, the second user equipment receives the video stream sent by the first user equipment, and the frame identification information of the transmitted video frame in the video stream; wherein the second frame related information And including at least one of the following: frame identification information of the second video frame; frame related information generated based on frame identification information of the second video frame. For example, the first user equipment sends the video stream to the second user equipment, and also sends the frame identifier information of the sent video frame in the video stream to the second user equipment, where the second user equipment receives the video stream, and the video stream The frame identification information of the transmitted video frame. The second user equipment determines, according to the screen capture operation of the second user, the second video frame corresponding to the current screen capture, and sends the second frame related information of the second video frame to the first user equipment, where the second video The second frame related information of the frame includes but is not limited to: frame identification information of the second video frame; frame related information generated based on the frame identification information of the second video frame.

For example, the smart glasses send the frame identification information corresponding to the video frames in the already transmitted video stream to the tablet B while transmitting the video stream. The tablet computer B detects the screen capture operation of the user B. Based on the screen of the current screen capture, it is determined that the screen capture screen corresponds to the second video frame, and the second frame related information corresponding to the second video frame is sent to the smart glasses, wherein the second video The frame related information includes, but is not limited to, frame identification information of the second video frame, frame related information generated based on the frame identification information of the second video frame, where the frame identification information of the second video frame may be the encoding of the video frame. The frame-related information generated based on the frame identification information of the second video frame may be the decoding end time of the video frame or the total length of the codec and the transmission time information, etc., at the start time or the number corresponding to the video frame.

It should be understood by those skilled in the art that the content of the second frame related information in the foregoing embodiment is only an example, and the content of other second frame related information existing in the prior art or in the future, if applicable to the present application, The scope of protection of the application is hereby incorporated by reference.

In some embodiments, the frame identification information includes encoding start time information of the second video frame. For example, the first user equipment performs an encoding process on the video frame, and sends the frame identification information of the corresponding video stream and the transmitted video frame in the video stream to the second user equipment, where the frame identification information of the video frame includes The encoding start time of the video frame. In some embodiments, the second frame related information includes decoding end time information and codec and transmission total duration information of the second video frame. The second user equipment receives and presents the video stream, and records a corresponding decoding end time, determines a corresponding second video frame based on the screen capture operation, and determines a corresponding edit according to the encoding start time and the decoding end time of the second video frame. Decode and transmit total duration information.

For example, the smart glasses record the encoding start time of each video frame, and after encoding, send the encoding start time of the video frame and the transmitted video frame to the tablet B. Tablet B receives and presents the video frame and records the decoding end time of the video frame. The screen is operated by the screen capture operation of the user E, and the corresponding second video frame is determined, and the codec and the total transmission duration information of the second video frame are determined according to the coding start time and the decoding end time corresponding to the second video frame. Then, the tablet B sends the second frame related information of the second video frame to the smart glasses, where the second frame related information includes, but is not limited to, an encoding start time of the second video frame, a codec of the second video frame, and Transfer total time information, etc.

In some embodiments, in step S23, the second user equipment acquires the labeling operation information of the second video frame by the user in real time; wherein, in step S24, the second user equipment sends the first user equipment to the first user equipment. The annotation operation is sent in real time. For example, the second user equipment acquires the corresponding labeling operation information in real time based on the operation of the second user, for example, collecting the corresponding labeling operation information at a certain time interval. Then, the second user equipment sends the obtained annotation operation information to the first user equipment in real time.

For example, the tablet computer B collects the labeling operation of the user B on the screen capture screen, for example, the user B draws a circle, an arrow, a text, a box and the like on the screen. Tablet B records the position and path of the marked brush. For example, through multiple points on the screen, the position corresponding to the corresponding point is obtained, and the position where the multiple points are connected is marked. Tablet PC B obtains the corresponding labeling operation in real time and sends it to the smart glasses in real time, such as collecting and sending labeling operations at a frequency of 50 ms.

In some embodiments, the method further includes step S25 (not shown). In step S25, the second user equipment receives the first video frame sent by the first user equipment, where the first video frame is used as a preferred frame for presenting the labeling operation, in the second video frame. Loading the first video frame to replace the second video frame in a display window, wherein the labeling operation is displayed on the first video frame. For example, the second user equipment determines a second video frame corresponding to the current screen capture, and sends the second frame related information of the second video frame to the first user equipment; the first user equipment is based on the second video frame. Determining, by the second frame, the uncoded first video frame corresponding to the second video frame, and sending the first video frame to the second user equipment, where the second user equipment receives and presents the first video frame, and Obtaining the labeling operation information of the first video frame by the second user.

For example, the tablet ethyl enters the screen capture mode by the operation of the user, determines the second video frame corresponding to the current picture, and sends the second frame related information of the second video frame to the smart glasses end, where the second frame related information includes However, it is not limited to: an encoding start time of the second video frame or a number corresponding to the video frame. The smart glasses determine a corresponding uncoded first video frame according to the second frame related information sent by the tablet B, and send the first video frame to the tablet B, such as by using lossless compression to the first video. The frame is sent to the tablet B, or the first video frame is sent to the tablet B through the lossy lossy compression. The lossy compression process guarantees a higher quality than the locally buffered video frame of the tablet. The tablet B receives and presents the first video frame, such as being presented in the form of a small window next to the current video, or displaying the first video frame on a large screen, presenting the current video in the form of a small window, and the like. Subsequently, the tablet computer B obtains the labeling operation information and the like regarding the first video frame according to the operation of the second user.

In some embodiments, the method further includes step S26 (not shown). In step S26, the second user equipment receives the first video frame and the second frame related information sent by the first user equipment, where the first video frame is used as a preferred frame for presenting the labeling operation. Determining, according to the second frame related information, that the first video frame is used to replace the second video frame, and loading the first video frame in a display window of the second video frame to replace the a second video frame, wherein the labeling operation is displayed on the first video frame.

For example, the tablet B receives the unencoded first video frame and the second frame related information sent by the smart glasses, wherein the second frame related information includes a screenshot time of the second video frame and a video frame of the second video frame. Number, etc. In some embodiments, the tablet ethyl unit performs a plurality of screen capture operations on the operation of the user B, and the tablet computer B determines the screen capture operation corresponding to the first video frame according to the second frame related information, for example, according to the second frame screenshot time. A screen capture operation corresponding to a video frame, and the second frame related information is presented in the first video frame. The tablet computer B determines the current corresponding screen capture operation according to the second frame related information, and presents the window in the form of a small window next to the current video, or displays the first video frame on a large screen, and presents the current video in the form of a small window, etc. When the first video frame is presented, the tablet B presents the second frame related information in the first video frame presented, such as the screenshot time of the frame time in the first video frame or the frame number of the frame in the video frame. Wait. Subsequently, the tablet computer B obtains the labeling operation information and the like regarding the first video frame according to the operation of the second user.

In some embodiments, in step S21, the second user equipment receives the video stream sent by the first user equipment to the second user equipment and the third user equipment; wherein, in step S24, the second user equipment The first user equipment and the third user equipment send the labeling operation information.

For example, user A holds smart glasses, user B holds tablet B, user C holds tablet C, smart glasses and tablet B, and tablet C establishes video communication through wired or wireless network, and smart glasses will be currently collected. The picture is encoded and sent to each tablet B and buffered for a period of time or a certain number of video frames. After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture image, and sends the second frame related information corresponding to the second video frame to the smart glasses and The tablet C, wherein the second frame related information includes but is not limited to: the second video frame identification information, the second video frame encoding start time, the second video frame decoding end time, and the second video frame codec and transmission total duration Information, etc. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. Tablet B generates real-time annotation operation information according to user B's labeling operation, and sends the labeling operation information to smart glasses and tablet PC in real time.

FIG. 4 illustrates a method for real-time annotation of a video frame at a third user equipment end according to still another aspect of the present application, wherein the method includes step S31, step S32, step S33, step S34, and step S35. In step S31, the third user equipment receives the video stream sent by the first user equipment to the second user equipment and the third user equipment; in step S32, the third user equipment receives the second user equipment in the video. The second frame related information of the second video frame intercepted in the stream; in step S33, the third user equipment determines, according to the second frame related information, a corresponding to the second video frame in the video stream. a third video frame; in step S34, the third user equipment receives the labeling operation information of the second video frame by the second user equipment; in step S35, the third user equipment is configured according to the labeling operation information. A corresponding labeling operation is presented in real time on the third video frame.

In some embodiments, the method further includes step S36 (not shown). In step S36, the third user equipment receives the first video frame sent by the first user equipment, where the first video is used as a preferred frame for presenting the labeling operation, in the third video frame. Loading the first video frame to replace the third video frame in a display window, wherein the labeling operation is displayed on the first video frame.

For example, the tablet ethyl enters the screen capture mode by the operation of the user, determines the second video frame corresponding to the current picture, and sends the second frame related information of the second video frame to the smart glasses end, where the second frame related information includes However, it is not limited to: an encoding start time of the second video frame or a number corresponding to the video frame. The smart glasses determine a corresponding unencoded first video frame according to the second frame related information sent by the tablet B, and send the first video frame to the tablet B and the tablet C, as by lossless compression The first video frame is sent to the tablet B and the tablet C, or the first video frame is sent to the tablet B and the tablet C through the lossy lossy compression, the lossy compression process is guaranteed to be better than the tablet B And the quality of the video frame cached locally by the tablet C is high. The tablet computer C receives and presents the first video frame, such as being presented in the form of a small window next to the current video, or displaying the first video frame on a large screen, presenting the current video in the form of a small window, and the like. Subsequently, the tablet computer C receives the labeling operation information sent by the tablet computer B, and presents the labeling operation in the first video frame.

In some embodiments, the method further includes step S37 (not shown). In step S37, the third user equipment receives the first video frame and the second frame related information sent by the first user equipment, where the first video frame is used as a preferred frame for presenting the labeling operation. Determining, according to the second frame related information, that the first video frame is used to replace the third video frame, and loading the first video frame in a display window of the third video frame to replace the first video frame a three video frame, wherein the labeling operation is displayed on the first video frame.

For example, the tablet computer C receives the unencoded first video frame and the second frame related information sent by the smart glasses, wherein the second frame related information includes a screenshot time of the second video frame, and a video frame of the second video frame. Number, etc. The tablet computer C receives and presents the first video frame, such as being presented as a small window next to the current video, or displaying the first video frame on a large screen, presenting the current video as a small window, etc., while presenting the first video. At the time of the frame, the tablet C presents the second frame related information in the first video frame presented, such as the screenshot time of the frame time in the first video frame or the frame number of the frame in the video frame. Subsequently, the tablet computer C receives the labeling operation information sent by the tablet computer B, and presents the labeling operation in the first video frame.

FIG. 5 illustrates a method for real-time annotation of a video frame at a network device end according to still another aspect of the present application, wherein the method includes step S41, step S42, step S43, step S44, and step S45. In step S41, the network device receives and forwards the video stream sent by the first user equipment to the second user equipment; in step S42, the network device receives the second video intercepted by the second user equipment in the video stream. a second frame related information of the frame; in step S43, the network device forwards the second frame related information to the first user equipment; in step S44, the network device receives the second user equipment to the first The labeling operation information of the two video frames; in step S45, the network device forwards the labeling operation information to the first user equipment.

For example, user A holds smart glasses, user B holds tablet B, and smart glasses and tablet B communicate video through the cloud. The smart glasses encode the currently collected picture and send it to the cloud, and the cloud forwards it to the tablet B. The smart glasses cache a period of time or a certain number of video frames when the video is sent. After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture image, and sends the second frame related information corresponding to the second video frame to the cloud, The cloud forwards to the smart glasses, where the second frame related information includes, but is not limited to, the second video frame identification information, the second video frame encoding start time, the second video frame decoding end time, and the second video frame encoding and decoding. Total length information, etc. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. The tablet computer B generates real-time annotation operation information according to the labeling operation of the user B, and sends the labeling operation information to the cloud, and is sent by the cloud to the smart glasses. After receiving the labeling operation information, the smart glasses display in the preset area of the smart glasses. Corresponding uncoded first video frame, and corresponding corresponding labeling operation is presented in real time at a position corresponding to the first video frame.

In some embodiments, in step S41, the network device receives and forwards the video stream sent by the first user equipment to the second user equipment, and the frame identification information of the transmitted video frame in the video stream. For example, the first user equipment performs encoding processing on the video frame, and sends the corresponding video stream and the frame identification information of the transmitted video frame in the video stream to the network device, where the network device sends the video stream and the frame of the transmitted video frame. The identification information is forwarded to the second user equipment, wherein the frame identification information includes an encoding start time of the video frame. In another embodiment, in step S43, the network device determines, according to the second frame related information, frame identification information of a video frame corresponding to the second video frame in the video stream, and the The frame identification information of the video frame corresponding to the two video frames is sent to the first user equipment.

For example, the cloud receives the video stream sent by the smart glasses and the frame identification information of the transmitted video frames in the video stream, such as the encoding start time of each video frame. The cloud forwards the video stream and the frame identification information corresponding to the transmitted video frame to the tablet B. Tablet B receives and presents the video frame and records the decoding end time of the video frame. The screen capture operation of the tablet E is performed by the user E, and the corresponding second video frame is determined, and the second frame related information of the second video frame is sent to the cloud, where the second frame related information includes the decoding end corresponding to the second video frame. Time or video number of the second video frame, etc. The cloud receives the second frame related information of the second video frame sent by the tablet B, and determines the frame identification information of the corresponding second frame based on the second frame related information, such as according to the decoding end time or the second of the second video frame. The video number of the video frame or the like determines the encoding start time of the second frame or the video number of the second video frame, and the like.

In some embodiments, in step S41, the network device receives and forwards the video stream sent by the first user equipment to the second user equipment and the third user equipment; wherein, in step S43, the network equipment uses the second frame. The related information is forwarded to the first user equipment and the third user equipment; wherein, in step S45, the network device forwards the labeling operation information to the first user equipment and the third user equipment.

For example, user A holds smart glasses, user B holds tablet B, user C holds tablet C, smart glasses, tablet B, and tablet C establish video communication through network devices, and smart glasses will capture the current picture. The code is sent to the network device and buffered for a period of time or a certain number of video frames, and the network device sends the video stream to the tablet B and the tablet C.

After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture, and sends the second frame related information corresponding to the second video frame to the network device. The second frame related information includes, but is not limited to, a second video frame identification information, a second video frame encoding start time, a second video frame decoding end time, and a second video frame codec and transmission total duration information. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. The network device forwards the second frame related information to the first user equipment and the second user equipment.

The tablet computer B generates real-time annotation operation information according to the labeling operation of the user B, and transmits the labeling operation information to the smart glasses and the tablet computer C through the network device in real time, and the smart glasses receive the labeling operation information, and the preset in the smart glasses The area displays a corresponding uncoded first video frame, and presents a corresponding labeling operation in real time at a position corresponding to the first video frame. Similarly, the tablet propyl receives the second frame related information and the labeled operation information, and finds the corresponding third video frame in the locally cached codec video library according to the second frame related information. And presenting a corresponding labeling operation in the third video frame based on the third video frame and the labeling operation information.

6 illustrates a method for real-time annotation of video frames in accordance with an aspect of the present application, wherein the method includes:

The first user equipment sends a video stream to the second user equipment;

FIG. 7 illustrates a method for real-time annotation of a video frame according to another aspect of the present application, wherein the method includes:

The first user equipment sends a video stream to the network device;

Receiving, by the network device, the labeling operation information of the second video frame by the second user equipment, and forwarding the labeling operation information to the first user equipment;

A method for real-time annotation of a video frame according to still another aspect of the present application, wherein the method comprises:

The first user equipment sends a video stream to the network device;

FIG. 8 illustrates a first user equipment for real-time annotation of a video frame according to an aspect of the present application, wherein the device includes a video transmission module 11, a frame information receiving module 12, a video frame determination module 13, and an annotation receiving module. 14 and annotation presentation module 15. The video sending module 11 is configured to send a video stream to the second user equipment, where the frame information receiving module 12 is configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream. a video frame determining module 13 configured to determine, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream, and an annotation receiving module 14 configured to receive the second The labeling operation information of the second video frame by the user equipment; the labeling presentation module 15 is configured to present a corresponding labeling operation in real time on the first video frame according to the labeling operation information.

Specifically, the video sending module 11 is configured to send a video stream to the second user equipment. For example, the first user equipment establishes a communication connection with the second user equipment through a wired or wireless network, and the first user equipment encodes the video stream to the second user equipment by using a video communication manner.

The frame information receiving module 12 is configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream. For example, the second user equipment determines the second frame related information of the video frame corresponding to the screen capture screen based on the screen capture operation of the second user, and then the first user equipment receives the second frame related to the second video frame sent by the second user equipment. Information, wherein the second frame related information includes, but is not limited to, second video frame identification information, a second video frame encoding start time, a second video frame decoding end time, and a second video frame codec and transmission total duration information. .

The video frame determining module 13 is configured to determine, according to the second frame related information, a first video frame in the video stream that corresponds to the second video frame. For example, the first user equipment locally stores a period of time or a certain number of transmitted unencoded video frames, and the first user equipment stores the uncoded locally according to the second frame related information sent by the second user equipment. In the video frame, the uncoded first video frame corresponding to the screen capture is determined.

The label receiving module 14 is configured to receive the labeling operation information of the second video frame by the second user equipment. For example, the second user equipment generates corresponding labeling operation information based on the labeling operation of the second user, and sends the labeling operation information to the first user equipment in real time, and the first user receives the labeling operation information.

The annotation presentation module 15 is configured to present a corresponding annotation operation on the first video frame in real time according to the annotation operation information. For example, the second user equipment displays a corresponding labeling operation in real time on the first video frame based on the received labeling operation information, such as displaying the first video frame in a small window in the current interface, and corresponding to the first video frame in the current interface. The location presents a corresponding annotation operation at a rate of one frame every 50 ms.

In some embodiments, the device also includes a storage module 16 (not shown). a storage module 16 for storing a video frame in the video stream, where the video frame determining module 13 is configured to determine, from the stored video frame, the second video frame according to the second frame related information. Corresponding first video frame. For example, the first user equipment sends the video stream to the second user equipment, and locally stores a period of time or a certain number of uncoded video frames, wherein a period of time or a certain number may be a preset fixed value, The threshold may be dynamically adjusted according to the network condition or the transmission rate; then, the first user equipment determines, according to the second frame related information of the second video frame sent by the second user equipment, the corresponding unrecognized in the locally stored video frame. The first video frame of the codec. In other embodiments, the stored video frame meets, but is not limited to, at least one of the following: a time interval between a transmission time of the stored video frame and a current time is less than or equal to a video frame storage duration threshold; the stored The cumulative number of video frames is less than or equal to a predetermined number of video frame storage thresholds.

In some embodiments, the device further includes a threshold adjustment module 17 (not shown). The threshold adjustment module 17 is configured to acquire codec and total transmission duration information of the video frame in the video stream, and adjust the video frame storage duration threshold or the number of the video frame storage according to the codec and total transmission duration information. Threshold. For example, the first user equipment records the encoding start time of each video frame, and after encoding, sends the video frame to the second user equipment, and the second user equipment receives and records each video frame decoding end time; subsequently, the second user equipment The video frame decoding end time is sent to the first user equipment, and the first user equipment calculates the codec and total transmission duration information of the current video frame based on the encoding start time and the decoding end time, or the second user equipment is based on the encoding start time and The decoding end time calculates the codec and total transmission duration information of the current video frame, and sends the codec and the total transmission duration information to the first user equipment. The first user equipment adjusts the video frame storage duration threshold or the video frame storage threshold according to the codec and the total transmission duration information. If the duration information is used as a unit time reference, setting a certain multiple of the video frame duration is The video frame stores a duration threshold; for example, the number of video frames that can be sent in the duration information is calculated according to the duration information and the rate at which the first user equipment sends the video frame, and the number is used as a unit number to set a certain multiple of the video frame. The number is used as a threshold for the number of video frames stored.

In some embodiments, the video sending module 11 is configured to send, to the second user equipment, a video stream and frame identification information of the transmitted video frame in the video stream, where the video frame determining module 13 is configured to The second frame related information determines a first video frame corresponding to the second video frame in the video stream, where frame identification information of the first video frame corresponds to the second frame related information. The frame identification information of the video frame may be a codec time corresponding to the video frame, or may be a number corresponding to the video. In some embodiments, the video sending module 11 is configured to perform encoding processing on multiple video frames to be transmitted, and send frame identification information of the corresponding video stream and the transmitted video frame in the video stream to the second user equipment. . For example, the first user equipment performs encoding processing on a plurality of video frames to be transmitted, and acquires an encoding start time of the plurality of video frames to be transmitted, and sends the multiple video frames and their encoding start time to the second. User equipment. In some embodiments, the frame identification information of the transmitted video frame in the video stream includes encoding start time information of the transmitted video frame.

It should be understood by those skilled in the art that the content of the frame identification information in the foregoing embodiment is only an example, and the content of other frame identification information that appears in the prior art or in the future, if applicable to the present application, should also belong to the protection of the present application. The scope is hereby incorporated by reference.

In some embodiments, the device also includes a video frame rendering module 18 (not shown). The video frame presentation module 18 is configured to present the first video frame. The annotation presentation module 15 is configured to superimpose a corresponding annotation operation on the first video frame according to the annotation operation information. For example, the first user equipment determines the first video frame that has not been coded, and displays the first video frame in a preset position in the current interface or in a small window; subsequently, the first user equipment receives the annotation according to the real-time reception. The operation information is superimposed on the corresponding position of the first video frame to present a corresponding labeling operation.

In some embodiments, the device also includes a first preferred frame module 19 (not shown). The first preferred frame module 19 is configured to send the first video frame to the second user equipment as a preferred frame that presents the labeling operation. For example, the first user equipment determines the first video frame that has not been coded, and sends the first video frame to the second user equipment for the second user equipment to present the first video frame of higher quality.

In some embodiments, the video sending module 11 is configured to send a video stream to the second user equipment and the third user equipment. For example, a communication connection is established between the first user equipment, the second user equipment, and the third user equipment, where the first user equipment is the current video frame sender, and the second user equipment and the third user equipment are the current video frame receiving. The first user equipment sends a video stream to the second user equipment and the third user equipment through the communication connection.

In some embodiments, the device further includes a second preferred frame module 010 (not shown). The second preferred frame module 010 is configured to send the first video frame to the second user equipment and/or the third user equipment as a preferred frame for presenting the labeling operation. For example, the first user equipment determines a corresponding first video frame in the locally cached video frame according to the second frame related information, and sends the first video frame to the second user equipment and/or the third user equipment. After the second user equipment and/or the third user equipment receives the uncoded first video frame, the first video frame is presented, and the second user and/or the third user may perform an annotation operation based on the first video frame.

In some embodiments, the second preferred frame module 010 is configured to send the first video frame and the second frame related information to the second user equipment and/or the third user equipment, where The first video frame is used as a preferred frame to present the annotation operation in the second user device or the third user device.

9 illustrates a second user equipment for real-time annotation of a video frame according to another aspect of the present application, wherein the apparatus includes a video receiving module 21, a frame information determining module 22, an annotation acquiring module 23, and an annotation transmitting module 24. . The video receiving module 21 is configured to receive the video stream sent by the first user equipment, and the frame information determining module 22 is configured to send the intercepted number to the first user equipment according to a screenshot operation of the user in the video stream. a second frame related information of the second video frame; an annotation obtaining module 23, configured to acquire the labeling operation information of the second video frame by the user; and an annotation sending module 24, configured to send the Label the operation information. For example, the second user equipment receives and presents the video stream sent by the first user equipment; and the second user equipment determines, according to the screen capture operation of the second user, the second video frame corresponding to the current screen capture, and the second video frame is The second frame related information is sent to the first user equipment. Then, the second user equipment generates the labeling operation information based on the labeling operation of the second user, and sends the labeling operation information to the first user equipment.

In some embodiments, the video receiving module 21 is configured to receive a video stream sent by the first user equipment, and frame identification information of the transmitted video frame in the video stream, where the second frame related information includes the following At least one of: frame identification information of the second video frame; frame related information generated based on frame identification information of the second video frame. For example, the first user equipment sends the video stream to the second user equipment, and also sends the frame identifier information of the sent video frame in the video stream to the second user equipment, where the second user equipment receives the video stream, and the video stream The frame identification information of the transmitted video frame. The second user equipment determines, according to the screen capture operation of the second user, the second video frame corresponding to the current screen capture, and sends the second frame related information of the second video frame to the first user equipment, where the second video The second frame related information of the frame includes but is not limited to: frame identification information of the second video frame; frame related information generated based on the frame identification information of the second video frame.

In some embodiments, the annotation obtaining module 23 is configured to acquire the labeling operation information of the second video frame by the user in real time, where the label sending module 24 is configured to send the identifier to the first user equipment in real time. Label operation. For example, the second user equipment acquires the corresponding labeling operation information in real time based on the operation of the second user, for example, collecting the corresponding labeling operation information at a certain time interval. Then, the second user equipment sends the acquired annotation operation information to the first user equipment in real time.

In some embodiments, the device also includes a first video frame replacement module 25 (not shown). a first video frame replacement module 25, configured to receive a first video frame sent by the first user equipment, where the first video frame is used as a preferred frame for presenting the labeling operation, in the second video Loading the first video frame to replace the second video frame in a display window of the frame, wherein the labeling operation is displayed on the first video frame. For example, the second user equipment determines a second video frame corresponding to the current screen shot and transmits the second frame related information of the second video frame to the first user equipment; the first user equipment is based on the second video frame. The second frame related information determines the unencoded first video frame corresponding to the second video frame, and sends the first video frame to the second user equipment, where the second user equipment receives and presents the first video frame. And obtaining the labeling operation information of the first video frame by the second user.

In some embodiments, the device further includes a first video frame labeling module 26 (not shown). a first video frame labeling module 26, configured to receive a first video frame and the second frame related information that are sent by the first user equipment, where the first video frame is used as a preference for presenting the labeling operation a frame, determining, according to the second frame related information, that the first video frame is used to replace the second video frame, and loading the first video frame to replace the first video frame in a display window of the second video frame The second video frame is described, wherein the labeling operation is displayed on the first video frame.

In some embodiments, the video receiving module 21 is configured to receive a video stream that is sent by the first user equipment to the second user equipment and the third user equipment, where the label sending module 24 is configured to send to the first user equipment. And the third user equipment sends the labeling operation information.

For example, user A holds smart glasses, user B holds tablet B, user C holds tablet C, smart glasses and tablet B, and tablet C establishes video communication through wired or wireless network, and smart glasses will be currently collected. The picture is encoded and sent to each tablet B and buffered for a period of time or a certain number of video frames. After receiving and decoding, the tablet B receives the video stream, and based on the screen capture operation of the user B, determines a second video frame corresponding to the screen capture image, and sends the second frame related information corresponding to the second video frame to the smart glasses and The tablet C, wherein the second frame related information includes but is not limited to: the second video frame identification information, the second video frame encoding start time, the second video frame decoding end time, and the second video frame codec and transmission total duration Information, etc. The smart glasses receive the second frame related information of the second video frame, and determine a corresponding uncoded first video frame in the locally stored video frame based on the second frame related information. The tablet computer B generates real-time labeling operation information according to the labeling operation of the user B, and sends the labeling operation information to the smart glasses and the tablet computer C in real time.

FIG. 10 illustrates an apparatus for real-time labeling a video frame at a third user equipment end according to still another aspect of the present application, wherein the apparatus includes a third video receiving module 31, a third frame information receiving module 32, and a third The three video frame determining module 33, the third label receiving module 34, and the third rendering module 35. a third video receiving module 31, configured to receive a video stream that is sent by the first user equipment to the second user equipment and the third user equipment, where the third frame information receiving module 32 is configured to receive the second user equipment in the a second frame related information of the second video frame that is intercepted in the video stream; the third video frame determining module 33 is configured to determine, according to the second frame related information, that the video stream corresponds to the second video frame a third video frame, a third annotation receiving module 34, configured to receive the labeling operation information of the second video frame by the second user equipment, and a third rendering module 35, configured to The corresponding labeling operation is presented in real time on the third video frame.

In some embodiments, the device also includes a preferred frame reception presentation module 36 (not shown). a preferred frame receiving presentation module 36, configured to receive a first video frame sent by the first user equipment, where the first video is used as a preferred frame for presenting the labeling operation, in the third video frame Loading the first video frame to replace the third video frame in a display window, wherein the labeling operation is displayed on the first video frame.

In some embodiments, the device further includes a preferred frame annotation presentation module 37 (not shown). a frame annotation presentation module 37, configured to receive a first video frame and the second frame related information sent by the first user equipment, where the first video frame is used as a preferred frame for presenting the labeling operation Determining, according to the second frame related information, that the first video frame is used to replace the third video frame, and loading the first video frame in a display window of the third video frame to replace the first video frame a three video frame, wherein the labeling operation is displayed on the first video frame.

FIG. 11 illustrates a network device for real-time annotation of a video frame according to still another aspect of the present application, wherein the device includes a video forwarding module 41, a frame information receiving module 42, a frame information forwarding module 43, and an annotation receiving module 44. And annotating the forwarding module 45. The video forwarding module 41 is configured to receive and forward a video stream that is sent by the first user equipment to the second user equipment, and the frame information receiving module 42 is configured to receive the second information that is captured by the second user equipment in the video stream. a second frame related information of the video frame, a frame information forwarding module 43, configured to forward the second frame related information to the first user equipment, and an annotation receiving module 44, configured to receive the second user equipment The labeling operation information of the second video frame is used to forward the labeling operation information to the first user equipment.

In some embodiments, the video forwarding module 41 is configured to receive and forward a video stream sent by the first user equipment to the second user equipment, and frame identification information of the transmitted video frame in the video stream. For example, the first user equipment performs encoding processing on the video frame, and sends the corresponding video stream and the frame identification information of the transmitted video frame in the video stream to the network device, where the network device sends the video stream and the frame of the transmitted video frame. The identification information is forwarded to the second user equipment, wherein the frame identification information includes an encoding start time of the video frame. In another embodiment, the frame information forwarding module 43 is configured to determine, according to the second frame related information, frame identification information of a video frame corresponding to the second video frame in the video stream, and The frame identification information of the video frame corresponding to the second video frame is sent to the first user equipment.

In some embodiments, the video forwarding module 41 is configured to receive and forward a video stream sent by the first user equipment to the second user equipment and the third user equipment, where the frame information forwarding module 43 is configured to use the second The frame-related information is forwarded to the first user equipment and the third user equipment. The label forwarding module 45 is configured to forward the labeling operation information to the first user equipment and the third user equipment.

According to an aspect of the present application, there is provided a system for real-time annotation of a video frame, wherein the system comprises: the first user equipment according to any of the above embodiments and any of the above embodiments The second user equipment is described; in other embodiments, the system further comprises: the network device as described in any of the above embodiments.

According to an aspect of the present application, there is provided a system for real-time annotation of a video frame, wherein the system comprises: the first user equipment according to any of the above embodiments, according to any of the above embodiments The second user equipment and the third user equipment as described in any of the above embodiments; in other embodiments, the system further includes: the network device as described in any of the foregoing embodiments.

The present application also provides a computer readable storage medium storing computer code, the method of any of which is performed when the computer code is executed.

The present application also provides a computer program product that is executed as described in any of the foregoing when the computer program product is executed by a computer device.

The application also provides a computer device, the computer device comprising:

One or more processors;

a memory for storing one or more computer programs;

When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method of any of the preceding.

Figure 12 illustrates an exemplary system that can be used to implement various embodiments described in this application;

As shown in FIG. 12, in some embodiments, system 300 can be used as a device for any real-time annotation of video frames as any of the described embodiments. In some embodiments, system 300 can include and be coupled to one or more computer readable media (eg, system memory or NVM/storage device 320) having instructions and configured to execute The instructions are one or more processors (eg, processor(s) 305) that implement the modules to perform the actions described herein.

For one embodiment, system control module 310 can include any suitable interface controller to provide to at least one of processor(s) 305 and/or any suitable device or component in communication with system control module 310. Any suitable interface.

System control module 310 can include a memory controller module 330 to provide an interface to system memory 315. The memory controller module 330 can be a hardware module, a software module, and/or a firmware module.

System memory 315 can be used, for example, to load and store data and/or instructions for system 300. For one embodiment, system memory 315 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, system memory 315 can include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, system control module 310 can include one or more input/output (I/O) controllers to provide an interface to NVM/storage device 320 and communication interface(s) 325.

For example, NVM/storage device 320 can be used to store data and/or instructions. NVM/storage device 320 may comprise any suitable non-volatile memory (eg, flash memory) and/or may include any suitable non-volatile storage device(s) (eg, one or more hard disk drives ( HDD), one or more compact disc (CD) drives and/or one or more digital versatile disc (DVD) drives).

The NVM/storage device 320 can include storage resources that are physically part of the device on which the system 300 is installed, or that can be accessed by the device without having to be part of the device. For example, NVM/storage device 320 can be accessed via network via communication interface(s) 325.

The communication interface(s) 325 can provide an interface to the system 300 to communicate over one or more networks and/or with any other suitable device. System 300 can wirelessly communicate with one or more components of a wireless network in accordance with any of one or more wireless network standards and/or protocols.

For one embodiment, at least one of the processor(s) 305 can be packaged with the logic of one or more controllers of the system control module 310 (eg, the memory controller module 330). For one embodiment, at least one of the processor(s) 305 can be packaged with the logic of one or more controllers of the system control module 310 to form a system in package (SiP). For one embodiment, at least one of the processor(s) 305 can be integrated on the same mold as the logic of one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 can be integrated with the logic of one or more controllers of the system control module 310 on the same mold to form a system on a chip (SoC).

In various embodiments, system 300 can be, but is not limited to, a server, workstation, desktop computing device, or mobile computing device (eg, a laptop computing device, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, system 300 can have more or fewer components and/or different architectures. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit ( ASIC) and speakers.

It should be noted that the present application can be implemented in software and/or a combination of software and hardware, for example, using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Likewise, the software programs (including related data structures) of the present application can be stored in a computer readable recording medium such as a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some of the steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.

In addition, a portion of the application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide a method and/or technical solution in accordance with the present application. It should be understood by those skilled in the art that the form of computer program instructions in a computer readable medium includes, but is not limited to, source files, executable files, installation package files, etc., accordingly, the manner in which the computer program instructions are executed by the computer includes but not Limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installation. program. Here, the computer readable medium can be any available computer readable storage medium or communication medium that can be accessed by a computer.

Communication media includes media that can be transferred from one system to another by communication signals including, for example, computer readable instructions, data structures, program modules or other data. Communication media can include conductive transmission media such as cables and wires (eg, fiber optics, coaxial, etc.) and wireless (unguided transmission) media capable of propagating energy waves, such as acoustic, electromagnetic, RF, microwave, and infrared. . Computer readable instructions, data structures, program modules or other data may be embodied, for example, as modulated data signals in a wireless medium, such as a carrier wave or a similar mechanism, such as embodied in a portion of a spread spectrum technique. The term "modulated data signal" refers to a signal whose one or more features are altered or set in such a manner as to encode information in the signal. Modulation can be analog, digital or hybrid modulation techniques.

The computer readable storage medium may comprise, by way of example and not limitation, vols and non-volatile, implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Mobile and non-removable media. For example, a computer readable storage medium includes, but is not limited to, volatile memory such as random access memory (RAM, DRAM, SRAM); and nonvolatile memory such as flash memory, various read only memories (ROM, PROM, EPROM) , EEPROM), magnetic and ferromagnetic/ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, tapes, CDs, DVDs); or other currently known media or later developed for storage in computer systems Computer readable information/data used.

Herein, an embodiment in accordance with the present application includes a device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, triggering The apparatus operates based on the aforementioned methods and/or technical solutions in accordance with various embodiments of the present application.

It is obvious to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the present embodiments are to be considered as illustrative and not restrictive, and the scope of the invention is defined by the appended claims instead All changes in the meaning and scope of equivalent elements are included in this application. Any reference signs in the claims should not be construed as limiting the claim. In addition, it is to be understood that the word "comprising" does not exclude other elements or steps. A plurality of units or devices recited in the device claims may also be implemented by a unit or device by software or hardware. The first, second, etc. words are used to denote names and do not denote any particular order.

Claims

A method for real-time annotation of a video frame on a first user equipment side, wherein the method includes:

Sending a video stream to the second user equipment;

Receiving second frame related information of the second video frame intercepted by the second user equipment in the video stream;

Determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Receiving, by the second user equipment, labeling operation information of the second video frame;

And correspondingly presenting a corresponding labeling operation on the first video frame according to the labeling operation information.
The method of claim 1 wherein the method further comprises:

Storing video frames in the video stream;

The determining, by the second frame related information, the first video frame corresponding to the second video frame in the video stream, including:

Determining, from the stored video frames, a first video frame corresponding to the second video frame according to the second frame related information.
The method of claim 2 wherein said stored video frames satisfy at least one of:

The time interval between the sending time of the stored video frame and the current time is less than or equal to the video frame storage time threshold;

The cumulative number of stored video frames is less than or equal to a predetermined number of video frame storage thresholds.
The method of claim 3, wherein the method further comprises:

Obtaining codec and total transmission duration information of the video frame in the video stream;

And adjusting the video frame storage duration threshold or the video frame storage threshold according to the codec and transmission total duration information.
The method of claim 1, wherein the transmitting the video stream to the second user equipment comprises:

Transmitting, to the second user equipment, a video stream and frame identification information of the transmitted video frame in the video stream;

The determining, by the second frame related information, the first video frame corresponding to the second video frame in the video stream, including:

Determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream, where frame identification information of the first video frame corresponds to the second frame related information .
The method of claim 5, wherein the transmitting the video stream and the frame identification information of the transmitted video frame in the video stream to the second user equipment comprises:

And encoding the plurality of video frames to be transmitted, and transmitting the frame identification information of the corresponding video stream and the transmitted video frame in the video stream to the second user equipment.
The method of claim 6, wherein the frame identification information of the transmitted video frame in the video stream includes encoding start time information of the transmitted video frame.
The method of claim 1 wherein the method further comprises:

Presenting the first video frame;

The performing the corresponding labeling operation on the first video frame in real time according to the labeling operation information, including:

And displaying, according to the labeling operation information, a corresponding labeling operation on the first video frame.
The method of any of claims 1 to 8, wherein the method further comprises:

The first video frame is sent to the second user equipment as a preferred frame that presents the labeling operation.
The method according to any one of claims 1 to 8, wherein the transmitting the video stream to the second user equipment comprises:

The video stream is sent to the second user equipment and the third user equipment.
The method of claim 10, wherein the method further comprises:

Transmitting the first video frame to the second user equipment and/or the third user equipment as a preferred frame presenting the labeling operation.
The method of claim 11, wherein the transmitting the first video frame to the second user equipment and/or the third user equipment comprises:

Transmitting the first video frame and the second frame related information to the second user equipment and/or the third user equipment, wherein the first video frame is used as the second user equipment Or the preferred frame of the labeling operation is presented in the third user equipment.
A method for real-time annotation of a video frame on a second user equipment side, wherein the method includes:

Receiving a video stream sent by the first user equipment;

Transmitting second frame related information of the intercepted second video frame to the first user equipment according to a screenshot operation of the user in the video stream;

Obtaining, by the user, the labeling operation information of the second video frame;

Sending the labeling operation information to the first user equipment.
The method of claim 13, wherein the receiving the video stream sent by the first user equipment comprises:

Receiving a video stream sent by the first user equipment, and frame identification information of the transmitted video frame in the video stream;

The second frame related information includes at least one of the following:

Frame identification information of the second video frame;

Frame related information generated based on frame identification information of the second video frame.
The method of claim 14, wherein the frame identification information comprises encoding start time information of the second video frame.
The method of claim 15, wherein the second frame related information comprises decoding end time information and codec and transmission total duration information of the second video frame.
The method according to claim 14, wherein the obtaining the labeling operation information of the second video frame by the user comprises:

Obtaining, in real time, the labeling operation information of the second video frame by the user;

The sending the labeling operation information to the first user equipment includes:

Transmitting the annotation operation information to the first user equipment in real time.
The method of claim 13 wherein the method further comprises:

Receiving, by the first user equipment, a first video frame, where the first video frame is used as a preferred frame for presenting the labeling operation;

Loading the first video frame to replace the second video frame in a display window of the second video frame, wherein the labeling operation is displayed on the first video frame.
The method of claim 13 wherein the method further comprises:

Receiving, by the first user equipment, the first video frame and the second frame related information, where the first video frame is used as a preferred frame for presenting the labeling operation;

Determining, according to the second frame related information, that the first video frame is used to replace the second video frame;

Loading the first video frame to replace the second video frame in a display window of the second video frame, wherein the labeling operation is displayed on the first video frame.
The method of claim 13, wherein the receiving the video stream sent by the first user equipment comprises:

Receiving a video stream that is sent by the first user equipment to the second user equipment and the third user equipment;

The sending the labeling operation information to the first user equipment includes:

And sending the labeling operation information to the first user equipment and the third user equipment.
A method for real-time annotation of a video frame on a third user equipment side, wherein the method includes:

Receiving a video stream that is sent by the first user equipment to the second user equipment and the third user equipment;

Receiving second frame related information of the second video frame intercepted by the second user equipment in the video stream;

Determining, according to the second frame related information, a third video frame corresponding to the second video frame in the video stream;

Receiving, by the second user equipment, labeling operation information of the second video frame;

And correspondingly presenting a corresponding labeling operation on the third video frame according to the labeling operation information.
The method of claim 21, wherein the method further comprises:

Receiving, by the first user equipment, a first video frame, where the first video is used as a preferred frame for presenting the labeling operation;

Loading the first video frame to replace the third video frame in a display window of the third video frame, wherein the labeling operation is displayed on the first video frame.
The method of claim 21, wherein the method further comprises:

Receiving, by the first user equipment, the first video frame and the second frame related information, where the first video frame is used as a preferred frame for presenting the labeling operation;

Determining, according to the second frame related information, that the first video frame is used to replace the third video frame;

Loading the first video frame to replace the third video frame in a display window of the third video frame, wherein the labeling operation is displayed on the first video frame.
A method for real-time annotation of a video frame on a network device side, wherein the method includes:

Receiving and forwarding a video stream sent by the first user equipment to the second user equipment;

Receiving second frame related information of the second video frame intercepted by the second user equipment in the video stream;

Forwarding the second frame related information to the first user equipment;

Receiving, by the second user equipment, labeling operation information of the second video frame;

Forwarding the labeling operation information to the first user equipment.
The method of claim 24, wherein the receiving and forwarding the video stream sent by the first user equipment to the second user equipment comprises:

Receiving and forwarding a video stream sent by the first user equipment to the second user equipment, and frame identification information of the transmitted video frame in the video stream.
The method of claim 25, wherein the forwarding the second frame related information to the first user equipment comprises:

Determining, according to the second frame related information, frame identification information of a video frame corresponding to the second video frame in the video stream;

Sending frame identification information of the video frame corresponding to the second video frame to the first user equipment.
The method of claim 24, wherein the receiving and forwarding the video stream sent by the first user equipment to the second user equipment comprises:

Receiving and forwarding a video stream sent by the first user equipment to the second user equipment and the third user equipment;

The forwarding the second frame related information to the first user equipment includes:

Forwarding the second frame related information to the first user equipment and the third user equipment;

The forwarding the labeling operation information to the first user equipment includes:

And forwarding the labeling operation information to the first user equipment and the third user equipment.
A method for real-time annotation of a video frame, wherein the method comprises:

The first user equipment sends a video stream to the second user equipment;

Receiving, by the second user equipment, the video stream, and sending, according to a screenshot operation of the user in the video stream, second frame related information of the intercepted second video frame to the first user equipment;

Receiving, by the first user equipment, the second frame related information, and determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Obtaining, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the first user equipment;

The first user equipment receives the labeling operation information, and presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information.
A method for real-time annotation of a video frame, wherein the method comprises:

The first user equipment sends a video stream to the network device;

Receiving, by the network device, the video stream, and forwarding the video stream to a second user equipment;

Receiving, by the second user equipment, the video stream, and sending, according to a screenshot operation of the user in the video stream, second frame related information of the intercepted second video frame to the network device;

Receiving, by the network device, the second frame related information, and forwarding the second frame related information to the first user equipment;

Receiving, by the first user equipment, the second frame related information, and determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Obtaining, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the network device;

The network device receives the labeling operation information of the second video frame by the second user equipment, and forwards the labeling operation information to the first user equipment;

The first user equipment receives the labeling operation information, and presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information.
A method for real-time annotation of a video frame, wherein the method comprises:

The first user equipment sends a video stream to the second user equipment and the third user equipment;

Transmitting, by the second user equipment, the second frame related information of the intercepted second video frame to the first user equipment and the third user equipment according to a screenshot operation of the user in the video stream;

And acquiring, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the first user equipment and the third user equipment;

Receiving, by the first user equipment, the second frame related information, determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream, and receiving the labeling operation information And displaying a corresponding labeling operation in real time on the first video frame according to the labeling operation information;

Receiving, by the third user equipment, the video stream, receiving second frame related information of the second video frame, receiving the labeling operation information, determining, according to the second frame related information, the video stream and the The third video frame corresponding to the second video frame presents a corresponding labeling operation in real time on the third video frame according to the labeling operation information.
A method for real-time annotation of a video frame, wherein the method comprises:

The first user equipment sends a video stream to the network device;

Receiving, by the network device, the video stream, and forwarding the video stream to the second user equipment and the third user equipment;

Receiving, by the second user equipment, the video stream, and sending, according to a screenshot operation of the user in the video stream, second frame related information of the intercepted second video frame to the network device;

Receiving, by the network device, the second frame related information, and forwarding the second frame related information to the first user equipment and the third user equipment;

Receiving, by the first user equipment, the second frame related information, and determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

Obtaining, by the second user equipment, the labeling operation information of the second video frame by the user, and sending the labeling operation information to the network device;

The network device receives the labeling operation information of the second video frame by the second user equipment, and forwards the labeling operation information to the first user equipment and the third user equipment;

The first user equipment receives the labeling operation information, and presents a corresponding labeling operation in real time on the first video frame according to the labeling operation information;

Receiving, by the third user equipment, the video stream, receiving second frame related information of the second video frame, receiving the labeling operation information, determining, according to the second frame related information, the video stream and the The third video frame corresponding to the second video frame presents a corresponding labeling operation in real time on the third video frame according to the labeling operation information.
A first user equipment for real-time annotation of a video frame, wherein the device comprises:

a video sending module, configured to send a video stream to the second user equipment;

a frame information receiving module, configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream;

a video frame determining module, configured to determine, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream;

An annotation receiving module, configured to receive the labeling operation information of the second video frame by the second user equipment;

And an annotation presentation module, configured to present a corresponding annotation operation on the first video frame in real time according to the annotation operation information.
The device of claim 32, wherein the video transmitting module is configured to:

Transmitting, to the second user equipment, a video stream and frame identification information of the transmitted video frame in the video stream;

The video frame determining module is configured to:

Determining, according to the second frame related information, a first video frame corresponding to the second video frame in the video stream, where frame identification information of the first video frame corresponds to the second frame related information .
A second user equipment for real-time annotation of a video frame, wherein the device includes:

a video receiving module, configured to receive a video stream sent by the first user equipment;

a frame information determining module, configured to send second frame related information of the intercepted second video frame to the first user equipment according to a screenshot operation of the user in the video stream;

An annotation obtaining module, configured to acquire the labeling operation information of the second video frame by the user;

And an annotation sending module, configured to send the labeling operation information to the first user equipment.
A third user equipment for real-time annotation of a video frame, wherein the device comprises:

a third video receiving module, configured to receive a video stream that is sent by the first user equipment to the second user equipment and the third user equipment;

a third frame information receiving module, configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream;

a third video frame determining module, configured to determine, according to the second frame related information, a third video frame corresponding to the second video frame in the video stream;

a third label receiving module, configured to receive the labeling operation information of the second video frame by the second user equipment;

The third rendering module is configured to present a corresponding labeling operation on the third video frame in real time according to the labeling operation information.
A network device for real-time annotation of a video frame, wherein the device includes:

a video forwarding module, configured to receive and forward a video stream sent by the first user equipment to the second user equipment;

a frame information receiving module, configured to receive second frame related information of the second video frame that is intercepted by the second user equipment in the video stream;

a frame information forwarding module, configured to forward the second frame related information to the first user equipment;

An annotation receiving module, configured to receive the labeling operation information of the second video frame by the second user equipment;

And an annotation forwarding module, configured to forward the labeling operation information to the first user equipment.
A system for real-time annotation of video frames, wherein the system comprises: a first user device according to any one of claims 32 or 33 and a second user device according to claim 34.
The system of claim 37, wherein the system further comprises: the network device of claim 36.
A system for real-time annotation of video frames, wherein the system comprises: a first user equipment according to any one of claims 32 or 33, a second user equipment according to claim 34, and A third user equipment as claimed in claim 35.
The system of claim 39, wherein the system further comprises: the network device of claim 36.
A device for real-time annotation of a video frame, wherein the device includes:

Processor;

A memory arranged to store computer executable instructions that, when executed, cause the processor to perform the operations of the method of any one of claims 1 to 27.
A computer readable medium storing instructions which, when executed, cause a system to perform the operations of the method of any one of claims 1 to 27.