CN117478972A

CN117478972A - Flow control method, device, equipment and medium for remote video transmission

Info

Publication number: CN117478972A
Application number: CN202311308345.9A
Authority: CN
Inventors: 胡建强; 蒙杰华; 陈洪; 王苏阳
Original assignee: Shanghai Jiuchi Network Technology Co ltd
Current assignee: Shanghai Jiuchi Network Technology Co ltd
Priority date: 2023-10-10
Filing date: 2023-10-10
Publication date: 2024-01-30

Abstract

The application discloses a flow control method, a flow control device, flow control equipment and flow control media for remote video transmission, and belongs to the technical field of data transmission. The method comprises the following steps: responding to a remote Cheng Zhuomian request instruction transmitted by a main control end, and determining the system state of a controlled end; determining a video transmission target mode of the system from a preset state mode association relation according to the system state, generating a video difference frame according to an original video frame of a desktop of the system if the video transmission target mode is a first stream saving mode, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a first stream saving mode ending event matched with a preset event type exists or not; if the video transmission target mode exists, switching the video transmission target mode into a normal transmission mode, and continuously transmitting the original video frame to the main control terminal. The scheme determines the corresponding video transmission mode according to the system state of the controlled terminal, can reduce bandwidth consumption when the controlled terminal is idle, saves flow and improves transmission rate. The method can ensure the response to the user operation in time and improve the real-time performance of remote control.

Description

Flow control method, device, equipment and medium for remote video transmission

Technical Field

The application belongs to the technical field of data transmission, and particularly relates to a flow control method, a flow control device, flow control equipment and flow control media for remote video transmission.

Background

With the continuous development of technology, in order to solve the problem that a user cannot directly operate when accessing a computer or a server remotely, a remote desktop video playing technology has also developed, so that the user can access resources on the remote computer or the server more conveniently. For example, in an online education scene, a teacher can completely control the desktop interface of a remote computer on the own computer through a remote desktop video playing technology, so that playing of a presentation is realized.

In the process of realizing remote desktop video playing, a sending end needs to collect video data, and then the video data is compressed and encoded through an encoder, so that the data volume is reduced and the transmission efficiency is improved. The encoded data is encapsulated into network packets for transmission over a network. The data packets are then transmitted to the receiving end via the internet or other network transmission medium. After receiving the data packet, the receiving end performs decapsulation processing on the data packet to extract the coded data stream and other control information. The decoder performs inverse operation on the encoded data stream according to a certain decoding algorithm to recover the original video data. And finally, the receiving end combines the decoded video data into a complete video frame and plays the complete video frame by using a player.

In the remote assistance process, because a large amount of high-quality video needs to be transmitted in real time, the process extremely consumes network bandwidth, and if the controlled terminal is in a screen locking or screen protecting state, video frames are continuously transmitted to the main control terminal, so that unnecessary bandwidth consumption and energy consumption are brought.

Disclosure of Invention

The embodiment of the application provides a flow control method, a device, equipment and a medium for remote video transmission, which aim to solve the problems that in the prior art, the system state of a controlled terminal is not detected in real time in the remote assistance process, and high-quality video is continuously transmitted all the time, so that unnecessary bandwidth consumption and energy consumption are brought. By means of the flow control method for remote video transmission, the system state of the controlled terminal is monitored in real time to determine the corresponding video transmission mode, and bandwidth consumption can be reduced when the controlled terminal is idle, so that flow is saved and transmission rate is improved. And the response to the user operation can be ensured in time, and the real-time performance of remote control is improved.

In a first aspect, an embodiment of the present application provides a flow control method for remote video transmission, where the method includes:

responding to a far Cheng Zhuomian request instruction transmitted by a main control end, and determining the system state of a controlled end by using a preset system service monitoring application;

According to the system state, determining a video transmission target mode of the system from a preset state mode association relation;

if the video transmission target mode is determined to be a first stream saving mode, generating a video difference frame according to an original video frame of a system desktop, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a stream saving mode ending event matched with a preset event type exists or not;

if the current-saving mode ending event is monitored, the video transmission target mode is switched to a normal transmission mode, and the original video frame is continuously transmitted to the main control terminal.

Further, after determining the video transmission target mode of the system from the preset state mode association relation according to the system state, the method further comprises:

if the video transmission target mode is determined to be the second stream saving mode, the video frame in any form is not sent to the main control terminal, and whether a stream saving mode ending event matched with the preset event type exists or not is continuously monitored.

And if the video transmission target mode is determined to be a normal transmission mode, continuously transmitting the original video frame to a main control terminal.

Further, determining the system state of the controlled terminal by using a preset system service monitoring application comprises the following steps:

acquiring monitoring parameters and acquiring a system menu message type by using a preset system service monitoring application;

if the first monitoring parameter is acquired, determining that the system is in a screen locking state;

accordingly, determining a video transmission target mode of the system according to the system state includes:

if the video transmission target mode is in the screen locking state, determining that the video transmission target mode of the system is a first stream saving mode.

Further, after acquiring the monitoring parameters and acquiring the system menu message type by using the preset system service monitoring application, the method further comprises:

if the first message type is acquired, determining that the system is in a screen protection state;

and if the video transmission target mode is in the screen protection state, determining that the video transmission target mode of the system is a second stream saving mode.

If the second monitoring parameters are acquired, determining that the system is in a login state, and continuously acquiring the monitoring parameters;

if the video transmission target mode is in the login state, determining that the video transmission target mode of the system is a normal transmission mode;

if the system state is monitored to be switched from the login state to the screen locking state, determining that the video transmission target mode of the system is switched to the first stream saving mode.

Further, before generating a video difference frame according to an original video frame of a system desktop and continuously transmitting the video difference frame to the master control terminal if the video transmission target mode is determined to be the first stream saving mode, and continuously monitoring whether a stream saving mode ending event matched with a preset event type exists or not, the method further comprises:

continuously collecting screen image data, coding the screen image data by using preset coding equipment, and generating coding data and coding data parameters;

and generating a screen frame instruction according to the encoded data and the encoded data parameters, and transmitting the screen frame instruction to a main control end for the main control end to decode and render the encoded data according to the screen frame instruction.

In a second aspect, embodiments of the present application provide a flow control device for remote video transmission, the device including:

the system state determining module is used for determining the system state of the controlled terminal by using a preset system service monitoring application in response to a remote Cheng Zhuomian request instruction transmitted by the main control terminal;

the video transmission target mode determining module is used for determining a video transmission target mode of the system according to the system state, continuously transmitting a video difference frame to the main control terminal if the video difference frame is in a first stream saving mode, and continuously monitoring whether a preset stream saving mode ending event exists or not;

the video difference frame generation module is used for generating a video difference frame according to an original video frame of a system desktop if the video transmission target mode is determined to be a first stream saving mode, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a first stream saving mode ending event matched with a preset event type exists or not;

and the video transmission target mode switching module is used for switching the video transmission target mode into a normal transmission mode if the stream saving mode ending event exists, and continuously transmitting the original video frame to the main control terminal.

Further, the apparatus further includes a second stream saving mode determining module, where the second stream saving mode determining module is configured to:

Further, the apparatus further includes a normal transmission mode determining module, where the normal transmission mode determining module is configured to:

Further, the system state determining module is configured to:

correspondingly, the video transmission target mode determining module is used for:

Further, the device further comprises a screen protection state determining module, wherein the screen protection state determining module is used for:

Further, the device further comprises a screen locking state determining module, wherein the screen locking state determining module is used for:

Further, the device further comprises a screen frame instruction generating module, wherein the screen frame instruction generating module is used for:

In a third aspect, embodiments of the present application provide an electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, the program or instruction implementing the steps of the method according to the first aspect when executed by the processor.

In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, embodiments of the present application provide a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and where the processor is configured to execute a program or instructions to implement a method according to the first aspect.

In the embodiment of the application, a preset system service monitoring application is used for determining the system state of a controlled end in response to a remote Cheng Zhuomian request instruction transmitted by a main control end; according to the system state, determining a video transmission target mode of the system from a preset state mode association relation; if the video transmission target mode is determined to be a first stream saving mode, generating a video difference frame according to an original video frame of a system desktop, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a first stream saving mode ending event matched with a preset event type exists or not; if the first stream saving mode ending event is monitored, switching a video transmission target mode to a normal transmission mode, and continuously transmitting the original video frame to a main control terminal. By the traffic control method for remote video transmission, the system state of the controlled terminal is monitored in real time to determine the corresponding video transmission mode, and the bandwidth consumption can be reduced when the controlled terminal is idle, so that traffic is saved and the transmission rate is improved. And the response to the user operation can be ensured in time, and the real-time performance of remote control is improved.

Drawings

Fig. 1 is a flow chart of a flow control method for remote video transmission according to an embodiment of the present application;

fig. 2 is a flow chart of a flow control method for remote video transmission according to a second embodiment of the present application;

fig. 3 is a flow chart of a flow control method for remote video transmission according to a third embodiment of the present application;

FIG. 4 is a binary block diagram of a screen frame instruction provided in embodiment III of the present application;

fig. 5 is a schematic structural diagram of a flow control device for remote video transmission according to a fourth embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the following detailed description of specific embodiments thereof is given with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the matters related to the present application are shown in the accompanying drawings. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently, or at the same time. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The following describes in detail, by means of specific embodiments and application scenarios thereof, a flow control method, apparatus, device and medium for remote video transmission provided in the embodiments of the present application with reference to the accompanying drawings.

Example 1

Fig. 1 is a flow chart of a flow control method for remote video transmission according to an embodiment of the present application. As shown in fig. 1, the method specifically comprises the following steps:

s101, responding to a far Cheng Zhuomian request instruction transmitted by a main control end, and determining the system state of a controlled end by using a preset system service monitoring application.

Firstly, the usage scenario of the scheme may be a scenario in which the controlled end continuously monitors the own system state after receiving the far Cheng Zhuomian request instruction transmitted by the main control end, determines the corresponding video transmission mode according to the own state, and transmits the original video frame or the video difference frame to the main control end according to the corresponding video transmission mode.

Based on the above usage scenario, it can be understood that the execution subject of the present application may be a controlled end, which is not limited herein.

In the scheme, the method is executed by the controlled terminal.

The controlled end can be a computer which needs to be remotely managed or monitored, and receives and responds to a remote desktop request instruction sent by the main control end.

In this scheme, the master control end may refer to a computer or a device for controlling a remote desktop session, where the master control end may send a remote desktop request instruction to the controlled end and receive screen information and operation input of the controlled end.

The remote desktop request instruction may be a command or signal sent by the master to the slave requesting the establishment of a remote desktop session. This instruction may contain connection information, authorization credentials, etc. to ensure legitimate remote access.

The preset system service monitoring application can be an application program running on the controlled end, and the task of the preset system service monitoring application is to monitor the state of the controlled end in real time and make corresponding decisions. Specifically, the preset system service monitoring application may include a callback function for monitoring a system service message and a function for monitoring a system menu message, where the callback function for monitoring the system service message may be used to determine whether the system is in a screen locking state or a login state; the function of listening to system menu messages may be used to determine if the system is in a screen saver state.

The system state may refer to a current state of the controlled-end computer, and may include a login state, a screen locking state, and a screen protection state. The screen locking state can indicate that the screen of the computer is locked, and a password or other credentials are required to be input to unlock the computer; the login status may indicate that the user has logged into the computer and that the desktop environment is visible. The screen saver state may be when the screen of the computer enters a protected mode, where the screen may display a cyclically played screen saver animation without requiring the user to enter a password to unlock the computer. In this state, although the screen is in the protected mode, the user can be restored to the normal desktop at any time without having to log in again.

The system service listening application on the controlled end may continually listen to the network port or other communication channel to listen for remote desktop request instructions from the master end. Upon receipt of the remote desktop request instruction, the controlled end system service listening application may verify the legitimacy of the request, ensure that the request is from an authorized master and contains the necessary authentication information to prevent unauthorized access. If the request is legitimate, the system service listening application may initiate a remote desktop session to enable the master to access the desktop of the slave. During the remote desktop session, the system service listening application may continuously monitor the system state according to a preset state flag.

S102, determining a video transmission target mode of the system from a preset state mode association relation according to the system state.

The preset association relationship of the state modes may refer to the association relationship of the video transmission modes determined in advance in different system states.

The video transmission mode comprises three transmission modes, namely a first stream-saving mode, a second stream-saving mode and a normal transmission mode, wherein a controlled end continuously transmits video difference frames to a main control end in the first stream-saving mode; the controlled terminal does not transmit any video frame to the main control terminal in the second stream-saving mode; in the normal transmission mode, the controlled end continuously transmits the original video frame to the main control end.

When the system is in a screen locking state or a screen protecting state, the video transmission target mode is set to be a stream saving mode. This is because in these states, the user typically does not interact with the computer, and therefore only differential frames of video need to be transmitted to reduce bandwidth and resource usage.

When the system is in a login state, the video transmission target mode is generally set to a normal mode. This is because the user needs the complete original video frame for normal computer operation and interaction. Therefore, the state mode association relationship between the system state and the video transmission target mode may be represented as a lock screen state-a first stream saving mode; logging status-normal transmission mode; screen saver state-second stream saving mode.

The remote desktop software can be used in advance to set an associated video transmission mode for each state, namely, a preset state mode association relationship is configured in the remote desktop software, and when the system state of the controlled terminal is determined, the corresponding video transmission target mode can be matched in the preset state mode association relationship.

On the basis of the above technical solutions, optionally, after determining the video transmission target mode of the system from the preset state mode association relationship according to the system state, the method further includes:

If the video transmission target mode is determined to be the second stream saving mode, the video frame in any form is not sent to the main control terminal, and whether a second stream saving mode ending event matched with the preset event type exists or not is continuously monitored.

In this scheme, the second streaming saving mode may be a video transmission mode, which is generally used to reduce bandwidth and resource consumption of data transmission. In the second stream-saving mode, the controlled end will not transmit screen content or video to the main control end, so as to save bandwidth and resources. The second streaming saving mode typically corresponds to a screensaver state in which, although animations or screensavers may be running, it is typically not necessary to transmit these animations or images to the master because they are played on the local computer and not viewed at the remote end.

The second stream saving mode end event may be a trigger mechanism or event that indicates that the system needs to end the current second stream saving mode and switch to another video transmission mode when it occurs. Specifically, when the related mouse event, keyboard event and specific operation event are detected, the second stream saving mode ending event matched with the preset event type is regarded as present. A mouse event may be the presence of mouse activity by the user, such as a mouse click or movement; a keyboard event may be the presence of a keyboard operation by a user, such as pressing any key on a keyboard; the specific operation event may be that the user performs a specific operation, such as pressing a certain custom shortcut key.

In the second streaming saving mode, it is ensured that the remote control software stops sending any form of video frames including video difference frames and original video frames to the master. And determining whether there is a second stream saving mode end event matching the preset event type by preset and configuration mechanisms for monitoring the stream saving mode end event, wherein the events can be mouse operation, keyboard input or other user interaction activities of a user. When the related second stream saving mode ending event is monitored, a screen protection state ending instruction can be sent to the server and the main control end, and the controlled end sets the video transmission mode to be a normal transmission mode.

In the scheme, the second stream saving mode is set, so that a user can enjoy high-quality video transmission when in need, bandwidth and resource consumption are reduced when not in need, and user experience is improved. Meanwhile, the second stream-saving mode ending event can enable the controlled terminal to dynamically switch the transmission mode, so that timely response to user operation is ensured.

In this scheme, in the normal transmission mode, the system can continuously capture the original video frames of the whole screen, and transmit the original video frames to the master control end by using image coding and transmission protocols. And finally, setting a transmission system to continuously transmit the original video frames, so that the main control terminal can view and interact with the display of the remote computer in real time.

In the scheme, the original video frames are continuously transmitted in the normal transmission mode, so that high-quality display can be provided, faster response and real-time interaction can be realized, and user experience is improved.

And S103, if the video transmission target mode is determined to be the first stream saving mode, generating a video difference frame according to an original video frame of a system desktop, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a stream saving mode ending event matched with a preset event type exists.

The first streaming saving mode may be a video transmission mode that aims to minimize bandwidth and resource usage. In this mode, not the complete original video frame is transmitted, but parts that are changed from the previous frame, which are called video difference frames, are transmitted, contributing to a reduction in the amount of data transmitted. The first power saving mode typically corresponds to a lock state in which, although most of the user interface content may no longer change, there are still some significant changes that need to be transmitted to the remote control or remote desktop. For example, when the user intends to unlock the screen, a password or other authentication needs to be entered. These inputs are typically made through a keyboard or mouse and need to be transmitted to a remote computer to complete the unlocking operation.

The original video frame may be a complete image of each frame on the screen. It contains information for all visible elements on the screen, typically taking up more bandwidth and resources.

The video difference frame may be a changed portion with respect to the previous frame. It contains only the pixel information that changes on the screen, not the image of the entire screen. Transmitting the video difference frame can significantly reduce the amount of data transmitted because only the portion where the change occurs needs to be transmitted.

The preset event type may be a specific set of user activities or system events for monitoring the situation that the user needs to end the streaming saving mode. The method can comprise a first stream saving mode ending event and a second stream saving mode ending event, wherein the system state corresponding to the first stream saving mode ending event is a screen locking state, and the system state corresponding to the second stream saving mode event is a screen protecting state.

The first stream saving mode end event may be a trigger mechanism or event that indicates that the system needs to end the current first stream saving mode and switch to another video transmission mode when it occurs. The system state corresponding to the first current saving mode is a screen locking state, the screen locking state further comprises a screen locking state and a log-out state in the log-in state, specifically, when a keyboard event or a mouse event is detected, for example, when a user inputs a user name and a password and clicks a related button, the user can be regarded as the user to be converted from the log-out state to the log-in state, a log-in detection function can be started at the moment, an instruction is sent to inform a main control end, if the system user log-in is monitored at the moment, it is determined that a first current saving mode end event exists, and the video transmission mode can be set to a normal transmission mode. If the user inputs the computer unlocking password and clicks the related button, the user can be regarded as the user to switch from the screen locking state in the login state to the screen unlocking state, the screen unlocking detection function can be started, an instruction is sent to the main control end, if the system unlocking is monitored at the moment, the first current saving mode ending event is determined to exist, and the video transmission mode can be set to be a normal transmission mode.

In the first stream saving mode, the controlled end needs to continuously generate the video difference frame. These difference frames are a changing part relative to the previous frame, and image processing techniques can be used to calculate the video difference frames. The generation of the difference frame may be achieved by comparing pixel data of the current frame with that of the previous frame. Only the pixels that have changed are included in the difference frame. The generated video difference frames need to be continuously transmitted to the main control terminal, and specifically, the transmission of the difference frames can be carried out by using a network transmission protocol. And determining whether a first stream saving mode end event matched with a preset event type exists through preset and configuration mechanisms for monitoring the stream saving mode end event.

And S104, if the first stream saving mode ending event is monitored, switching the video transmission target mode to a normal transmission mode, and continuously transmitting the original video frame to a main control terminal.

The normal transmission mode may refer to transmitting the complete original video frame at the highest resolution and image quality to ensure the sharpest display.

The original video frame may be a screen captured image representing the image content in the entire field of view of the screen at a particular point in time. The original video frames are images that have not been compressed or processed, preserving high quality and high resolution details.

When the first stream-saving mode ending event is monitored, the operation of switching the transmission mode is executed, and the transmission mode is switched from the stream-saving mode to the normal transmission mode, specifically, the logic operation can be preset in the remote control software. After switching to the normal transmission mode, the transmission settings may be updated to ensure that the original video frames are transmitted, including setting the appropriate image compression parameters, frame rate, and resolution. In the normal transmission mode, the system can continuously capture the original video frames of the whole screen and transmit the original video frames to the master using the image coding and transmission protocol. Finally, a transmission system is arranged to continuously transmit the original video frames so as to ensure that the main control terminal can view and interact with the display of the remote computer in real time.

Example two

Fig. 2 is a flow chart of a flow control method for remote video transmission according to a second embodiment of the present application. As shown in fig. 2, the method specifically comprises the following steps:

s201, acquiring monitoring parameters and acquiring system menu message types by using a preset system service monitoring application.

The snoop parameter may be information or conditions used to determine the state of the system. In this scheme, the listening parameter may be used to indicate whether the system is currently in a locked or logged on state.

The system menu message type may be a type of information used to monitor whether it is in a screen saver state, and may include specific messages or events in the system menu that may trigger in the screen saver state. Monitoring these messages can be used to determine if the system is in a screen saver state.

A listening parameter that determines whether a system state is in a lock screen state or a login state may be defined in advance in a system service listening application, and a system menu message type that determines whether the system is in a screen-saver state may be defined. When a corresponding listening parameter or system menu message type is detected, a corresponding system state may be determined.

On the basis of the above technical solutions, optionally, after acquiring the listening parameter and acquiring the system menu message type by using a preset system service listening application, the method further includes:

correspondingly, according to the system state, determining a video transmission target mode of the system from a preset state mode association relation, wherein the method comprises the following steps:

and determining the video transmission target mode of the system as a second stream-saving mode.

In this solution, the first message type may be a status flag corresponding to a screen protection status, in this solution, a function of monitoring a system menu message may monitor the system menu message wm_syncomand, capture the wm_syncomand message in a window, determine that the message is executed completely by determining a CommandType of the message, if the message is executed completely, determine that the controlled terminal is in the screen protection status at this time, and associate the screen protection status with the message executed completely, and when the parameter sc_ SCREENSAVE is received, consider that the first message type is obtained, and accordingly determine that the system is in the screen protection status.

Because the state mode association relationship is preset, the first message type, the corresponding system state and the corresponding target stream saving mode can be added in the preset state mode association relationship. In this scheme, the preset state mode association relationship may be represented as a screen protection state-sc_ SCREENSAVE-second stream saving mode, and when the first message type is acquired, the video transmission target mode of the system may be determined to be the second stream saving mode by querying the preset state mode association relationship.

In the scheme, whether the controlled terminal is in the screen protection state or not can be automatically detected by setting the first message type, so that the automation of the system is improved. And prepare for entering the second stream-saving mode automatically afterwards, make users stop transmitting the video frame under the condition of not using the computer temporarily, thus reduce bandwidth occupation and resource consumption, save the flow.

In this scheme, the second monitoring parameter may be a state flag corresponding to the login state, in this scheme, the callback function of the service message of the monitoring system includes different monitoring parameters, and the system may be determined to be in a screen locking state or a login state according to the received monitoring parameters, where specific code statements are as follows:

WTS_SESSION_LOGON(0x5)；

this code statement indicates that the user has logged into the conversation identified by lParam, i.e. the controlled end is in a logged-on state.

WTS_SESSION_UNLOCK(0x8)；

This code statement indicates that the session identified by lParam is unlocked, i.e., the controlled end is in a logged-on state.

The 0x5 and 0x8 in the code statement are corresponding second monitoring parameters, the second monitoring parameters can be associated with the login state, and if the two monitoring parameters are detected, the controlled terminal is in the login state. A loop or timer may then be set to continuously acquire and monitor the value of the listening parameter to ensure that the system is known at any time whether there is a change in state. When the value of the monitoring parameter is continuously acquired and monitored, if the acquired monitoring parameter is changed from the second monitoring parameter to the first monitoring parameter, the system state associated with the first monitoring parameter can be queried, so that the system is determined to be in a screen locking state.

When the system is in a login state, the video transmission target mode of the system can be determined to be a normal transmission mode from a preset state mode association relation, and if the acquired monitoring parameters are converted into first monitoring parameters from second monitoring parameters, the video transmission target mode can be determined to be a first stream saving mode again from the preset state mode association relation.

In the scheme, whether the system state is switched from the login state to the screen locking state or not is monitored in real time, and the first current saving mode can be timely used when a user is switched to the screen locking state, so that bandwidth consumption is reduced, flow is saved, and transmission rate is improved.

S202, if the first monitoring parameter is acquired, determining that the system is in a screen locking state.

The first monitoring parameter may be a state flag corresponding to the screen locking state, in this scheme, the callback function of the monitoring system service message contains different monitoring parameters, and the system can be determined to be in the screen locking state or the login state according to the received monitoring parameters, where specific code statements are as follows:

WTS_SESSION_LOGOFF(0x6)；

this code statement indicates that the user has logged out of the session identified by lParam, i.e. the controlled end is in the lock state.

WTS_SESSION_LOCK(0x7)；

This code statement indicates that the session identified by lParam has been locked, i.e., the controlled end is in a locked state.

The 0x6 and 0x7 in the code statement are corresponding first monitoring parameters, the first monitoring parameters can be associated with the screen locking state, and if the two monitoring parameters are detected, the controlled terminal is in the screen locking state.

S203, determining the video transmission target mode of the system as a first stream saving mode.

Because the state mode association relationship is preset, a monitoring parameter, a corresponding system state and a corresponding target stream saving mode can be added in the preset state mode association relationship. For example, when the system is in a screen locking state, the monitoring parameters are 0x6 and 0x7, the target current saving mode is the first current saving mode, and the preset state mode association relationship can be expressed as a screen locking state-0 x6 and 0x 7-first current saving mode, and after the monitoring parameters are acquired, the video transmission target mode of the system can be determined to be the first current saving mode by inquiring the preset state mode association relationship.

When it is required to detect whether the first stream saving mode end event exists, the detection can be performed by the following codes:

WTS_CONSOLE_CONNECT(0x1)；

this code indicates that the user entered the interface for entering the user's password, but has not entered the desktop, at which point he has not logged in. When detecting that the monitoring parameter of 0x1 exists, the login detection function can be started, whether the user is successfully logged in or not is determined in real time, and if detecting that the monitoring parameters of 0x5 or 0x8 exist, the user is considered to be successfully logged in, and a first stream saving mode ending event exists.

In the embodiment, by setting the monitoring parameters, whether the controlled terminal is in the screen locking state or not can be automatically detected, manual intervention is not needed, and the automation of the system is improved. And the method and the device provide for automatically entering the first stream saving mode later, so that only video difference frames are transmitted in a screen locking state, thereby reducing bandwidth occupation and resource consumption and saving traffic.

Example III

Fig. 3 is a flow chart of a flow control method for remote video transmission according to a third embodiment of the present application. As shown in fig. 3, the method specifically comprises the following steps:

s301, responding to a far Cheng Zhuomian request instruction transmitted by a main control end, and determining the system state of a controlled end by using a preset system service monitoring application.

S302, determining a video transmission target mode of the system from a preset state mode association relation according to the system state.

S303, continuously collecting screen image data, coding the screen image data by using preset coding equipment, and generating coding data and coding data parameters.

The screen image data may refer to image information captured from a screen of the controlled end computer, and may include contents such as images, text, icons, and the like, which are currently displayed on the screen. These image data may be represented as a collection of pixels, typically captured in the form of image frames.

The preset encoding device may be an encoder for encoding the screen image data into a format that can be transmitted or stored. In the scheme, the encoder is used for compressing screen image data, reducing the size of the data while maintaining the original image quality, improving the transmission and image processing efficiency, and expanding and processing partial privacy content. After the screen image data is acquired, the data is sent to an encoder, and the encoder generates corresponding encoded data and other relevant parameters after encoding.

The encoded data may be screen image data that has been subjected to encoding processing. It is a form of compressed data, usually in binary form. The encoded data may be generated by an encoding device for efficient transmission or storage and decoded at a receiving end to restore the original image data.

The encoded data parameters may refer to other generation parameters associated with the encoding format, such as the encoding format of RGB32, ARGB, I420, I444, and NV 12.

The screen image data of the controlled end computer may be captured using a suitable method or library, and in particular, may be captured using an on-screen image frame capture tool provided by the operating system. The captured screen image data is then converted into encoded data using a preset encoding device and associated encoded data parameters are generated.

S304, generating a screen frame instruction according to the encoded data and the encoded data parameters, and transmitting the screen frame instruction to a main control end for the main control end to decode and render the encoded data according to the screen frame instruction.

The screen frame instruction may be a data structure or a message containing metadata that contains information about the encoded data for decoding and rendering at the remote master. These instructions may include coding algorithm type, image quality, image resolution, frame rate, etc.

Fig. 4 is a binary structure diagram of a screen frame instruction provided in the third embodiment of the present application, as shown in fig. 4, at a controlled end, after encoded data and encoded data parameters are generated, a screen frame instruction may be created, specifically, according to the obtained encoded parameters and encoded data parameters, a protobuf protocol may be used to package the screen frame instruction into protobuf data, so that a master end knows how to decode and display an image. Then converting the binary data into binary data by using a method provided by protobuf, and sending the binary data to a master control terminal. After the main control receives the binary data of the screen frame instruction, the binary data is restored into protobuf data, namely the screen frame instruction, by using a protobuf protocol. And determining coding parameters according to parameters carried in the screen frame instruction, setting the vpx decoder, and determining coding data and coding data parameters according to the parameters carried in the screen frame instruction for converting the coding data. And decoding the encoded data by using a vpx decoder to obtain decoded data. The decoded data is converted into applicable image data, for example, image data such as ARGB and RGB32, by libyuv. And finally, providing the image data to a renderer for rendering and on-screen.

S305, if the video transmission target mode is determined to be the first stream saving mode, generating a video difference frame according to the original video frame of the system desktop, continuously transmitting the video difference frame to the main control terminal, and continuously monitoring whether a first stream saving mode ending event matched with the preset event type exists.

And S306, if the first stream saving mode ending event is monitored, switching the video transmission target mode to a normal transmission mode, and continuously transmitting the original video frame to a main control terminal.

In this embodiment, by generating the screen frame command, it is possible to more precisely control how the image is decoded and rendered at the master control end, so as to save network bandwidth and ensure optimal display of the image.

Example IV

Fig. 5 is a schematic structural diagram of a flow control device for remote video transmission according to a fourth embodiment of the present application. As shown in fig. 5, the method specifically includes the following steps:

a system state determining module 501, configured to determine a system state of a controlled terminal using a preset system service listening application in response to a remote Cheng Zhuomian request instruction transmitted by a master terminal;

the video transmission target mode determining module 502 is configured to determine a video transmission target mode of the system according to the system state, if the video transmission target mode is a first stream saving mode, continuously transmit a video difference frame to the master control end, and continuously monitor whether a preset stream saving mode ending event exists;

The video difference frame generating module 503 is configured to generate a video difference frame according to an original video frame of a system desktop if the video transmission target mode is determined to be a first current saving mode, continuously transmit the video difference frame to a master control end, and continuously monitor whether a first current saving mode ending event matched with a preset event type exists;

the video transmission target mode switching module 504 is configured to switch the video transmission target mode to the normal transmission mode and continuously transmit the original video frame to the master control end if the stream saving mode end event exists.

Further, the system state determining module is configured to:

In this embodiment of the present application, the system state determining module is configured to determine, in response to a remote Cheng Zhuomian request instruction transmitted by the master control end, a system state of the controlled end using a preset system service listening application; the video transmission target mode determining module is used for determining a video transmission target mode of the system according to the system state, continuously transmitting a video difference frame to the main control terminal if the video difference frame is in a first stream saving mode, and continuously monitoring whether a preset stream saving mode ending event exists or not; and the video transmission target mode switching module is used for switching the video transmission target mode into a normal transmission mode if the stream saving mode ending event exists, and continuously transmitting the original video frame to the main control terminal. By the flow control device for remote video transmission, the system state of the controlled terminal is monitored in real time to determine the corresponding video transmission mode, and the bandwidth consumption can be reduced when the controlled terminal is idle, so that the flow is saved and the transmission rate is improved. And the response to the user operation can be ensured in time, and the real-time performance of remote control is improved.

The flow control device for remote video transmission in the embodiment of the application may be a device, or may be a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. By way of example, the mobile electronic device may be a cell phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, wearable device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device may be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

A flow control device for remote video transmission in an embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The flow control device for remote video transmission provided in the embodiment of the present application can implement each process implemented by the above method embodiments, and in order to avoid repetition, details are not repeated here.

Example five

As shown in fig. 6, the embodiment of the present application further provides an electronic device 600, including a processor 601, a memory 602, and a program or an instruction stored in the memory 602 and capable of being executed on the processor 601, where the program or the instruction implements each process of the foregoing embodiment of a flow control apparatus for remote video transmission when executed by the processor 601, and the process can achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.

Example six

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the processes of the foregoing embodiment of the flow control device for remote video transmission are implemented, and the same technical effects can be achieved, so that repetition is avoided, and no further description is provided herein.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium such as a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and the like.

Example seven

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or an instruction, implementing each process of the above-mentioned embodiment of the flow control device for remote video transmission, and achieving the same technical effect, so as to avoid repetition, and no further description is provided here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

The foregoing description is only of the preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, but is capable of numerous obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the present application. Therefore, while the present application has been described in connection with the above embodiments, the present application is not limited to the above embodiments, but may include many other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A method for controlling flow of remote video transmission, the method being performed by a controlled end, the method comprising:

if the video transmission target mode is determined to be a first stream saving mode, generating a video difference frame according to an original video frame of a system desktop, continuously transmitting the video difference frame to a main control terminal, and continuously monitoring whether a first stream saving mode ending event matched with a preset event type exists or not;

If the first stream saving mode ending event is monitored, switching a video transmission target mode to a normal transmission mode, and continuously transmitting the original video frame to a main control terminal.

2. The flow control method for remote video transmission according to claim 1, wherein after determining a video transmission target mode of a system from a preset state mode association relationship according to the system state, the method further comprises:

3. The flow control method for remote video transmission according to claim 1, wherein after determining a video transmission target mode of a system from a preset state mode association relationship according to the system state, the method further comprises:

4. The method for controlling a flow of remote video transmission according to claim 1, wherein determining the system status of the controlled terminal using a preset system service listening application comprises:

and determining the video transmission target mode of the system as a first stream saving mode.

5. The method according to claim 4, wherein after acquiring the listening parameter using a preset system service listening application and acquiring the system menu message type, the method further comprises:

6. The method according to claim 4, wherein after acquiring the listening parameter using a preset system service listening application and acquiring the system menu message type, the method further comprises:

7. The method according to claim 1, wherein, before generating a video difference frame according to an original video frame of a system desktop if the video transmission target mode is determined to be the first stream saving mode, continuously transmitting the video difference frame to the master control end, and continuously monitoring whether there is a first stream saving mode ending event matching a preset event type, the method further comprises:

8. A flow control device for remote video transmission, the device being configured at a controlled end, the device comprising:

9. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, the program or instruction when executed by the processor implementing the steps of a method of flow control for remote video transmission as claimed in any one of claims 1 to 7.

10. A readable storage medium, wherein a program or instructions is stored on the readable storage medium, which when executed by a processor, implements the steps of a method of flow control for remote video transmission according to any one of claims 1-7.