CN115562893A

CN115562893A - Image transmission method, device, equipment and storage medium

Info

Publication number: CN115562893A
Application number: CN202211561393.4A
Authority: CN
Inventors: 王新攀; 王轶男
Original assignee: Beijing Zhixiang Technology Co Ltd
Current assignee: Beijing Zhixiang Technology Co Ltd
Priority date: 2022-12-07
Filing date: 2022-12-07
Publication date: 2023-01-03
Anticipated expiration: 2042-12-07
Also published as: CN115562893B

Abstract

The embodiment of the invention provides an image transmission method, an image transmission device, image transmission equipment and a storage medium, belonging to the technical field of computers, wherein the method comprises the following steps: receiving a remote operation instruction sent by a target client; performing target operation on the target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed. According to the method provided by the embodiment of the invention, the first device performs the target operation on the target server according to the remote operation instruction and updates the target data in the shared memory, so that the second device can obtain the updated screen image of the target server when the remote operation is completed based on the target data in the shared memory, the quick transmission of the updated screen image of the target server between the first device and the second device is realized, and the transmission efficiency of the updated region of the target server is improved.

Description

Image transmission method, device, equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to an image transmission method, an image transmission apparatus, an image transmission device, and a storage medium.

Background

In the cloud desktop system, compared with a local operation mode, besides the introduction of user operation, two steps of screen image capture and image coding transmission are added, and the performance of the two steps directly influences the operation performance of a remote desktop.

In the related art, in order to transmit the changed image data between the image capture module and the image coding module, network communication, pipeline communication, queue communication and other modes are required, but the modes are not suitable for transmitting a large amount of data, and the communication brings large delay.

Disclosure of Invention

To solve the problems in the prior art, embodiments of the present invention provide an image transmission method, an image transmission apparatus, an image transmission device, and a storage medium.

Specifically, the embodiment of the invention provides the following technical scheme:

in a first aspect, an embodiment of the present invention provides an image transmission method, including:

receiving a remote operation instruction sent by a target client;

performing target operation on a target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; and the target data is used for the second equipment to acquire a screen image of the target server when the target operation is completed.

Further, the target data includes at least one of: the method comprises the following steps of (1) locking data access, array of change areas, original image memory data, target indication information and coding request marks;

the data access lock is used for locking data;

the change area array is used for representing an area changed in an initial image of a target server after the remote operation instruction is executed;

the memory data of the original image is used for storing image information;

the target indication information is used for indicating that the initial image of the target server side is updated;

the encoding request mark is used for indicating whether the operation authority of acquiring the target data is requested or not.

Further, the operating the target server according to the remote operation instruction to update the target data in the shared memory includes at least one of the following:

under the condition that the coding request mark is set to be a first target value, emptying the coding request mark after waiting for a preset time period; the first target value represents the operation authority of the second equipment for requesting to acquire the target data;

analyzing the operation instruction, and updating the data access lock to acquire the operation authority of the target data under the condition that the initial image of the target server is determined to be changed;

analyzing the operation instruction, and updating target indication information under the condition that the initial image of the target server is determined to be changed;

analyzing the operation instruction, determining a changed area in an initial image of a target server, and updating the changed area in the initial image to the changed area array to obtain an updated changed area array;

and updating the memory data of the original image according to the updated change area array.

Further, the obtaining the operation authority of the target data includes:

acquiring target indication information and operation permission of a change area array in the target data; and/or the presence of a gas in the gas,

and acquiring the target indication information, the array of the change area and the operation authority of the memory data of the original image in the target data.

In a second aspect, an embodiment of the present invention provides an image transmission method, including:

acquiring target data in a shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on a target server according to a remote operation instruction sent by a target client;

acquiring a target image according to the target data in the shared memory; and the target image is a screen image of the target server when the target operation is finished.

the data access lock is used for locking data;

the change area array is used for representing an area which is changed in an initial image of a target server after the remote operation instruction is executed;

the original image memory data is used for storing image information;

Further, the obtaining a target image according to target data in the shared memory includes:

acquiring a change area array in the target data;

updating the memory data of the original image according to the array of the change area;

and acquiring the target image according to the updated memory data of the original image.

Further, before the obtaining the array of change areas in the target data, the method further includes:

acquiring target indication information in the target data;

and acquiring a change area array in the target data set according to the target indication information in the target data.

Further, after the target indication information in the target data is obtained, the method further includes:

and updating the data access lock according to the target indication information in the target data to acquire the operation authority of the target data.

Further, the obtaining the operation authority of the target data includes:

Further, before the updating the data access lock, the method further includes:

setting an encoding request flag in target data to a first target value; the first target value represents an operation authority of the second device for requesting to acquire target data.

In a third aspect, an embodiment of the present invention further provides an image transmission apparatus, including:

the receiving module is used for receiving a remote operation instruction sent by a target client;

the updating module is used for carrying out target operation on the target server according to the remote operation instruction and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed.

In a fourth aspect, an embodiment of the present invention further provides an image transmission apparatus, including:

the first acquisition module is used for acquiring target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client;

the second acquisition module acquires a target image according to target data in the shared memory; the target image is a screen image of the target server when the target operation is completed.

In a fifth aspect, an embodiment of the present invention further provides a first device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the image method according to the first aspect is implemented.

In a sixth aspect, the present invention further provides a second device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the image method according to the second aspect when executing the program.

In a seventh aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the image transmission method according to the first aspect or the image transmission method according to the second aspect.

In an eighth aspect, the present invention further provides a computer program product, which includes a computer program, and when executed by a processor, the computer program implements the image transmission method according to the first aspect or the image transmission method according to the second aspect.

According to the image transmission method, the image transmission device, the image transmission equipment and the storage medium provided by the embodiment of the invention, the first equipment receives the remote operation instruction sent by the target client, and performs target operation on the target server according to the remote operation instruction, so that the target data in the shared memory is updated, the second equipment can quickly and effectively acquire the updated target data of the first equipment based on the shared memory, and further the second equipment can acquire the updated screen image of the target server when the remote operation is completed according to the acquired target data, so that the quick transmission of the updated screen image of the target server between the first equipment and the second equipment is realized, the transmission efficiency of an update area of the target server is improved, the transmission of a large amount of image data between the first equipment and the second equipment can be effectively met, the problem of image transmission delay is solved, and the system overhead is reduced.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

Fig. 1 is a schematic flowchart of an image transmission method according to an embodiment of the present invention;

FIG. 2 is a second flowchart illustrating an image transmission method according to an embodiment of the present invention;

FIG. 3 is a third schematic flowchart of an image transmission method according to an embodiment of the present invention;

FIG. 4 is a fourth flowchart illustrating an image transmission method according to an embodiment of the present invention;

FIG. 5 is a fifth flowchart illustrating an image transmission method according to an embodiment of the present invention;

FIG. 6 is a sixth flowchart illustrating an image transmission method according to an embodiment of the present invention;

fig. 7 is a seventh schematic flowchart of an image transmission method according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an image transmission apparatus according to an embodiment of the present invention;

fig. 9 is a second schematic structural diagram of an image transmission device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of a first device provided in an embodiment of the present invention;

fig. 11 is a schematic structural diagram of a second device provided in an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The method of the embodiment of the invention can be applied to an image transmission scene, and realizes the efficient and rapid transmission of the image.

In the related art, in order to transmit the changed image data between the image capture module (first device) and the image encoding module (second device), network communication, pipeline communication, queue communication, and the like are required, but these methods are not suitable for transmitting a large amount of data, and the communication itself causes a large delay.

According to the image transmission method provided by the embodiment of the invention, the first device updates the target data in the shared memory by receiving the remote operation instruction sent by the target client and performing target operation on the target server according to the remote operation instruction, so that the second device can quickly and effectively acquire the updated target data of the first device based on the shared memory, and further the second device can acquire the updated screen image of the target server when the remote operation is completed according to the acquired target data, so that the quick transmission of the updated screen image of the target server between the first device and the second device is realized, the transmission efficiency of the update area of the target server is improved, the transmission of a large amount of image data between the first device and the second device can be effectively met, the problem of image transmission delay is solved, and the system overhead is reduced.

In order to facilitate clearer understanding of technical solutions of the embodiments of the present application, some technical contents related to the embodiments of the present application are first introduced.

As shown in fig. 1, in the cloud desktop system, a user operation is transmitted to the desktop system through a remote desktop protocol in step 11, and application software in the desktop system responds after receiving an operation request in step 12, and changes are made to a part of the content displayed on a screen in step 13. And then, the remote desktop server acquires the image change in the step 14a or 14b in a screen image capturing mode or the like, compresses and codes the changed image or the full-screen image, and transmits the compressed and coded image or full-screen image to the remote desktop client of the user in the step 15 for displaying back. Compared with a local operation mode, the method has the advantages that two steps of screen image capture and image coding transmission are added besides the user operation incoming. The two-step performance directly affects the remote desktop operational performance.

The conventional method of screen image capture includes step 14a, after the application software completes the image modification, the codec (first device) acquires the screen change by means of timing query or the like. The method has no pertinence in screen image capture, needs a large amount of memory to copy and compare operation, and has poor performance. Or step 14b, monitoring the operation instruction of the application software for updating the screen graph, and transmitting the monitored screen change data with large amount. The screen image capture of the existing scheme needs to copy a large amount of image data periodically or monitor the original image updating process of application software. The resulting overhead is large. In the two modes, network communication, pipeline communication, queue communication and other modes are required to transmit the changed image data between the image capture module (first device) and the image coding module (second device), but the modes are not suitable for transmitting a large amount of data, and the communication brings large delay.

The technical solution of the present invention will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.

Fig. 2 is a flowchart illustrating an embodiment of an image transmission method according to the present invention. As shown in fig. 2, the method provided by this embodiment includes:

step 201, receiving a remote operation instruction sent by a target client;

specifically, in the related art, for the image update acquisition and image encoding transmission process of the remote desktop protocol server, in order to transmit the changed image data between the first device and the second device, network communication, pipe communication, queue communication and other modes are required, but these modes are not suitable for transmitting a large amount of data, and the communication itself causes a large delay; optionally, the first device is an image capture module, and the second device is an image encoding module.

In order to solve the above problem, in the embodiment of the present invention, the first device first receives a remote operation instruction sent by a target client, and optionally, the remote operation instruction is a drawing instruction.

Step 202, performing target operation on a target server according to a remote operation instruction, and updating target data in a shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed.

Specifically, after receiving a remote operation instruction sent by a target client, the first device in the embodiment of the present invention may perform a target operation on a server device that needs to be remotely operated according to the received remote operation instruction; optionally, the target operation may cause the screen image of the server to be updated; when the first device completes target operation on the server according to the remote operation instruction, the target data in the shared memory is also updated; the shared memory is a shared memory between the first device and the second device, the target data is used for the second device to acquire a screen image of the target server when the target operation is completed, optionally, updated screen image information of the target server is recorded in the target data, so that the second device can quickly and effectively acquire the updated target data of the first device based on the shared memory between the first device and the second device, and further the second device can acquire the updated screen image of the target server when the remote operation is completed according to the acquired target data in the shared memory, thereby realizing the quick and effective transmission of the updated screen image of the target server between the first device and the second device; optionally, after acquiring the updated screen image of the target server, the second device may encode the updated image and send the encoded image to the target client, so that the target client acquires the updated screen image of the target server. In the method of the embodiment of the invention, the memory occupation is not additionally increased in the image transmission process, the update region transmission efficiency of the target server side can be effectively improved, the subsequent image coding efficiency is improved, the complete decoupling between the first device and the second device can be realized, and the realization of a remote desktop protocol algorithm in a multi-thread and multi-process environment is facilitated.

For example, the first device receives a remote operation instruction sent by the target client, and when the target operation is completed on the target server according to the remote operation instruction, the screen image a of the target server is updated to be the image a, the first device may record area information of the image a, which changes compared with the image a, in a shared memory between the first device and the second device, and may also record information corresponding to the updated image a in the shared memory, so that the second device may obtain the updated screen image a of the target server according to the information in the shared memory.

According to the method of the embodiment, the first device receives the remote operation instruction sent by the target client, and performs the target operation on the target server according to the remote operation instruction, so as to update the target data in the shared memory, and therefore the second device can quickly and effectively obtain the updated target data of the first device based on the shared memory, and further the second device can obtain the updated screen image of the target server when the remote operation is completed according to the obtained target data, so that the quick transmission of the updated screen image of the target server between the first device and the second device is realized, the transmission efficiency of the update area of the target server is improved, the transmission of a large amount of image data between the first device and the second device can be effectively met, the problem of image transmission delay is solved, and the system overhead is reduced.

In an embodiment, the target data comprises at least one of: the method comprises the following steps of (1) locking data access, array of change areas, original image memory data, target indication information and coding request marks;

the data access lock is used for locking data;

the change area array is used for representing an area which is changed in an initial image of the target server after the remote operation instruction is executed;

the original image memory data is used for storing image information;

the encoding request flag is used for indicating whether the operation authority of acquiring the target data is requested or not.

Specifically, after receiving a remote operation instruction sent by a target client, the first device may perform a target operation on a server device that needs to be remotely operated according to the received remote operation instruction, and update target data in the shared memory. In order to enable the second device to quickly and effectively acquire the screen image of the target server when the target operation is completed based on the target data in the shared memory, the target data in the shared memory updated by the first device includes at least one of the following items: the system comprises a data access lock, a change area array, original image memory data, target indication information and a coding request mark.

Optionally, the data access lock is used for locking data, so that when the first device updates the data in the shared memory, the data in the shared memory can be locked, thereby avoiding the data in the shared memory from being erroneously operated by other devices, and also avoiding time-phase collision when the other devices operate the data in the shared memory with the first device; optionally, the data access lock is a status field in the shared memory, which is used to indicate whether data protected by locking in the shared memory is in an accessible state, and may be lock status data supporting atomic comparison of the operating memory region, the operating system lock handle, or other shared libraries.

Optionally, the change area array is used for indicating an area in which a change occurs in the initial image of the target server after the remote operation instruction is executed, so that which area in the screen of the target server is changed can be quickly determined according to the change area array in the shared memory, the change area in the screen of the target server is more specific and detailed, the data processing amount is reduced, and the transmission efficiency is improved; optionally, the change area array is used for recording the display area affected by the image drawing command between two image codes, and the change area array comprises a plurality of rectangles which are not intersected with each other, and the union of the rectangles represents the affected display area.

Optionally, the original image memory data is used for storing image information, so that an image of the target server can be acquired based on the image memory data; optionally, the original image memory records the brightness and color information of each screen pixel in a matrix manner.

Optionally, the target indication information is used to indicate that the initial image of the target server has been updated, so that the second device can timely and effectively know that the screen image of the target client has been updated according to the target indication information in the shared memory, and thus the second device can timely acquire the updated image of the target server. Optionally, the target indication information may include a first counter, and when the first counter is increased, the screen image of the target client is updated; and the degree of the increase of the first counter can indicate the size of the screen image change area of the target client and the degree of the change.

The encoding request flag is used for indicating whether to request to acquire the operation authority of the data in the shared memory, so that the second device can request to acquire the operation authority of the target data through the encoding request flag, the second device can preferentially operate the data in the shared memory, and time conflict between the second device and the first device when the second device operates the data in the shared memory is avoided. Optionally, when the second device cannot acquire the data access lock for a long time, the second device may learn, by the first device, that the second device needs to acquire the operation permission of the data in the shared memory by updating the coding request flag in the shared memory, so as to improve the interactivity between the first device and the second device.

Optionally, the operating the target server according to the remote operation instruction, and updating the target data in the shared memory includes at least one of:

under the condition that the coding request mark is set as a first target value, the coding request mark is emptied after waiting for a preset time period; the first target value represents the operation authority of the second device for requesting to acquire the target data;

analyzing the operation instruction, updating the data access lock and acquiring the operation authority of the target data under the condition that the initial image of the target server is determined to be changed;

analyzing the operation instruction, and updating the target indication information under the condition of determining that the initial image of the target server side is changed;

analyzing the operation instruction, determining a changed area in an initial image of the target server, updating the changed area in the initial image to a change area array, and obtaining an updated change area array;

Specifically, in the embodiment of the present invention, after operating a target server according to a remote operation instruction, a first device updates target data in a shared memory in time, so as to implement recording of an update area of a server image and an updated server image; and then the second device can quickly and effectively acquire the updated screen image of the target server based on the target data in the shared memory.

Optionally, the first device updates the target data in the shared memory, where the updating includes checking, by the first device, a coding request flag in the target data when a screen image of the target server changes, and if the coding request flag is set as a first target value, indicating that the second device requests to acquire an operation permission of the target data, after the first device waits for a preset time period, that is, after the second device completes an operation, the first device may clear the coding request flag in the shared memory, so that the first device may acquire the operation permission of the data in the shared memory, and thus update the data in the shared memory.

Optionally, the updating of the target data in the shared memory by the first device includes that the first device analyzes the operation instruction, and when it is determined that the target operation corresponding to the operation instruction may affect the initial image of the target server to be changed, the first device may update the data access lock in the shared memory, so that after the first device acquires the authority, the first device may update the data in the shared memory, thereby implementing the recording of the updated region of the server image and the updated server image. Optionally, in the process of performing operation update on the data in the shared memory by the first device and the second device, only one device can execute the part protected by the data access lock at the same time, and the other device needs to wait to acquire the lock. Therefore, the shorter the execution time of the part to be protected is, the higher the parallelism of the two processes is, and the higher the image processing efficiency is.

Optionally, the updating of the target data in the shared memory by the first device includes analyzing the operation instruction, and updating the target indication information in the shared memory when determining whether the target operation corresponding to the operation instruction will cause the initial image of the target server to change, so that the second device can know that the screen image of the target server has changed according to the target indication information in the shared memory.

Optionally, the updating of the target data in the shared memory by the first device includes analyzing the operation instruction, and updating the changed area in the initial image of the target server to the changed area array when determining whether the target operation corresponding to the operation instruction will cause the change of the initial image of the target server, so that the second device can quickly determine which area in the screen of the target server has been changed based on the changed area array in the shared memory, so that the changed area in the screen of the target server is more specific and detailed, the data processing amount is reduced, and the transmission efficiency is improved.

Optionally, the first device updates the target data in the shared memory, including updating the original image memory data in the shared memory according to the updated change area array, so that the second device can obtain the updated screen image of the target server according to the original image memory data in the shared memory.

For example, as shown in fig. 3, the process of the first device operating the target server according to the remote operation instruction and updating the target data in the shared memory specifically includes:

when the first device receives the drawing command, it first checks whether the encoding request flag in the shared memory is set. If not, an action indicating that the second device does not have an emergency requires the acquisition of a data access lock, so the first device can directly start the image command parsing step. If the image command is set, the first device needs to wait for a certain time before starting to analyze the image command, so that the second device has an opportunity to obtain a data access lock. Optionally, after the waiting time is over, the first device needs to clear the flag bit in the shared memory. In the image command analysis step, the first device acquires whether the image command updates a screen display area visible to a user and a screen display area modified by the target server after the operation instruction is executed.

And after the first device finishes analyzing the operation instruction, formally executing the target operation and finishing updating the shared memory. The process is as follows: a data access lock is first acquired. And acquiring an update counter in the target indication information after locking, and executing increment operation, wherein the increment is determined according to the size of the modified display area, so that the second equipment can evaluate the urgency of starting the next screen coding action according to the value of the update counter in the target indication information. And the first device updates the array of the change area after acquiring the lock, so that the whole change area represented by the array is the union of the complete screen change area before executing the drawing command and a newly added screen change area caused by the command. And after the lock is acquired, starting to specifically execute the drawing command, and updating the original image memory influenced by the drawing command. After the steps are executed, the first device releases the data access lock and waits for a new drawing command to arrive. It should be noted that, in the embodiment of the present invention, an image command sent by application software is also monitored. However, after the image command is listened to, the additional operations are only: and acquiring a data access lock, increasing an update counter in the target indication information, and updating the change area array. And complicated memory copying and interprocess communication mechanisms are not involved. There is little overhead in processing a large number of image commands.

In the method of the embodiment, the first device can effectively record the updated screen image of the target server in the shared memory by updating the target data in the shared memory, so that the second device can timely and effectively know that the screen image of the target client is updated according to the target data in the shared memory, and timely and accurately acquire the updated image of the target server, thereby realizing the rapid transmission of the updated screen image of the target server between the first device and the second device, and improving the transmission efficiency of the update area of the target server.

In one embodiment, the obtaining of the operation authority of the target data includes:

acquiring target indication information and operation authority of a change area array in target data; and/or the presence of a gas in the gas,

Specifically, the first device may update the data access lock and obtain the operation permission of the data in the shared memory when the first device analyzes the operation instruction and determines that the initial image of the target server is changed; optionally, the first device may obtain the target indication information, the change area array, and the operation permission of the original image memory data in the target data, and the first device may also reduce the operation range protected by the data access lock, and only obtain the target indication information and the operation permission of the change area array in the target data, so as to shorten the data access lock locking time, so that the target data in the shared memory may be reasonably and effectively utilized by the first device and the second device, and improve the parallel execution degree of the first device and the second device on the target data in the shared memory and the utilization efficiency of the target data in the shared memory.

For example, as shown in fig. 4, the first device may reduce the operation range protected by the data access lock, and only obtain the target indication information and the operation permission of the change area array in the target data, so that the locking time of the data access lock is shortened, the target data in the shared memory may be reasonably and effectively utilized by the first device and the second device, and the parallel execution degree of the first device and the second device on the target data in the shared memory and the utilization efficiency of the target data in the shared memory are improved. Optionally, in the process of performing operation and update on data in the shared memory by the first device and the second device, only one device may execute the part protected by the data access lock at the same time, and the other device needs to wait to acquire the lock. Therefore, the shorter the execution time of the locked portion is, the higher the parallel execution degree of the target data in the shared memory is, and the higher the image processing efficiency is. Illustratively, as shown in fig. 4, to improve the execution parallelism of the first device and the second device, the operating range protected by the data access lock may be reduced in the case where a short screenout of the remote desktop client display is allowed to occur. The steps of rendering the image commands (executing the operation instructions) and updating the original image memory can be put outside the data access lock. Optionally, in the process of capturing the updated screen image of the target server, the steps of increasing the update counter and updating the array of the change area in the target indication information may be completed quickly, and the steps of rendering the image command (executing the operation instruction) and updating the memory of the original image are related to the complexity of the image command and the size of the update area, which may take a long time. After the operation range protected by the data access lock is reduced, the locking time of the data access lock of the first device caused by the image capturing process can be very short, so that the execution parallelism of the first device and the second device can be effectively improved.

In the method of the above embodiment, when the first device obtains the operation right of the target data in the shared memory, the locking time of the data access lock may be shortened by reducing the operation range protected by the data access lock, so that the target data in the shared memory may be reasonably and effectively utilized by the first device and the second device, and the parallel execution degree of the first device and the second device on the target data in the shared memory and the utilization efficiency of the target data in the shared memory are improved.

Fig. 5 is a flowchart illustrating an image transmission method according to another embodiment of the present invention. As shown in fig. 5, the method provided by this embodiment includes:

step 501, the second device obtains target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client;

specifically, the first device performs target operation on the target server according to the remote operation instruction, after the target data in the shared memory is updated, the target data records the screen update area of the target server and information of the updated screen image of the target server, and then the second device can acquire the updated target data in the shared memory between the first device and the second device.

Step 502, obtaining a target image according to target data in a shared memory; the target image is a screen image of the target server when the target operation is completed.

Specifically, after the second device obtains the updated target data from the shared memory between the first device and the second device, the second device may obtain the screen image of the target server when the target operation is completed according to the information of the updated screen area of the target server and the updated screen image of the target server recorded in the shared memory.

According to the method of the embodiment, after the second device obtains the updated target data from the shared memory between the first device and the second device, the updated screen image of the target server when the remote operation is completed can be obtained according to the information of the updated screen area of the target server and the updated screen image of the target server recorded in the target data, so that the updated screen image of the target server can be quickly transmitted between the first device and the second device, the transmission efficiency of the updated screen area of the target server is improved, the transmission of a large amount of image data between the first device and the second device can be effectively met, the problem of image transmission delay is solved, and the system overhead is reduced.

the data access lock is used for locking data;

the original image memory data is used for storing image information;

Specifically, in order to enable the second device to quickly and effectively acquire the screen image of the target server when the target operation is completed based on the target data in the shared memory, optionally, the target data in the shared memory acquired by the second device includes at least one of the following: the system comprises a data access lock, a change area array, original image memory data, target indication information and a coding request mark.

Optionally, the data access lock is used for locking the data, so that when the second device acquires the data in the shared memory, the data in the shared memory can be locked, thereby avoiding the data in the shared memory from being erroneously operated by other devices, and also avoiding time collision when the other devices and the second device operate the data in the shared memory; optionally, the data access lock is a status field in the shared memory, which is used to indicate whether data protected by locking in the shared memory is in an accessible state, and may be lock status data supporting atomic comparison of the operating memory region, the operating system lock handle, or other shared libraries.

Optionally, the change area array is used for indicating an area in which a change occurs in the initial image of the target server after the remote operation instruction is executed, so that the second device can quickly determine which area in the screen of the target server has been changed according to the change area array, so that the change area in the screen of the target server acquired by the second device is more specific and detailed, the data processing amount is reduced, and the transmission efficiency is improved; optionally, the change area array is used for recording the display area affected by the image drawing command between two image codes, and the change area array comprises a plurality of rectangles which are not intersected with each other, and the union of the rectangles represents the affected display area.

Optionally, the original image memory data is used for storing image information, so that the second device can acquire the image of the target server based on the image memory data in the shared memory; optionally, the original image memory records the brightness and color information of each screen pixel in a matrix manner.

Optionally, the target indication information is used to indicate that the initial image of the target server has been updated, so that after the second device acquires the target indication information in the shared memory, the second device can timely and effectively know that the screen image of the target client has been updated, and thus the second device can timely acquire the updated image of the target server. Optionally, the target indication information may include a first counter, and when the first counter is increased, the screen image of the target client is indicated to be updated; and the degree of the increase of the first counter can indicate the size of the screen image change area of the target client and the degree of the change.

Optionally, the encoding request flag is used to indicate whether to request to acquire the operation permission of the target data, so that the second device may request to acquire the operation permission of the target data through the encoding request flag, thereby preferentially operating the target data in the shared memory, and avoiding time phase conflict between the second device and the first device when operating the data in the shared memory. Optionally, when the second device cannot acquire the data access lock for a long time, the second device may learn, by the first device, that the second device needs to acquire the operation permission of the data in the shared memory by updating the coding request flag in the shared memory, so as to improve the interactivity between the first device and the second device.

In one embodiment, acquiring a target image according to target data in a shared memory includes:

acquiring a change area array in target data;

and acquiring a target image according to the updated memory data of the original image.

Specifically, the change area array in the shared memory between the first device and the second device is used to indicate an area in which a change occurs in the initial image of the target server when the target operation is completed, so that the second device can determine which area in the screen of the target server has changed by acquiring the change area array in the shared memory, and change the original screen image of the target server recorded in the original image memory data, that is, can acquire the updated screen image of the target server, thereby implementing fast transmission of the updated screen image of the target server between the first device and the second device.

According to the method of the embodiment, the second device obtains the change area array in the shared memory, so that the second device can quickly determine which area in the screen of the target server is changed according to the change area array, the change area in the screen of the target server obtained by the second device is more specific and detailed, the data processing amount is reduced, and the transmission efficiency is improved; furthermore, the second device updates the memory data of the original image according to the acquired array of the change area, so that the updated screen image of the target server can be acquired, and the quick transmission of the updated screen image of the target server between the first device and the second device is realized.

In one embodiment, before obtaining the array of change areas in the target data, the method further includes:

acquiring target indication information in target data;

Specifically, after the second device acquires the target indication information in the shared memory, the second device can timely and effectively know that the screen image of the target client is updated based on the target indication information, and then the second device can timely acquire the updated screen image of the target server, so that the efficiency of acquiring the updated image of the target server by the second device is improved.

According to the method of the embodiment, the second device can timely and effectively know that the screen image of the target server is updated by acquiring the target indication information in the shared memory, so that the updated screen image of the target server is timely acquired, and the efficiency of acquiring the updated image of the target server by the second device is improved.

In an embodiment, after the target indication information in the target data is acquired, the method further includes:

Specifically, after the second device obtains the target indication information in the target data, if the target indication information indicates that the screen image of the target server has been updated, the second device may lock the data in the shared memory, avoid the other devices performing a misoperation on the data in the shared memory, and also avoid time phase conflicts when the other devices and the second device operate on the data in the shared memory, so that the second device may effectively obtain the updated screen image of the target server based on the data in the shared memory.

For example, as shown in fig. 6, the process of the second device obtaining the target image according to the target data in the shared memory specifically includes:

firstly, the second device checks whether the target indication information in the shared memory indicates that the original image memory of the target server side is changed, and the change area is represented by a change area array. If the target indication information in the shared memory indicates that the original image of the target server is unchanged, the second device does not need to acquire the screen image of the target server, and only needs to wait for the next updating check time and then check again. Under the condition that the target indication information indicates that the original image memory of the target server side is changed, executing the following steps: firstly, acquiring a data access lock; if the second device cannot acquire the data access lock for a long time, the second device may set a coding request flag and notify the first device, so that the second device gets a chance to acquire the lock. Optionally, after the second device acquires the lock, the second device clears the target indication information, clears the change area array in the shared memory after acquiring the complete content of the change area array in the shared memory, determines the modified screen display area of the target server according to the acquired change area array, and acquires the original image memory data of the modified area. After the above steps are completed, the second device releases the data access lock. Optionally, after the second device releases the data access lock, an image encoding step is started, which implements encoding, compression, and conversion of pixel data in the memory of the original image, and takes more computation time. And after the image coding step is completed, repeating the circulating process, and checking again after the next updating checking time comes. In the embodiment of the application, communication is performed only through the state in the shared memory and the original image memory area between the image capture process of the first device and the image coding process of the second device. By sharing the memory, not only the image pixel change is transmitted, but also the change area is transmitted, so that the image coding can be efficiently executed aiming at the change area; the set updating counter and the coding request mark in the target indication information enable the two processes to be in high-efficiency synchronization state, and reduce coding delay caused by state synchronization between the two processes.

The data and the marks in the shared memory of the first device and the second device can correctly work in a cross-process mode, and can correctly work between the virtual machine and the host machine in the cloud desktop environment. Therefore, the isolation between the remote desktop image coding and the user operation environment can be realized to a greater extent, and the safety of the cloud desktop system is guaranteed. Optionally, if after the first device finishes image capture in the cloud desktop, the encoding transmission of the second device is finished outside the cloud desktop, and it can be ensured that higher data security exists between the cloud desktop and the user client.

In the method of the above embodiment, the second device locks the data in the shared memory, so as to avoid the data in the shared memory from being misoperated by other devices, and also avoid time phase collision when the other devices and the second device operate the data in the shared memory, so that the second device can further and effectively obtain the updated screen image of the target server.

Specifically, when the second device acquires the target indication information in the target data and the target indication information indicates that the screen image of the target server is updated, the second device may update the data access lock and acquire the operation permission of the data in the shared memory, so that time conflict between other devices and the second device when the other devices operate the data in the shared memory is avoided, and the second device may effectively acquire the updated screen image of the target server; optionally, the second device may obtain the target indication information, the change area array, and the operation permission of the original image memory data in the shared memory, and the second device may also reduce the operation range protected by the data access lock, and only obtain the target indication information and the operation permission of the change area array in the shared memory, so that the data access lock locking time is shortened, the data in the shared memory may be reasonably and effectively utilized by the first device and the second device, and the parallel execution degree of the first device and the second device on the data in the shared memory and the utilization efficiency of the target data in the shared memory are improved.

For example, as shown in fig. 7, the second device may reduce the operation range protected by the data access lock, and only obtain the target indication information and the operation permission of the change area array in the shared memory, so that the locking time of the data access lock is shortened, the target data in the shared memory may be reasonably and effectively utilized by the first device and the second device, and the parallel execution degree of the first device and the second device on the target data in the shared memory and the utilization efficiency of the target data in the shared memory are improved. Optionally, in the process of performing operation update on the data in the shared memory by the first device and the second device, only one device can execute the part protected by the data access lock at the same time, and the other device needs to wait to acquire the lock. Therefore, the shorter the execution time of the part to be protected is, the higher the parallelism of the two processes is, and the higher the image processing efficiency is. Illustratively, as shown in fig. 7, to improve the execution parallelism of the first device and the second device, the operating range protected by the data access lock may be reduced in the case where a short screenout of the remote desktop client display is allowed to occur. The step of acquiring the original image memory data of the modified area by the second device may be put outside the data access lock. At this time, the steps of clearing the target indication information protected by the data access lock, obtaining the array content of the change area and clearing the array of the change area are carried out, and the data access lock is released quickly because the actual execution speed is high. And after the data access lock is released, and the steps of acquiring the memory data of the original image in the modified area in a time-consuming manner and encoding the image are completed, entering the cycle period of the next image encoding process. After the operation range protected by the data access lock is reduced, the locking time of the data access lock of the second device caused by the image coding process can be very short, so that the execution parallelism of the first device and the second device can be effectively improved.

Optionally, when the first device is an image capture module and the second device is an image encoding module, the image capture process and the image encoding process are executed in cooperation, and shared memory data used together are protected by a data access lock. When the screen image is updated rapidly, multiple image capture processes may occur between two image encoding processes. When the screen image update is slow, there may be no image capture process between the two image encoding processes.

Through the steps, image capture of the first device and data transmission and state synchronization between image coding processes of the second device can be completely realized through data and marks in the shared memory, and other modes such as network communication, pipeline communication, queue communication and the like are not needed for assistance. By means of a memory sharing mode, image capture and image coding of the remote desktop server can work correctly in a cross-process mode, and can work correctly between the virtual machine and the host machine in a cloud desktop environment.

According to the method in the embodiment, when the second device obtains the operation right of the data in the shared memory, the operation range protected by the data access lock can be reduced, so that the locking time of the data access lock is shortened, the target data in the shared memory can be reasonably and effectively utilized by the first device and the second device, and the parallel execution degree of the target data in the shared memory and the utilization efficiency of the target data in the shared memory by the first device and the second device are improved.

In one embodiment, before updating the data access lock, the method further comprises:

setting an encoding request flag in target data to a first target value; the first target value represents an operation right of the second device to request acquisition of the target data.

Specifically, if the second device cannot acquire the data access lock for a long time, the second device may set the coding request flag and notify the first device, so that the second device has an opportunity to acquire the lock. The encoding request flag is used for indicating whether to request to acquire the operation permission of the data in the shared memory, so that the second device can request to acquire the operation permission of the target data through the encoding request flag, the target data in the shared memory is preferentially operated, time phase conflict between the second device and the first device when the second device operates the data in the shared memory is avoided, and interactivity between the first device and the second device is improved.

In the method of the foregoing embodiment, the second device sets the coding request flag in the shared memory to the first target value, so that the second device can request the operation permission to acquire the target data through the coding request flag, so that the second device preferentially operates the target data in the shared memory, and avoids a time conflict when the second device and the first device operate the data in the shared memory. Optionally, when the second device cannot acquire the data access lock for a long time, the second device may be known by the first device by updating the code request flag in the shared memory, so as to improve the interactivity between the first device and the second device.

The image transmission device provided by the present invention is described below, and the image transmission device described below and the image transmission method described above may be referred to in correspondence with each other.

Fig. 8 is a schematic structural diagram of an image transmission device provided by the present invention. The image transmission device provided by the embodiment comprises:

the receiving module 810 is configured to receive a remote operation instruction sent by a target client;

an updating module 820, configured to perform a target operation on the target server according to the remote operation instruction, and update target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed.

Optionally, the target data comprises at least one of: the method comprises the following steps of (1) locking data access, array of change areas, original image memory data, target indication information and coding request marks;

the data access lock is used for locking data;

the original image memory data is used for storing image information;

Optionally, the update module 820 is specifically configured to: under the condition that the coding request mark is set to be a first target value, emptying the coding request mark after waiting for a preset time period; the first target value represents the operation authority of the second device for requesting to acquire the target data;

analyzing the operation instruction, updating the data access lock under the condition of determining that the initial image of the target server side is changed, and acquiring the operation authority of the target data;

analyzing the operation instruction, determining a changed area in the initial image of the target server, and updating the changed area in the initial image to a change area array to obtain an updated change area array;

Optionally, the update module 820 is specifically configured to: acquiring target indication information and operation authority of a change area array in target data; and/or the presence of a gas in the gas,

Fig. 9 is a schematic structural diagram of an image transmission device provided by the present invention. The image transmission device provided by the embodiment comprises:

a first obtaining module 910, configured to obtain target data in a shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client;

a second obtaining module 920, obtaining a target image according to target data in the shared memory; the target image is a screen image of the target server when the target operation is completed.

the data access lock is used for locking data;

the original image memory data is used for storing image information;

Optionally, the second obtaining module 920 is specifically configured to:

acquiring a change area array in target data;

Optionally, the first obtaining module 910 is specifically configured to obtain target indication information in target data;

Optionally, the first obtaining module 910 is further configured to:

and updating the data access lock according to the target indication information in the target data, and acquiring the operation authority of the target data.

Optionally, the first obtaining module 910 is specifically configured to obtain target indication information in the target data and an operation permission of the change area array; and/or the presence of a gas in the gas,

Optionally, the first obtaining module 910 is specifically configured to set an encoding request flag in target data to a first target value; the first target value represents an operation authority of the second device for requesting to acquire the target data.

The apparatus of the embodiment of the present invention is configured to perform the method of any of the foregoing method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.

Fig. 10 illustrates a physical structure diagram of a first device, and the electronic device may include: a processor (processor) 1010, a communication Interface (Communications Interface) 1020, a memory (memory) 1030, and a communication bus 1040, wherein the processor 1010, the communication Interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. Processor 1010 may invoke logic instructions in memory 1030 to perform an image transfer method comprising: receiving a remote operation instruction sent by a target client; performing target operation on the target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed.

Furthermore, the logic instructions in the memory 1030 can be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Fig. 11 illustrates a physical structure diagram of a second device, and the electronic device may include: a processor (processor) 1110, a communication Interface (Communications Interface) 1120, a memory (memory) 1130, and a communication bus 1140, wherein the processor 1110, the communication Interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. Processor 1110 may invoke logic instructions in memory 1130 to perform an image transfer method comprising: acquiring target data in a shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client; acquiring a target image according to target data in the shared memory; the target image is a screen image of the target server when the target operation is completed.

In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the image transmission method provided by the above methods, the method comprising: receiving a remote operation instruction sent by a target client; performing target operation on a target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second equipment to obtain a screen image of the target server when the target operation is completed; or, acquiring target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client; acquiring a target image according to target data in the shared memory; the target image is a screen image of the target server when the target operation is completed.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the image transmission method provided above, the method comprising: receiving a remote operation instruction sent by a target client; performing target operation on the target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second equipment to obtain a screen image of the target server when the target operation is completed; or, acquiring target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on the target server according to a remote operation instruction sent by the target client; acquiring a target image according to target data in the shared memory; the target image is a screen image of the target server when the target operation is completed.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. An image transmission method applied to a first device includes:

receiving a remote operation instruction sent by a target client;

performing target operation on a target server according to the remote operation instruction, and updating target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is used for the second device to obtain a screen image of the target server when the target operation is completed.

2. The image transmission method according to claim 1, wherein the target data includes at least one of: the method comprises the following steps of (1) locking data access, array of change areas, original image memory data, target indication information and coding request marks;

the data access lock is used for locking data;

the memory data of the original image is used for storing image information;

3. The image transmission method according to claim 2, wherein the operating the target server according to the remote operation instruction to update the target data in the shared memory includes at least one of:

4. The image transmission method according to claim 3, wherein the operation authority for acquiring the target data comprises:

5. An image transmission method applied to a second device includes:

6. The image transmission method according to claim 5, wherein the target data includes at least one of: the method comprises the following steps of (1) locking data access, array of change areas, original image memory data, target indication information and coding request marks;

the data access lock is used for locking data;

the original image memory data is used for storing image information;

7. The image transmission method according to claim 6, wherein the obtaining the target image according to the target data in the shared memory comprises:

acquiring a change area array in the target data;

8. The image transmission method according to claim 7, wherein before the acquiring the array of change areas in the target data, the method further comprises:

acquiring target indication information in the target data;

9. The image transmission method according to claim 7 or 8, wherein after the acquiring the target indication information in the target data, further comprising:

10. The image transmission method according to claim 9, wherein the operation authority for acquiring the target data includes:

11. The image transmission method according to claim 9, wherein before the updating the data access lock, further comprising:

12. An image transmission apparatus, comprising:

13. An image transmission apparatus, comprising:

the first acquisition module is used for acquiring target data in the shared memory; the shared memory is a shared memory between the first device and the second device; the target data is determined after the first device performs target operation on a target server according to a remote operation instruction sent by a target client;

the second acquisition module acquires a target image according to the target data in the shared memory; and the target image is a screen image of the target server when the target operation is finished.

14. A first device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, implements the image transmission method according to any one of claims 1 to 4.

15. A second device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the image transmission method according to any of claims 5 to 11 when executing the program.

16. A non-transitory computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the image transmission method according to any one of claims 1 to 4 or the image transmission method according to any one of claims 5 to 11.