CN105786434B

CN105786434B - A kind of control method and equipment remotely shown

Info

Publication number: CN105786434B
Application number: CN201610164719.8A
Authority: CN
Inventors: 周铭鹏
Original assignee: Beijing Star Net Ruijie Networks Co Ltd
Current assignee: Beijing Star Net Ruijie Networks Co Ltd
Priority date: 2016-03-22
Filing date: 2016-03-22
Publication date: 2019-09-20
Anticipated expiration: 2036-03-22
Also published as: CN105786434A

Abstract

The invention discloses control methods and equipment that one kind is remotely shown, it is related to mobile Internet field, solves the problems, such as that the screenshot capture at per a moment is passed to control equipment to realize that this method of Screen sharing is unable to get preferable application in tele-control system based on android system by controlled device caused by limiting due to bandwidth.Scheme are as follows: control equipment receives the first drawing for order that controlled device is sent, first drawing for order includes for notifying control equipment controlled device to create the creation information of first surface, according to creation information creating second surface, receive the second drawing for order that controlled device is sent, second drawing for order includes the controlled device drafting operational order used when first surface draws all elements, drafting operation is carried out in second surface according to operational order is drawn, to draw out the identical element of all elements drawn with controlled device in first surface in second surface.During the present invention is for remotely controlling.

Description

Remote display control method and device

Technical Field

The invention relates to the field of mobile internet, in particular to a remote display control method and device.

Background

With the continuous development of mobile networks, mobile digital terminals such as mobile phones and tablet computers are increasingly popularized in daily life of people, and it is well known that an operating system for managing and controlling hardware and software of the mobile digital terminal is installed in the mobile digital terminal, wherein an Android (english: Android) system is one of the frequently installed operating systems. With the rapid development of communication technology, the application of the Android system is far beyond the scope of mobile digital terminals, and digital photo frames, internet protocol phones (VOIP), video conference terminals and other terminal devices based on the Android system are also successively appeared. In order to realize data interaction with these terminal devices, remote control has come to work, for example, data management of a mobile phone with an Android system can be realized by using a computer by adopting a remote control technology.

In the prior art, to monitor and manage a controlled device (e.g., a mobile phone) by a control device (e.g., a computer or a mobile phone), a remote management technology is required, a simple protocol Independent Computing Environment (english: simple protocol for Independent Computing Environment, abbreviated as "SPICE") protocol is commonly used at present, and how to display contents displayed on the controlled device on the control device, that is, screen sharing is a basis for realizing monitoring and management. In the SPICE protocol, a Display Channel (english: Display Channel) is provided to handle the Display function, wherein the Display buffer of the controlled device is mainly passed through by the Display Channel, that is, the controlled device passes through the Display Channel to transfer the screenshot of each moment to the control device, so that the control device displays the screenshot, thereby achieving the purpose of screen sharing.

The prior art has at least the following problems: the controlled device realizes screen sharing by transmitting the screen capture of each moment to the control device, and certain bandwidth is needed in the process of transmitting the screen capture, and in this case, the method cannot be well applied due to the limitation of the bandwidth for the remote control system based on the Android system.

Disclosure of Invention

The invention provides a control method and equipment for remote display, which solve the problem that the method for realizing screen sharing by transmitting each moment of screenshot to control equipment by the control equipment due to bandwidth limitation cannot be well applied to a remote control system based on an Android system.

In order to achieve the purpose, the invention adopts the following technical scheme:

in a first aspect of the present invention, a method for controlling remote display is provided, where the method is applied to a control device in a remote control system, the control device is used to implement remote control on a controlled device in the remote control system, and the controlled device is installed with an android system, and the method includes:

the control device receives a first drawing instruction sent by the controlled device; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface;

the control device creates a second surface according to the creation information;

the control device receives a second drawing instruction sent by the controlled device; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface;

and the control device performs drawing operation on the second surface according to the drawing operation instruction so as to draw the same elements as all the elements drawn on the first surface by the controlled device on the second surface.

With reference to the first aspect, in a possible implementation manner, the first drawing instruction further includes: the identification information of the first surface, before the control device receives the second drawing instruction sent by the control device, the method further includes:

the control equipment establishes a mapping relation between the second surface and the identification information of the first surface;

the second drawing instruction further comprises: the identification information of the first surface, before the control device performs the drawing operation on the second surface according to the drawing operation instruction, further includes:

the control device determines that the drawing operation instruction is used for executing the drawing operation on the second surface according to the identification information of the first surface and the mapping relation included in the second drawing instruction.

With reference to the first aspect and the foregoing possible implementation manners, in another possible implementation manner, after the controlling device performs a drawing operation on the second surface according to the drawing operation instruction, the method further includes:

the control device receives a third drawing instruction sent by the controlled device, where the third drawing instruction includes: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, direction, position, transparency, whether hidden or not displayed; the control equipment changes the attribute information of the second surface, which is the same as that of the first surface, according to the change information;

and/or the presence of a gas in the gas,

the control device receives a fourth drawing instruction sent by the controlled device, where the fourth drawing instruction includes: destruction information for notifying the control device that the first surface is destroyed by the control device; and the control equipment destroys the second surface according to the destruction information.

In a second aspect of the present invention, a control method for remote display is provided, where the control method is applied to a controlled device in a remote control system, the control device in the remote control system is used to implement remote control on the controlled device, and the controlled device is installed with an android system, and the method includes:

the controlled device creates a first surface and sends a first drawing instruction to the control device; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface;

the controlled device draws elements on the first surface and sends a second drawing instruction to the control device; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface.

With reference to the second aspect, in a possible implementation manner, the first drawing instruction further includes: identification information of the first surface;

the second drawing instruction further comprises: identification information of the first surface.

With reference to the second aspect and the foregoing possible implementation manners, in another possible implementation manner, after the controlled device draws an element on the first surface and sends a second drawing instruction to the control device, the method further includes:

the controlled device changes the attribute information of the first surface;

a third drawing instruction sent by the controlled device to the control device; wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, orientation, position, transparency, whether hidden or not displayed.

the controlled device destroys the first surface; a fourth drawing instruction sent by the controlled device to the control device; wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the first surface is destroyed by the control device;

and/or the presence of a gas in the gas,

when a first element of the first surface drawing changes, the controlled device determines whether the change of the first element affects a second element of the first surface drawing; if the change of the first element does not affect a second element drawn on the first surface, the controlled device sends a drawing operation instruction for instructing redrawing of an element same as the first element to the control device; if the change of the first element affects a second element of the first surface drawing, the controlled device sends a drawing operation instruction for instructing redrawing of the same element as the first element and the second element to the control device;

and/or the presence of a gas in the gas,

when the controlled device determines that an animation display process exists on the first surface, the controlled device determines to display only an initial state of the first surface and a final state of the first surface;

and/or the presence of a gas in the gas,

when the controlled device determines that there is an animation display process for the first element of the first surface, the controlled device determines to display only an initial state of the first element and a final state of the first element.

In a third aspect of the present invention, a control device is applied in a remote control system, where the control device is configured to implement remote control on a controlled device in the remote control system, and the controlled device is installed with an android system, and the control device includes: the device comprises a receiving unit, a creating unit and a drawing unit;

the receiving unit is used for receiving a first drawing instruction sent by the controlled equipment; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface;

the creating unit is used for creating a second surface according to the creating information received by the receiving unit;

the receiving unit is further configured to receive a second drawing instruction sent by the controlled device; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface;

the drawing unit is configured to perform a drawing operation on the second surface according to the drawing operation instruction received by the receiving unit, so as to draw, on the second surface, elements that are the same as all the elements drawn on the first surface by the controlled device.

With reference to the third aspect, in a possible implementation manner, the method further includes: the device comprises an establishing unit and a determining unit;

the first drawing instruction further comprises: the establishing unit is configured to establish a mapping relationship between the second surface and the identification information of the first surface before the receiving unit receives the second drawing instruction sent by the controlled device;

the second drawing instruction further comprises: the determination unit is configured to determine, according to the mapping relationship and the identification information of the first surface included in the second drawing instruction, that the drawing operation instruction is used to perform a drawing operation on the second surface before the drawing unit performs a drawing operation on the second surface according to the drawing operation instruction.

With reference to the third aspect and the possible implementations described above, in another possible implementation,

the receiving unit is further configured to receive a third drawing instruction sent by the controlled device after the drawing unit performs a drawing operation on the second surface according to the drawing operation instruction; wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, direction, position, transparency, whether hidden or not displayed; the control apparatus further includes: a modification unit; the changing unit is used for changing the attribute information of the second surface, which is the same as the attribute information of the first surface, according to the changing information received by the receiving unit;

and/or the presence of a gas in the gas,

the receiving unit is further configured to receive a fourth drawing instruction sent by the controlled device after the drawing unit performs a drawing operation on the second surface according to the drawing operation instruction; wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the first surface is destroyed by the control device; the control apparatus further includes: a destroying unit; the destroying unit is used for destroying the second surface according to the destroying information received by the receiving unit.

In a fourth aspect of the present invention, there is provided a controlled device, which is applied to a remote control system, where a control device in the remote control system is used to implement remote control on the controlled device, and the controlled device is installed with an android system, and the controlled device includes: the device comprises a creating unit, a sending unit and a drawing unit;

the creating unit is used for creating a first surface;

the sending unit is used for sending a first drawing instruction to the control equipment; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface;

the drawing unit is used for drawing elements on the first surface;

the sending unit is further used for sending a second drawing instruction to the control equipment; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface.

With reference to the fourth aspect, in a possible implementation manner, the first drawing instruction sent by the sending unit further includes: identification information of the first surface;

the second drawing instruction transmitted by the transmission unit further includes: identification information of the first surface.

With reference to the fourth aspect and the foregoing possible implementation manners, in another possible implementation manner, the controlled device further includes: a modification unit;

the changing unit is configured to change the attribute information of the first surface after the sending unit sends the second drawing instruction to the control device;

the sending unit is further configured to send a third drawing instruction to the control device; wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, orientation, position, transparency, whether hidden or not displayed.

With reference to the fourth aspect and the possible implementations described above, in another possible implementation,

the control apparatus further includes: a destroying unit; the destroying unit is used for destroying the first surface after the sending unit sends a second drawing instruction to the control equipment; the sending unit is further configured to send a fourth drawing instruction to the control device; wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the first surface is destroyed by the control device;

and/or the presence of a gas in the gas,

the control apparatus further includes: a determination unit; the determination unit is configured to determine whether a change in a first element of the first surface drawing affects a second element of the first surface drawing when the first element is changed after the sending unit sends a second drawing instruction to the control device; the sending unit is further configured to send, to the control device, a drawing operation instruction for instructing redrawing of an element that is the same as the first element if the change of the first element does not affect a second element of the first surface drawing; the sending unit is further configured to send, to the control device, a drawing operation instruction for instructing redrawing of an element that is the same as the first element and the second element if the change of the first element affects a second element of the first surface drawing;

and/or the presence of a gas in the gas,

the determination unit is used for determining that only the initial state of the first surface and the final state of the first surface are displayed when the animation display process of the first surface is determined after the sending unit sends the second drawing instruction to the control device;

and/or the presence of a gas in the gas,

the determination unit is configured to determine, when it is determined that there is an animation display process for the first element of the first surface after the transmission unit transmits the second drawing instruction to the control device, to display only an initial state of the first element and a final state of the first element.

According to the control method and the control device for remote display provided by the invention, the control device creates the second surface according to the received creation information included in the first drawing instruction sent by the controlled device, and performs drawing operation on the second surface according to the drawing operation instruction included in the second drawing instruction, so that the same elements as all elements drawn on the first surface by the controlled device are drawn on the second surface. The method has the advantages that the drawing instructions for creating the surface and drawing the elements are only transmitted between the control equipment and the controlled equipment, so that the requirement on bandwidth in the remote control process is reduced, and the problem that the method for realizing screen sharing by transmitting the screenshot of the controlled equipment to the control equipment at each moment due to bandwidth limitation cannot be well applied to the remote control system based on the Android system is solved.

And when a certain element on the first surface is changed, the controlled device sends a drawing operation instruction for redrawing the first element to the control device when determining that the change of the first element does not affect other elements drawn on the first surface, so that the number of the drawing operation instructions is reduced, and the requirement on bandwidth in the remote control process is further reduced. In addition, the requirement on bandwidth is further reduced by reducing the animation display process.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a schematic diagram of a remote control system according to an embodiment of the present invention;

fig. 2 is a flowchart of a method for controlling a remote display according to an embodiment of the present invention;

FIG. 3 is a flow chart of another method for controlling a remote display according to an embodiment of the present invention;

FIG. 4 is a flowchart of another remote display control method according to an embodiment of the present invention;

fig. 5 is a schematic drawing process diagram provided by the android system according to the embodiment of the present invention;

fig. 6 is a schematic diagram illustrating a control apparatus according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of another control apparatus according to an embodiment of the present invention;

fig. 8 is a schematic diagram illustrating a controlled device according to an embodiment of the present invention;

fig. 9 is a schematic diagram of another controlled device according to an embodiment of the present invention;

FIG. 10 is a schematic diagram of another control apparatus according to an embodiment of the present invention;

fig. 11 is a schematic diagram illustrating a composition of another controlled device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, there is shown a schematic diagram of a remote control system for implementing the control method of remote display of the present invention, the remote control system including a control device 01, and at least one controlled device 02. Wherein, install the android system in being controlled equipment 02. The specific implementation principle of the control method for providing remote display provided by the invention under the remote control system is as follows: the image in the controlled device 02 is synchronized to the control device 01 so as to realize the screen sharing between the two, the controlled device 02 first creates a first surface, and sends a first drawing command containing creation information for informing the control device 01 that the controlled device 02 created the first surface to the control device 01, so that the control device 01 can create a second surface according to the creation information contained in the received first drawing command, then the controlled device 02 draws elements on the first surface, and sends a second drawing command containing a drawing operation command used by the controlled device 02 when drawing all the elements on the first surface to the control device 01, so that the control device 01 can continue to draw the same elements on the second surface according to the drawing operation command contained in the received second drawing command, so that the same elements as all the elements drawn on the first surface by the controlled device 02 are drawn on the second surface, thereby realizing screen sharing between the control device 01 and the controlled device 02. By only transmitting drawing instructions for creating surfaces and drawing elements between the control device 01 and the controlled device 02, the requirement for bandwidth in the remote control process is reduced, namely the problem that screen sharing cannot be well applied in a remote control system based on an Android system by the method that the controlled device 02 transmits each moment of screenshot to the control device 01 due to bandwidth limitation is solved.

It should be noted that, in the embodiment of the present invention, the control Device 01 and the controlled Device 02 may interact through a Virtual Device Interface (VDI), but the interaction is not limited to the VDI. That is to say, the control device 01 and the controlled device 02 interact with each other to implement the remote display control method provided by the present invention, which also belongs to the protection scope of the present invention.

It should be noted that, in the embodiment of the present invention, the control device 01 may be a device that can perform playback, such as a mobile phone, a tablet, and a virtual machine. The controlled device 02 may be a device such as a mobile phone, a tablet, a virtual machine, etc. which is equipped with an Android system and can send a drawing instruction to the control device 01. The embodiment of the present invention does not specifically limit the specific forms of the control device 01 and the controlled device 02.

In addition, it should be noted that the connection lines in the remote control system shown in fig. 1 do not represent that the control device 01 and the controlled device 02 are directly connected, but are merely used to indicate that communication can be performed between the control device 01 and the controlled device 02.

Under the system architecture shown in fig. 1, the following embodiments provided by the present invention may be referred to for the specific embodiment process of the control method of remote display provided by the present invention.

An embodiment of the present invention provides a remote display control method, which is applied to a control device in a remote control system, where the control device is configured to implement remote control on a controlled device in the remote control system, and the controlled device is installed with an android system, as shown in fig. 2, the method may include:

101. the control device receives a first drawing instruction sent by the control device.

Wherein, the first drawing instruction may include: creation information for notifying the control device that the first surface is created by the control device.

Specifically, when an image in the controlled device is to be synchronized into the control device so as to implement screen sharing between the two, after the controlled device creates the first surface, a first drawing instruction including creation information for notifying the control device that the controlled device created the first surface may be transmitted to the control device, and at this time, the control device may receive the first drawing instruction transmitted by the controlled device.

102. The control device creates the second surface according to the creation information included in the first drawing instruction.

After the control device receives the first drawing instruction sent by the control device, the second surface can be created according to the creation information included in the first drawing instruction.

103. The control device receives the second drawing instruction sent by the control device.

Wherein the second drawing instruction comprises: and drawing operation instructions used by the controlled equipment when drawing all the elements on the first surface.

Specifically, after the controlled device creates the first surface, the drawing operation may be performed on the first surface, that is, the elements are drawn on the first surface, at this time, the controlled device may collect all drawing operation instructions during drawing all elements by itself, and send the drawing operation instructions used when drawing all elements on the first surface to the control device along with the second drawing instructions, at this time, the control device may receive the second drawing instructions.

104. And the control device performs drawing operation on the second surface according to the drawing operation instruction included in the second drawing instruction so as to draw the same elements on the second surface as all the elements drawn on the first surface by the control device.

After the control device receives the second drawing instruction, the drawing operation can be performed on the second surface according to the drawing operation instruction included in the second drawing instruction, so that the same elements as all the elements drawn on the first surface by the controlled device are drawn on the second surface, and thus the image displayed in the controlled device is displayed in the control device, that is, screen sharing between the control device and the controlled device is realized.

According to the control method of the remote display, the control device creates the second surface according to the received creation information included in the first drawing instruction sent by the controlled device, and performs drawing operation on the second surface according to the drawing operation instruction included in the second drawing instruction, so that the same elements as all elements drawn on the first surface by the controlled device are drawn on the second surface. The method has the advantages that the drawing instructions for creating the surface and drawing the elements are only transmitted between the control equipment and the controlled equipment, so that the requirement on bandwidth in the remote control process is reduced, and the problem that the method for realizing screen sharing by transmitting the screenshot of the controlled equipment to the control equipment at each moment due to bandwidth limitation cannot be well applied to the remote control system based on the Android system is solved.

An embodiment of the present invention provides another remote display control method, which is applied to a controlled device in a remote control system, where the control device in the remote control system is used to implement remote control on the controlled device, and the controlled device is installed with an android system, as shown in fig. 3, the method may include:

201. the controlled device creates a first surface and sends first drawing instructions to the control device.

202. The controlled device draws the element on the first surface and sends a second drawing instruction to the control device.

It should be noted that, the detailed description of step 201 and step 202 in the embodiment of the present invention is similar to the detailed description of the corresponding content in step 101 to step 104 in another embodiment of the present invention, and the detailed description of step 201 and step 202 may refer to the detailed description of the corresponding content in step 101 to step 104 in another embodiment of the present invention, and the embodiment of the present invention is not repeated here.

According to the control method of the remote display, the controlled device creates the first surface, sends the first drawing instruction comprising the creation information to the control device so that the control device creates the second surface, then the controlled device draws the elements on the first surface, and sends the second drawing instruction comprising the drawing operation instruction to the control device so that the control device draws the same elements on the second surface as all the elements drawn on the first surface by the controlled device. The method has the advantages that the drawing instructions for creating the surface and drawing the elements are only transmitted between the control equipment and the controlled equipment, so that the requirement on bandwidth in the remote control process is reduced, and the problem that the method for realizing screen sharing by transmitting the screenshot of the controlled equipment to the control equipment at each moment due to bandwidth limitation cannot be well applied to the remote control system based on the Android system is solved.

An embodiment of the present invention provides another remote display control method, which is applied to a remote control system, where the remote control system includes a control device and a controlled device, the control device is used to implement remote control on the controlled device, and the controlled device is installed with an android system, as shown in fig. 4, the method may include:

301. the controlled device creates a first surface.

302. The controlled device sends a first drawing instruction to the control device.

Wherein, the first drawing instruction comprises: creation information for notifying the control device that the first surface is created by the control device.

As is well known, the device installed with the Android system specifically displays activities (english: activities) as a unit, one display interface may correspond to one Activity, and one Activity corresponds to a Surface of a drawing (Surface), that is, when an interface needs to be displayed, the device firstly creates a Surface which can be drawn in the background, so that elements of the interface can be displayed on the created upper Surface, and, therefore, when the image in the controlled device is to be synchronized into the control device in order to achieve screen sharing between the two, the controlled device needs to notify the controlling device that a Surface has been created, that is, after the first surface is created by the controlled device, a first drawing instruction including creation information for notifying the control device that the first surface is created by the controlled device may be transmitted to the control device.

For example, the controlled device may transmit the first drawing instruction to the control device through the VDI. Specifically, in the SPICE protocol, the controlled device may send a first drawing instruction to the control device through the Display Channel.

It should be noted that, in the embodiment of the present invention, a specific format of the drawing instruction may be pre-negotiated by the controlled device and the control device, and the specific format of the drawing instruction is not specifically limited in this embodiment of the present invention.

In addition, since there is more than one surface created in the controlled device, the first drawing instruction may further include identification information of the first surface in order to enable drawing of the corresponding element on the corresponding surface later.

303. The control device receives a first drawing instruction sent by the control device.

304. The control device creates the second surface according to the creation information included in the first drawing instruction.

305. The control device establishes a mapping relationship of the second surface and the identification information of the first surface.

After the controlled device performs the drawing operation on a certain surface, the control device may perform the drawing operation on the surface corresponding to the certain surface, and after the second surface is created, the control device needs to establish a mapping relationship of the identification information of the first surface corresponding to the second surface.

For example, a table in which a surface created on the control device and identification information of a surface created in the controlled device corresponding to the surface are recorded may be stored in the control device.

306. The element is rendered on the first surface by the control device.

Wherein after the first surface is created by the control device, the various elements can be drawn on the first surface. For example, the controlled device may draw elements such as points, lines, paths, etc. on the first surface.

307. The controlled device sends a second drawing instruction to the control device.

Wherein the second drawing instruction comprises: and drawing operation instructions used by the controlled equipment when drawing all the elements on the first surface. The second drawing instruction may further include: identification information of the first surface.

Specifically, in the process that the controlled device draws elements on the first surface, the controlled device may capture the drawing operations on the bottom layer to obtain drawing operation instructions used when the first surface draws all the elements, and then send the drawing operation instructions and the identification information of the first surface, which are included in the second drawing instructions, to the control device.

308. The control device receives the second drawing instruction sent by the control device.

309. The control device determines a drawing operation instruction for performing a drawing operation on the second surface based on the identification information of the first surface and the mapping relation included in the second drawing instruction.

After the second drawing instruction received by the control device, it may be determined that the drawing operation instructions are used for performing the drawing operation on the second surface according to the identification information of the first surface included in the second drawing instruction and the mapping relationship established in step 305.

310. And the control device performs drawing operation on the second surface according to the drawing operation instruction included in the second drawing instruction so as to draw the same elements on the second surface as all the elements drawn on the first surface by the control device.

By repeatedly performing the above steps 301 to 310, synchronization of a plurality of images can be achieved on the controlled device and the control device.

311. The attribute information of the first surface is modified by the control device.

312. And a third drawing instruction sent by the control device to the control device.

Wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface was changed by the control device, the attribute information including, but not limited to, at least one of: size, depth, orientation (e.g., landscape, portrait), position, transparency, whether hidden or not displayed.

313. The control device receives a third drawing instruction transmitted by the control device.

314. The control device changes the same attribute information of the second surface as the first surface according to the change information included in the third drawing instruction.

For example, it is assumed that the attribute information is depth, that is, after a plurality of images are synchronized between the controlled device and the control device, if the stacking relationship of the plurality of images is changed by changing the depth of the first surface on the controlled device side, the controlled device may notify the controlled device of the change in depth so that the control device may make a corresponding change to achieve the same stacking relationship as the controlled device side.

It should be noted that, in order to enable the control device to know which surface the controlled device performs the operation of changing the attribute information, the third drawing instruction may further include the identification information of the first surface, so that the control device determines, according to the identification information of the first surface and the mapping relationship included in the third drawing operation instruction, that the change information is used for changing the attribute information of the second surface.

315. The first surface is destroyed by the control device.

316. And fourth drawing instructions sent by the control device to the control device.

Wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the first surface was destroyed by the control device.

317. The control device receives a fourth drawing instruction transmitted by the control device.

318. And the control equipment destroys the second surface according to the destruction information included in the fourth drawing instruction.

For example, if a surface is destroyed by the control device, the control device needs to be notified of destruction information for destroying the surface, so that the control device performs a corresponding operation.

It should be noted that, in order to enable the control device to know which surface the controlled device performs the destruction operation on, the fourth drawing instruction may further include identification information of the first surface, so that the control device determines that the destruction information is used for destroying the second surface according to the identification information of the first surface and the mapping relationship included in the fourth drawing operation instruction.

After the above steps 301 to 318 are performed, the whole process of remotely displaying the controlled device by the control device can be completed. Further, after the surface is created and the drawing of all elements on the surface is completed, if a certain element on the surface changes, in order to reduce the number of drawing operation instructions, the controlled device may perform the following process: when a first element of the first surface drawing changes, the control device determines whether the change of the first element affects a second element of the first surface drawing; if the change of the first element does not affect a second element drawn on the first surface, the controlled device sends a drawing operation instruction for instructing the control device to redraw the same element as the first element, so that the control device redraws the same element as the first element on the second surface; if the change of the first element affects a second element drawn by the first surface, the controlled device sends a drawing operation instruction for instructing to redraw the same elements as the first element and the second element to the control device, so that the control device redraws the same elements as the first element and the second element on the second surface.

It should be noted that, in the embodiment of the present invention, the first element refers to any one of all elements included in the first surface, and the second element refers to any one of other elements than the first element among all elements included in the first surface. Of course, if the change of the first element affects at least one second element, the controlled device needs to send a drawing operation instruction to the control device for instructing to redraw the same element as each of the first element and the at least one second element, so that the control device redraws the same element as the first element and the same element as each of the at least one second element on the second surface.

Exemplarily, as shown in fig. 5, a drawing process diagram provided by the android system is provided. In the drawing process, the drawing operation is mainly performed in the drawing (english: Draw) step shown in fig. 5. In the Draw step, various drawing operations are performed to Draw various elements in the surface, which are sources of data generation, i.e., the execution of the Draw step is reduced to save the number of drawing operation instructions. However, these calls are actually generated because a traversal (english: form transactions) step is executed, and the form transactions step is triggered when the system calls back a message, specifically, after receiving an invalidation (english: invoke) operation, the system calls back a message, so that the invocation of the invoke is reduced, that is, when it is determined that a change of the first element affects other elements of the first surface drawing, the invoke is called again, which facilitates the control device to redraw the same elements as the first element and the affected other elements.

It should be noted that, for whether the change of the first element affects the second element drawn on the first surface, the determination criterion is not specifically limited in this embodiment of the present invention, and may be set according to the requirement of the actual application scenario.

In addition, after the above steps 301 to 318 are executed, in order to reduce the number of drawing operation instructions, in the embodiment of the present invention, the controlled device may further execute the following process: when the controlled device determines that the animation display process exists on the first surface, the controlled device determines that only the initial state of the first surface and the final state of the first surface are displayed; alternatively, when the controlled device determines that there is an animation display process for the first element of the first surface, the controlled device determines to display only an initial state of the first element and a final state of the first element. That is, when the controlled device detects that the animation display process exists on the first surface or a certain element of the first surface, only the initial state and the final state in the animation display process are displayed in the controlled device, and accordingly, only the initial state and the final state need to be displayed on the side of the control device, and by reducing the display of the intermediate state in the animation display process, the number of drawing operation instructions is reduced, thereby reducing the bandwidth.

It should be noted that, in the embodiment of the present invention, a protocol used for implementing remote control may be an SPICE protocol, or may be another remote control protocol, and the embodiment of the present invention is not limited specifically herein.

An embodiment of the present invention provides a control device, which is applied to a remote control system, where the control device is configured to implement remote control on a controlled device in the remote control system, and the controlled device is installed with an android system, as shown in fig. 6, the control device may include: a receiving unit 41, a creating unit 42, a rendering unit 43.

The receiving unit 41 is configured to receive a first drawing instruction sent by the controlled device; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface.

The creating unit 42 is configured to create a second surface according to the creation information received by the receiving unit 41.

The receiving unit 41 is further configured to receive a second drawing instruction sent by the controlled device; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface.

The drawing unit 43 is configured to perform a drawing operation on the second surface according to the drawing operation instruction received by the receiving unit 41, so as to draw the same elements as all the elements drawn on the first surface by the controlled device on the second surface.

In this embodiment of the present invention, as shown in fig. 7, the control device may further include: a establishing unit 44 and a determining unit 45.

The first drawing instruction further comprises: the creating unit 44 is configured to create a mapping relationship between the second surface and the identification information of the first surface before the receiving unit 41 receives the second drawing instruction sent by the controlled device.

The second drawing instruction further comprises: the determining unit 45 is configured to determine, according to the mapping relationship and the identification information of the first surface included in the second drawing instruction, that the drawing operation instruction is used to perform a drawing operation on the second surface before the drawing unit 43 performs a drawing operation on the second surface according to the drawing operation instruction.

In this embodiment of the present invention, further, the receiving unit 41 is further configured to receive a third drawing instruction sent by the controlled device after the drawing unit 43 performs a drawing operation on the second surface according to the drawing operation instruction; wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, orientation, position, transparency, whether hidden or not displayed.

The control apparatus may further include: a modification unit 46.

The modifying unit 46 is configured to modify the attribute information of the second surface, which is the same as the attribute information of the first surface, according to the modification information received by the receiving unit 41.

In this embodiment of the present invention, further, the receiving unit 41 is further configured to receive a fourth drawing instruction sent by the controlled device after the drawing unit 43 performs a drawing operation on the second surface according to the drawing operation instruction; wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the controlled device has destroyed the first surface.

The control apparatus may further include: a destruction unit 47.

The destruction unit 47 is configured to destroy the second surface according to the destruction information received by the receiving unit 41.

It should be noted that, for the specific working process of each functional module in the control device provided in the embodiment of the present invention, reference may be made to the specific description of the corresponding process in the method embodiment, and details are not described here in the embodiment of the present invention.

The control device provided by the embodiment of the invention is used for executing the control method of the remote display, so that the same effect as the control method of the remote display can be achieved.

The embodiment of the present invention provides a controlled device, which is applied to a remote control system, where a control device in the remote control system is used to implement remote control on the controlled device, and an android system is installed on the controlled device, as shown in fig. 8, the controlled device may include: a creation unit 51, a transmission unit 52, a drawing unit 53.

The creating unit 51 is configured to create a first surface.

The sending unit 52 is configured to send a first drawing instruction to the control device; wherein the first drawing instruction comprises: creation information for notifying the control device that the controlled device created the first surface.

The drawing unit 53 is configured to draw an element on the first surface.

The sending unit 52 is further configured to send a second drawing instruction to the control device; wherein the second drawing instruction comprises: drawing operation instructions used by the controlled device when drawing all elements on the first surface.

In this embodiment of the present invention, further, the first drawing instruction sent by the sending unit 52 further includes: identification information of the first surface.

The second drawing instruction transmitted by the transmission unit 52 further includes: identification information of the first surface.

In this embodiment of the present invention, as shown in fig. 9, the controlled device may further include: a modification unit 54.

The changing unit 54 is configured to change the attribute information of the first surface after the sending unit 52 sends the second drawing instruction to the control device.

The sending unit 52 is further configured to send a third drawing instruction to the control device; wherein the third rendering instruction comprises: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, orientation, position, transparency, whether hidden or not displayed.

In this embodiment of the present invention, further, the control device may further include: a destruction unit 55.

The destruction unit 55 is configured to destroy the first surface after the sending unit 52 sends the second drawing instruction to the control device;

the sending unit 52 is further configured to send a fourth drawing instruction to the control device; wherein the fourth rendering instruction comprises: destruction information for notifying the control device that the controlled device has destroyed the first surface.

In this embodiment of the present invention, as shown in fig. 9, the controlled device may further include: a determination unit 56.

The determining unit 56 is configured to determine whether a change in the first element affects a second element of the first surface drawing when the first element of the first surface drawing changes after the sending unit 52 sends the second drawing instruction to the control device.

The sending unit 52 is further configured to send, to the control device, a drawing operation instruction for instructing to redraw an element that is the same as the first element if the change of the first element does not affect the second element of the first surface drawing.

The sending unit 52 is further configured to send, to the control device, a drawing operation instruction for instructing to redraw an element that is the same as the first element and the second element if the change of the first element affects a second element of the first surface drawing.

The determination unit 56 is configured to determine that only the initial state of the first surface and the final state of the first surface are displayed when it is determined that the animation display process exists for the first surface after the transmission unit 52 transmits the second drawing instruction to the control device;

or,

the determining unit 56 is configured to determine that only the initial state of the first element and the final state of the first element are displayed when it is determined that there is an animation display process for the first element of the first surface after the sending unit 52 sends the second drawing instruction to the control device.

It should be noted that, for the specific working process of each functional module in the controlled device provided in the embodiment of the present invention, reference may be made to the specific description of the corresponding process in the method embodiment, and details of the embodiment of the present invention are not described herein again.

The controlled device provided by the embodiment of the invention is used for executing the control method of the remote display, so that the same effect as the control method of the remote display can be achieved.

An embodiment of the present invention provides a control apparatus, as shown in fig. 10, including: a processor 61, a memory 62, a display 63 and a communication interface 64.

The memory 62 may be a volatile memory (RAM), such as a random-access memory (RAM); or a non-volatile memory (e.g., a read-only memory (ROM)), a flash memory (e.g., a flash memory), a hard disk (e.g., a Hard Disk Drive (HDD)), or a solid-state drive (e.g., a SSD); or a combination of the above types of memories for storing the relevant applications and configuration files that can implement the control method of the remote display of the present invention.

The processor 61 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement embodiments of the present invention, such as: one or more microprocessors (digital signal processors, DSP for short), or one or more Field Programmable Gate arrays (FPGA for short). The processor 61 may perform various functions of the control device by running or executing software programs and/or modules stored in the memory 62, as well as invoking data stored in the memory 62.

And a display 63 for displaying the image data processed by the processor 61.

The communication interface 64 is mainly used for realizing communication between the devices of the present embodiment.

The processor 61 is configured to execute the method for controlling remote display shown in fig. 2 to 4 of the present invention, so as to implement the functions of the receiving unit, the creating unit, and the drawing unit shown in fig. 6.

Further, the processor 61 is further configured to execute the remote display control method shown in fig. 2 to fig. 4 of the present invention, so as to implement the functions of the establishing unit, the determining unit, the changing unit and the destroying unit shown in fig. 7.

An embodiment of the present invention provides a controlled device, as shown in fig. 11, including: a processor 71, a memory 72, a display 73 and a communication interface 74.

A memory 72, which may be a cache memory, such as RAM; or a non-volatile memory such as a ROM, a flash memory, a HDD or an SSD; or a combination of the above types of memories for storing the relevant applications and configuration files that can implement the control method of the remote display of the present invention.

Processor 71 may be a CPU, an ASIC, or one or more integrated circuits configured to implement embodiments of the present invention, such as: one or more DSPs, or one or more FPGAs. The processor 71 may perform various functions of the controlled device by running or executing software programs and/or modules stored in the memory 72, as well as invoking data stored in the memory 72.

And a display 73 for displaying the image data processed by the processor 71.

The communication interface 74 is mainly used for realizing communication between devices of the present embodiment.

The processor 71 is configured to execute the method for controlling remote display shown in fig. 2 to 4 of the present invention, so as to implement the functions of the creating unit, the sending unit and the drawing unit shown in fig. 8.

Further, the processor 71 is further configured to execute the remote display control method shown in fig. 2 to 4 of the present invention, so as to implement the functions of the changing unit, the destroying unit and the determining unit shown in fig. 9.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solution of the present invention may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be implemented in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (english: processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims

1. A control method of remote display is applied to control equipment in a remote control system, the control equipment is used for realizing remote control of controlled equipment in the remote control system, and the controlled equipment is provided with an android system, the method comprises the following steps:

2. The method of claim 1, wherein the first drawing instruction further comprises: the identification information of the first surface, before the control device receives the second drawing instruction sent by the control device, the method further includes:

3. The method according to claim 1 or 2, wherein after the control device performs a drawing operation on the second surface according to the drawing operation instruction, the method further comprises:

and/or the presence of a gas in the gas,

4. A control method of remote display is applied to controlled equipment in a remote control system, the control equipment in the remote control system is used for realizing remote control on the controlled equipment, and an android system is installed on the controlled equipment, and the method comprises the following steps:

5. The method of claim 4, wherein the first drawing instruction further comprises: identification information of the first surface;

6. The method according to claim 4 or 5, further comprising, after the controlled device draws an element on the first surface and sends a second drawing instruction to the control device:

the controlled device changes the attribute information of the first surface;

7. The method according to claim 4 or 5, further comprising, after the controlled device draws an element on the first surface and sends a second drawing instruction to the control device:

and/or the presence of a gas in the gas,

8. The utility model provides a control device, its characterized in that is applied to in the remote control system, control device is used for realizing the remote control to controlled device in the remote control system, just controlled device installs tall and erect system, control device includes: the device comprises a receiving unit, a creating unit and a drawing unit;

9. The control apparatus according to claim 8, characterized by further comprising: the device comprises an establishing unit and a determining unit;

10. The control apparatus according to claim 8 or 9,

the receiving unit is further configured to receive, after the drawing unit performs a drawing operation on the second surface according to the drawing operation instruction, a third drawing instruction sent by the controlled device, where the third drawing instruction includes: change information for notifying the control device that the attribute information of the first surface is changed by the controlled device, the attribute information including at least one of: size, depth, direction, position, transparency, whether hidden or not displayed; the control apparatus further includes: a modification unit; the changing unit is used for changing the attribute information of the second surface, which is the same as the attribute information of the first surface, according to the changing information received by the receiving unit;

and/or the presence of a gas in the gas,

the receiving unit is further configured to receive, after the drawing unit performs a drawing operation on the second surface according to the drawing operation instruction, a fourth drawing instruction sent by the controlled device, where the fourth drawing instruction includes: destruction information for notifying the control device that the first surface is destroyed by the control device; the control apparatus further includes: a destroying unit; the destroying unit is used for destroying the second surface according to the destroying information received by the receiving unit.

11. The controlled device is applied to a remote control system, wherein a control device in the remote control system is used for realizing remote control of the controlled device, and the controlled device is provided with an android system, and the controlled device comprises: the device comprises a creating unit, a sending unit and a drawing unit;

the creating unit is used for creating a first surface;

the drawing unit is used for drawing elements on the first surface;

12. The controlled apparatus according to claim 11, wherein the first drawing instruction transmitted by the transmission unit further includes: identification information of the first surface;

13. The controlled device according to claim 11 or 12, characterized by further comprising: a modification unit;

14. The controlled apparatus according to claim 11 or 12,

and/or the presence of a gas in the gas,