WO2014029091A1

WO2014029091A1 - Window image transmission method, display method, terminal, server, and system

Info

Publication number: WO2014029091A1
Application number: PCT/CN2012/080475
Authority: WO
Inventors: 秦晋; 狄龙; 谢军安; 张楚雄; 张毅
Original assignee: 华为技术有限公司
Priority date: 2012-08-22
Filing date: 2012-08-22
Publication date: 2014-02-27
Also published as: CN103959246A; CN103959246B

Abstract

A window image transmission method comprises: in a process of operating an application, a virtual machine receiving window information (11) sent by a terminal and corresponding to the application; determining display area information (12) about the window according to the window information; and sending, according to the display area information about the window, image data corresponding to the display area information about the window to the terminal, so that the terminal performs display (13). A display method, a terminal, a server, and a system are further disclosed. In the solution, image data corresponding to display area information about a window is sent according to the display area information about the window displayed by an application in the terminal, and when a window display area is smaller than a whole window area, it is not required to send image data corresponding to the whole window, so as to achieve an effect of reducing the network load.

Description

Window image transmission method, display method, terminal, server and system

The embodiments of the present invention relate to the field of communications technologies, and in particular, to a window image transmission method, a display method, a terminal, a server, and a system. Background of the invention

At present, in the online application system, the terminal and the server generally interact with each other as follows: The terminal uploads a request for running the application generated by the user operation to the server, and the server calculates and processes the request after the application is run, and obtains the window image and sound of the application running. The data is sent to the terminal, so that the terminal decodes and displays the window image.

In order to effectively achieve resource utilization, one server runs multiple virtual machines at the same time, and each virtual machine runs a different application. After the terminal sends a request to the server to run the response, the server selects a specific virtual machine to run the application requested by the terminal, and then the virtual machine sends the window image of the application running to the terminal in real time. For the end user, data for multiple applications is displayed on the same terminal's desktop. However, each application corresponds to a virtual machine, that is, the terminal establishes a connection with multiple virtual machines to obtain images, sounds, and the like that are running on different virtual machines. Therefore, the terminal needs to receive the transmitted data of each virtual machine and decode it on the desktop of the terminal.

For the user, although the terminal displays data of multiple applications, the user may only focus on the window image displayed by one or several applications at the forefront of the desktop. At this point, if the virtual machine sends a window image of multiple application runs to the terminal, it will consume a lot of network bandwidth. Moreover, the terminal needs to decode the window image of multiple application operations, which increases the processing load of the terminal. Summary of the invention

An object of the embodiments of the present invention is to provide a method, a display method, a terminal, a server, and a system for transmitting a window image, and displaying a window when a plurality of windows are occluded.

A first aspect of the present invention provides a method for transmitting a window image, including: in a process of running an application, a receiving terminal sends window information of a window corresponding to the application, and determining a display area of the window according to the window information. And sending, according to the display area of the window, image data corresponding to the display area information of the window to the terminal, so that the terminal performs display.

A second aspect of the present invention provides a server, including: a receiving unit, configured to: when a virtual machine runs an application, receive, by the terminal, window information of a window corresponding to the application;

Determining a unit, determining, according to the window information, display area information of the window;

The transmission unit transmits, by the display area information of the window, image data corresponding to the display area information of the window to the terminal, so that the terminal performs display.

A third aspect of the present invention provides a method for displaying a window, including: in a process of running an application, sending, by a virtual machine, window information of a window corresponding to the application to a server, so that the virtual machine on the server is according to the window Information, determining display area information of the window; receiving image data corresponding to the display area information of the window sent by the server and displaying the image data. A fourth aspect of the present invention provides a terminal, including: a sending unit, configured to send, by a virtual machine, window information of a window corresponding to the application to a server, in a process of running an application, so that the virtual machine on the server is configured according to the virtual machine The window information, determining display area information of the window;

The receiving unit receives and displays the image data corresponding to the display area information of the window sent by the server. A third aspect of the present invention provides a window display system including the above server and the above terminal.

As a first possible implementation of the first aspect,

The window image transmission method, the display method, the terminal, the server, and the system according to the embodiment of the present invention send image data corresponding to the display area information of the window according to the display area information of the window displayed by the terminal, when the window display area is smaller than the entire window In the area, it is not necessary to send the image data corresponding to the entire window area, thereby reducing the network load and improving the terminal operating efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flow chart of a method for transmitting a window image according to an embodiment of the present invention.

FIG. 2 is a schematic diagram 1 of a window in a method for transmitting a window image according to an embodiment of the present invention.

FIG. 3 is a second schematic diagram of a window in a method for transmitting a window image according to an embodiment of the present invention.

FIG. 4 is a third schematic diagram of a window in a method for transmitting a window image according to an embodiment of the present invention.

FIG. 5 is a window intent 4 of a method for transmitting a window image according to an embodiment of the present invention.

FIG. 6 is a first schematic structural diagram of a server according to an embodiment of the present invention.

FIG. 7 is a schematic structural diagram 2 of a server according to an embodiment of the present invention.

FIG. 8 is a schematic flowchart 1 of a method for displaying a window according to an embodiment of the present invention.

FIG. 9 is a schematic flowchart 1 of a method for displaying a window according to an embodiment of the present invention.

FIG. 10 is a first schematic structural diagram of a terminal according to an embodiment of the present invention.

FIG. 11 is a second schematic structural diagram of a terminal according to an embodiment of the present invention.

FIG. 12 is a schematic structural diagram of a display system of a window according to an embodiment of the present invention. Mode for carrying out the invention

As shown in FIG. 1, an embodiment of the present invention provides a method for transmitting a window image, including:

Step 11: In the process of running the application, the receiving terminal sends the window information of the window corresponding to the application.

Step 12: Determine display area information of the window according to the window information;

Step 13: Send image data corresponding to the display area information of the window to the terminal according to the display area information of the window, so that the terminal performs display.

The execution body of the window image transmission method in the embodiment of the present invention may be a server-side virtual machine. Before performing step 11, the terminal sends a request for running the application carrying the application identifier to the server, and the server selects the application that the virtual machine runs the terminal request according to the identifier, and the virtual machine and the terminal establish the transmission application operation data and the receiving terminal sends the information. connection. Therefore, the terminal sends the window information of the application in the terminal display window or the user's operation for the application to the virtual machine running the application through the connection. The window is the window displayed by the application running on the terminal. The method for transmitting a window image according to an embodiment of the present invention sends image data corresponding to the display area information of the window according to the display area information of the window displayed by the terminal. When the window display area is smaller than the entire window area, the entire image does not need to be sent. The image data corresponding to the window area, thereby achieving the effect of reducing the network load.

Specifically, in the foregoing step 11, the window information sent by the terminal includes: a window display state, a coordinate of any corner point of the cover area of the window, and a width and a height of the cover area. Wherein, the coordinates include an X coordinate and a Y coordinate, and the coordinate may adopt a coordinate of a screen coordinate system or a coordinate of a window coordinate system, and the screen coordinate system refers to a point of the upper left corner of the display screen as a coordinate origin (0, 0), and a horizontal direction is an X axis. The vertical direction is the Y axis, and the window coordinate system refers to the coordinate point origin (0, 0) of the upper left corner of the window, the X axis for the horizontal direction, and the Y axis for the vertical direction; the cover area of the window may be one or more. Optionally, the window information may further include: a number of cloaking windows.

The message type of the terminal sending window information is Updata_Window_infrom, and the message format can be as shown in Table 1 below, so that the virtual machine parses the message to obtain the window information sent by the terminal - Table 1

In Table 1, when the window display status is full display or full coverage, the parameters of the number of cover windows, X coordinate, Y coordinate, width and height are empty. It should be noted that in the present application, the width refers to the length of the region on the X axis, and the height is the length of the region on the Y axis. In addition, in Table 1, the X coordinate and the Y coordinate may be the coordinates of any corner point of the first cover area, and Table 1 takes the value of the point of the upper left corner, and may also take the coordinates of other corner points. The embodiments of the present invention are not limited herein. The corresponding parameters in other tables refer to the description of this paragraph.

In Table 1, the corner of the covering area may be the upper left corner point, the upper right corner point, the lower left corner point or the lower right corner point of the covering area. Preferably, based on the adopted coordinate system, such as the screen coordinate system (the upper left corner of the display screen is the coordinate origin (0, 0)) or the window coordinate system (the upper left corner of the window is the coordinate origin (0, 0) ), then the upper left corner is preferred.

Alternatively, in the above step 11, the window information sent by the terminal includes: a window display state, a coordinate of any corner point of the window, a width and a height of the window, and a coordinate of the z-axis sequence larger than the corner of the window and covering the window. And the Z-axis order is larger than the window And cover the window width and height of the window. Wherein, the coordinates include an X coordinate and a Y coordinate, and the coordinate may adopt a coordinate of a screen coordinate system or a coordinate of a window coordinate system; a window whose z-axis sequence is larger than the window and covers the window may be one or more. Optionally, the window information may further include: a number of cloaking windows.

The message type of the terminal to send the window information is Updata_Window_infrom, and the message format can be as shown in Table 2 below, so that the virtual machine parses the message to obtain the window information sent by the terminal:

Table 2

In Table 2, when the window display state is full display or full coverage, the number of cover windows, X coordinate, Y coordinate, width and height are empty.

In Table 2, the X coordinate, the Y coordinate, the width and height of the window passing through any corner of the window, and the X coordinate, Y coordinate, and z-axis sequence of any corner of the window whose z-axis order is larger than the window and covers the window. The width and height of the window larger than the window and covering the window can determine the X coordinate, the Y coordinate, the width and the height of the cover area of any corner of the cover area of the window as shown in Table 1.

In Table 2, the corner point can be the upper left corner point, the upper right corner point, the lower left corner point, or the lower right corner point. Preferably, the upper left corner is preferred. A brief description of the Z-axis sequence, Z-axis sequence: When using the operating system, multiple windows are opened, resulting in a heap of overlapping windows. To indicate the position of these stacked windows, each window is labeled using the z-axis sequence, and each window has a unique position in the z-axis sequence. This window stack is along a virtual axis - the "z-axis", extending vertically from the screen. The window at the top of the z-axis sequence covers other windows in the Z-axis sequence. The window at the bottom of the Z-axis sequence is covered by other windows in the Z-axis sequence.

For example, the Windows system puts the window just created on top of the Z-axis sequence, and the user can change the Z-axis sequence by activating another window; the Windows system always places the active window on top of the Z-axis sequence, the application can use functions.

BringWindowToTopO places a window on top of the Z-axis sequence.

Usually, a single linked list of the system maintains the Z-axis sequence. The user changes the Z-axis sequence during the activation of different windows. For example, by using the GetNextWindow() parameter, you can get the handle of the previous or next window in the Z-axis sequence.

It can be seen that the operating system of the terminal can obtain the Z-axis order and window coordinate information of each window, including the coordinates of the top left corner and the width and height of each window.

Alternatively, in the above step 11, the window information sent by the terminal includes: coordinates of any corner point of the window, width and height of the window, a Z-axis order is greater than coordinates of any corner of the window of the window, and a Z-axis sequence is larger than the window. Window width and height. The coordinate includes the X coordinate and the Y coordinate, and the coordinate may adopt the coordinate of the screen coordinate system or the coordinate of the window coordinate system; the window whose Z axis sequence is larger than the window may be more than one. Optionally, the window information may further include: a number of cloaking windows.

The message type of the terminal sending window information is Updata_Window_infrom, and the message format can be as shown in Table 3 below, so that the virtual machine parses the message to obtain the window information sent by the terminal:

table 3

Parameter Type Description Message type string Indicates the message type, set to window display status update message,

Can be defined as Updata_Windwo_infrom_

X coordinate int X coordinate of any corner of the window

Y coordinate int Y coordinate of any corner of the window Width width int Width of the window Height Height int Height of the window Number of occlusion windows int Identifies how many windows have a z-order greater than this window

X int The first z-axis is greater than the X coordinate of any corner of the window of the window

Y int The first z-axis order is greater than the Y coordinate of any corner of the window of the window. width int The first z-axis order is greater than the width of the window of the window. Height int The first z-axis order is greater than the width of the window's window int second, third... The z-axis order is greater than the window's (X, Y, width, Height) in Table 3, through the window The X coordinate of the corner point, the Y coordinate, the width and height of the window, and the X coordinate, the Y coordinate, and the z axis sequence of any corner of the window whose z-axis order is larger than the window are larger than the width and height of the window of the window, and can be determined. The X coordinate, the Y coordinate, and the z-axis sequence of any corner point of the window as shown in Table 2 that is larger than the window and covering the window are larger than the window and the width and height of the window covering the window, thereby further determining The X coordinate, the Y coordinate, the width and height of the cover area of any corner point of the cover area of the window shown in Table 1.

In Table 3, the corner point can be the upper left corner point, the upper right corner point, the lower left corner point, or the lower right corner point. Preferably, the upper left corner is preferred. The terminal can send window information in the above three types of message formats, and the terminal and the virtual machine can negotiate in advance which format to use to send window information, so that the virtual machine can parse out the window information.

Optionally, in the foregoing step 11, the size of the window is in units of pixels. For example, if the window width is 640 and the height is 480, the width of the window is 640 pixels and the height is 480 pixels. When the screen resolution of the server and the terminal is the same, the two windows of the corresponding application may also be different. This difference can be divided into three categories:

(1) The window size is different: If the server's window size is 640*480, the size of the terminal's window is dragged by the user to 480*320.

(2) The position of the window is different: For example, the coordinates of the upper left corner of the window of the server are (50, 50), but the coordinates of the upper left corner of the window of the terminal are (40, 70).

(3) The window size and position are different: the combination of the above two cases.

The terminal window size and the window size of the virtual machine can be obtained through the API (Application Programming Interface) provided by the operating system, and will not be described here.

Therefore, the coordinates of the terminal and the sitting needs of the virtual machine can be converted to each other: For example, the window size of the virtual machine is X1*Y1, the size of the terminal window is Χ2*Υ2, and the coordinates of the upper left corner of the terminal window are (xl, yl), the virtual machine The coordinates of the upper left corner of the window are (x2, y2). For a window with coordinates (al, bl), width cl, and height dl at the terminal, the coordinates of the virtual machine become a2= (al-xl). +x2) *X2/X1; b2= (bl-yl+y2) *Y2/Yl _; width is c2=cl*X2/Xl; high is d2=dl*Y2/Yl o For example, for the upper left of the terminal window The angular coordinates are (100, 100), the window width is 480, the height is 320, the coordinates of the upper left corner of the window of the virtual machine are (50, 60), the width of the window is 800, and the height is 600. The coordinates of the upper left corner of any window on the terminal side. Such as (200, 200)) and width such as (150) and height as (80), ie xl=100, yl=100, XI=480, Yl=320, x2=50, y2=60, Χ2=800, Υ2 =600, al=200, bl=200, cl=150, dl=80, then according to a2=(al- xl+x2) *X2/X1; b2= (bl-yl+y2) *Y2/Yl _; c2 =cl*X2/Xl; d2=dl*Y2/Yl, the coordinates of this window converted to the virtual machine are: (250, 300), width is 2 50, height is 150;

A person skilled in the art can understand the coordinate conversion with reference to the prior art. The coordinate transformation can be performed on the virtual machine side or at the terminal. If the coordinate transformation is performed at the terminal, the terminal sends a window position request message to the virtual machine to obtain a virtual Machine window size and location information. If the coordinate transformation is performed on the virtual machine side, the window size and location information of the terminal may be included in the message (such as Table 1 - Table 3 format), or the terminal sends a separate window notification message carrying the window size and location information of the terminal to the virtual Machine, in order to facilitate the coordinate conversion of the virtual machine, not to repeat here. Optionally, in the foregoing step 11, the window information of the window sent by the terminal may be represented by a coordinate of a screen coordinate system (the screen coordinate system refers to a coordinate of the upper left corner of the display screen as a coordinate origin (0, 0)), or It is represented by the coordinates of the window coordinate system (the window coordinate system refers to the point in the upper left corner of the window as the coordinate origin (0, 0)). Preferably, the window information in the format of the table 1 can be represented by the coordinates in the window coordinate system. Since the virtual machine does not need to calculate the cover area according to the window information, the virtual machine can quickly obtain the cover area according to the coordinates in the window coordinate system. The window information in the format of Table 2 and Table 3 can be represented by the coordinates in the screen coordinate system.

Optionally, in the method for transmitting a window image according to the embodiment of the present invention, when the terminal establishes a window, or the Z-axis sequence of the window changes, or the coordinates of any corner of the window change, or the width or height of the window changes, the terminal The window information of the window needs to be updated to the corresponding virtual machine, so that the corresponding virtual machine receives the window information of the updated window sent by the terminal.

Specifically, in the foregoing step 12, determining the display area information of the window according to the window information of the window, including: mode 1, when the window information is in the format of the table 1:

When the window display state in the window information is full display, it is determined that the display area of the window is the entire window.

Or, when the window display status in the window information is full, the display area of the window is determined to be 0.

Or, according to the window display state in the window information is partially covered, according to the X coordinate, the Y coordinate, the width and the height of the cover field of any corner of the cover area in the window information, the display area of the window is determined to be the entire window except the cover area. The area, when there are a plurality of cover areas, determine a joint cover area composed of a plurality of cover areas, thereby determining a display area of the window as a field outside the joint cover area of the entire window.

For example, as shown in FIG. 2, there are four applications in the terminal corresponding to the window A, the window B, the window C, and the window D. For example, the coordinates of the upper left corner of the window A (Χ, Υ) are (50, 50), and the width is 150, the height is 100; the upper left corner of the window ( (X, Υ) coordinates are (70, 80), the width is 60, the height is 120; the upper left corner of the window C (Χ, Υ) coordinates are (100, 110), width is 140, height is 100; as the upper left corner of window D (Χ, Υ) coordinates are (210, 160), width is 20, and height is 20.

For example, the axis sequence of window C> the axis sequence of window D> the window Β axis order> window Α Ζ axis order.

As shown in FIG. 3, for the window A, the window display state of the window A is partially covered, and the (X, Y) coordinate of the upper left corner of the first cover field of the window A is (70, 80), the first cover The area is 60 wide and 70 high. The second cover area of the window Α The (X,Y) coordinate of the upper left corner is (100, 110), the second cover area is 100 wide, and the height is 40. According to the first covering area and the second covering area, a joint covering area composed of the first covering area and the second covering area is determined, wherein the coordinates are (100, 110), the width is 30, and the height is 40 area is an overlapping area.

Further, according to the joint cover area, the display area of the window 确定 is determined as the area of the entire window A except the joint cover area. As shown in FIG. 4, for the P window B, the window display state of the window B is partially covered, and the (Χ, Υ) coordinate of the upper left corner of the cover area of the window B is (100, 110), and the cover area width is 30, the height is 40, thereby determining the display area of the window 为 to remove the area outside the occlusion area for the entire window.

For window C, the window display state of window C is full display, and the display area of the window is determined to be the entire window C. For window D, the window display state of the window D is completely covered, and the display area of the window is determined to be 0.

Mode 2, when the window information is in Table 2 format:

Or, according to the window display state in the window information is completely covered, it is determined that the display area of the window is 0. Or, according to the window display state in the window information is partially covered, according to the X coordinate, the Υ coordinate, the width and the height of any corner of the window in the window information, and any corner of the window whose axis sequence is larger than the window and covers the window The X coordinate, Υ coordinate, width and height of the point, determine the X coordinate, Υ coordinate, width and height of the occlusion area of any corner of the occlusion field;

And determining, according to the X coordinate, the Υ coordinate, the width and the height of the occlusion area of the occlusion area, the display area of the window is an area other than the occlusion area of the entire window, and when the occlusion area is plural, determining a plurality of occlusion areas The joint cover area, thereby determining the display area of the window as the area of the entire window except the joint cover area.

For example, as shown in FIG. 2, there are four applications in the terminal corresponding to the window Α, the window Β, the window C, and the window D. For example, the coordinates (X, Y) of the upper left corner of the window 为 are (50, 50), The width is 150 and the height is 100; the upper left corner of the window Χ (Χ, Υ) has coordinates (70, 80), width 60, and height 120; window C's upper left corner (X, Υ) coordinates are (100) , 110), width is 140, height is 100; as in the upper left corner of window D (Χ, Υ) coordinates are (210, 160), width is 20, height is 20.

As shown in FIG. 3, for the thousand window A, the window display state of the window A is partially covered, and the coordinates of the (左, Υ) point according to the upper left corner of the window A are (50, 50), and the width is 150, high. For the 100, the (X,Y) coordinates of the upper left corner of the window 大于 that is larger than the window 遮 and cover the window 为 are (70, 80), the width is 60, the height is 120, and the ζ axis order is larger than the window Α and The (X,Y) coordinates of the upper left corner of the window C of the cover window are (100, 110), the width is 140, and the height is 100. The (Χ, Υ) coordinates of the upper left corner of the first cover area of the window 确定 are determined. For (70, 80), the first cover area is 60 wide and 70 high. The ( Χ, Υ) coordinates of the upper left corner of the second occlusion area of the window ( are (100, 110), and the second cover area is 100 wide and 40 high.

Further, according to the first covering area and the second covering area, determining a joint covering area composed of the first covering area and the first covering area, wherein the coordinates are (100, 110), the width is 30, and the height is 40 area is an overlapping area. The display area of the window 确定 is determined to be the area of the entire window A except the joint occlusion field.

As shown in FIG. 4, for the thousand window B, the window display state of the window B is partially covered. According to the upper left corner of the window B, the coordinates (X, Y) are (70, 80), the width is 60, and the height is 120, and the (Χ, Υ) coordinate of the upper left corner of the window C whose axis sequence is larger than the window 遮 and covers the window 为 is (100, 110), the width is 140, and the height is 100, and the upper left corner of the cover area of the window 确定 is determined. The coordinates of the point (Χ, Υ) are (100, 110), the width of the occlusion area is 30, and the height is 40, so that the display area of the window Β is determined to be the area outside the occlusion area of the entire window.

Mode 3, when the window information is in Table 3 format:

According to the X coordinate, the Y coordinate, the width and the height of any corner point of the window in the window information, and the X coordinate, the Y coordinate, the width and the height of any corner point of the window whose z-axis order is larger than the window, it is determined that the z-axis order is larger than the Whether the window of the window covers the window, if the window whose z-axis order is larger than the window does not cover the window, the window display state of the window is full display, and the display area of the window is determined to be the entire window;

If the z-axis fires over the window of the window, the X coordinate, the Y coordinate, the width and the height of any corner of the window in the window information, and the window whose z-axis order is larger than the window and covers the window The X coordinate, Y coordinate, width and height of a corner point, determine the X coordinate, Y coordinate, width and height of the cover area of any corner point of the cover area; According to the X coordinate, the Y coordinate, the width and the height of the corner of the cover area, it is determined whether the cover area completely covers the window. When there are multiple cover areas, it is determined whether the joint cover area composed of the plurality of cover areas is completely covered. The window; if the occlusion area or the joint occlusion area completely covers the window, the window display state of the window is full occlusion, thereby determining that the display field of the window is 0, and if the occlusion area or the joint occlusion area does not completely cover the window, The window display state of the window is partially covered, and the display area of the window is determined to be an area of the entire window except the cover area, or the display area of the window is determined to be an area of the entire window except the joint cover area.

For example, as shown in FIG. 2, there are four applications in the terminal corresponding to the window A, the window B, the window C, and the window D. For example, the coordinates (X, Y) of the upper left corner of the window A are (50, 50), and the width is wide. 150, the height is 100; the upper left corner of the window ( (X, Υ) coordinates are (70, 80), the width is 60, the height is 120; the upper left corner of the window C (Χ, Υ) coordinates are (100, 110), width is 140, height is 100; as in the upper left corner of window D (Χ, Υ) coordinates are (210, 160), width is 20, height is 20.

As shown in FIG. 3, for window A, according to the window information of window A, the (X, Y) coordinate of the upper left corner of window A is (50, 50), the width is 150, the height is 100, and the axis sequence is greater than The (Χ, Υ) coordinates of the upper left corner of the window Β window are (70, 80), the width is 60, and the height is 120. The coordinates of the upper left corner of the window 确定 are determined to be within the window area of the window Α. Further, according to the upper left corner point ((, Υ) of the window A, the coordinates (50, 50), the width 150, and the height of 100 degrees are calculated (X, Y) of the upper right corner of the window 为 is (200, 50), and Calculate the coordinates of the lower left corner point (50, 150), determine that the X-axis coordinate 70 of the upper left corner of the window Β is greater than the X-axis coordinate 50 of the upper left corner of the window ,, less than the X-axis coordinate 200 of the upper right corner of the window A, and It is determined that the Y-axis coordinate 80 of the upper left corner of the window B is smaller than the Y-axis coordinate 150 of the lower left corner of the window A, and is greater than the Y-axis coordinate 50 of the upper left corner of the window A, thereby determining that the upper left corner of the window B is located in the window A. Within the window area. Further, the z-axis order of the window B is determined to be larger than the window A and the window A is covered.

According to the window information of the window A, the (X, Y) coordinates of the upper left corner of the window A are (50, 50), the width is 150, the height is 100, and the top left corner of the window C whose axis sequence is larger than the window Α is The (Χ, Υ) coordinates are (100, 110), the width is 140, and the height is 100. It is determined that the coordinates of the upper left corner of the window C are located in the window area of the window Α. Further, according to the upper left corner point (X, Y) of the window A, the coordinates (50, 50), the width 150, and the height of 100 degrees are calculated (X, Y) of the upper right corner of the window 为 (200, 50), and Calculating the coordinates of the lower left corner point (50, 150), determining that the X-axis coordinate 100 of the upper left corner of the window C is greater than the X-axis coordinate 50 of the upper left corner of the window ,, less than the X-axis coordinate 200 of the upper right corner of the window A, and It is determined that the Y-axis coordinate 110 of the upper left corner of the window C is smaller than the Y-axis coordinate 150 of the lower left corner of the window A, and is greater than the Y-axis coordinate 50 of the upper left corner of the window A, thereby determining that the upper left corner of the window C is located in the window A. Within the window area. Further, the z-axis order of the window C is larger than the window A and the window A is covered.

According to the window information of window A, the (X, Y) coordinates of the upper left corner of window A are (50, 50), and the window! The coordinates of the upper left corner ( Χ, Υ) are (210, 160), the width is 20, and the height is 20. The coordinates of the upper right corner of the calculation window Χ (Χ, Υ) are (200, 50), and the lower left corner is calculated. Point coordinates (50, 150), determining that the X-axis coordinate 210 of the upper left corner of the window D is greater than the X-axis coordinate 200 of the upper right corner of the window ,, and the Y-axis coordinate 160 of the upper left corner of the window D is greater than the upper right corner of the window A The Y-axis coordinate of the point is 150, thereby determining that the upper left corner of the window D is outside the window area of the window A. Further determining that window D does not obscure window A.

In addition, it is assumed that the X-axis coordinate of the upper left corner of the window B is larger than the X-axis coordinate of the upper right corner of the window A, or the Y-axis coordinate of the upper left corner of the window B is larger than the Y-axis coordinate of the lower left corner of the window A, that is, The upper left corner of window B is not located in the window area of window A, then it is determined that window A is not covered by window B; it is assumed that the X-axis coordinate of the upper left corner of window C is larger than the window. The X-axis coordinate of the upper right corner of A, or the Y-axis coordinate of the upper left corner of window C is greater than the Y-axis coordinate of the lower left corner of window A, that is, the upper left corner of window C is not located in the window area of window A, then It is determined that the window A is not covered by the window C; therefore, it is determined that the window A is not covered by the window B, the window (;, the window D, and the window display state of the window A is the full display, and the display area of the window A is determined to be the entire window.

Further, according to the width 60 and the height 120 of the window B, and the coordinates of the upper left corner of the window A and the width and height of the window A, the covered area (i.e., the first covered area) in which the window A is covered by the window B is obtained. Specifically, according to the coordinates of the upper left corner of the window A and the width and height of the window A, the coordinates (X, Y) of the upper right corner of the window A are (200, 50) and the coordinates of the lower left corner (50, 150), and According to the coordinates of the upper left corner of the window B and the width and height of the window B, the coordinates (X, Y) of the upper right corner of the window B are calculated as (130, 80) and the coordinates of the lower left corner (70, 200), and the window B is determined. The upper right corner point is located in the area of the window A, and the lower left corner point of the window B is outside the area of the window A, then the width of the first covering area is determined to be the width 60 of the window B, and the height of the first covering area is the window A The Y-axis coordinate 150 of the lower left corner point minus the Y-axis coordinate 80 of the upper left corner of the window B is 70. Therefore, the (Χ, Υ) coordinate of the upper left corner of the first cover area of the window A is (70, 80), and the first cover area is 60 wide and 70 high. Further, it is assumed that the coordinates of the upper right corner of the window Β are located outside the area of the window ,, that is, the X-axis coordinate of the upper right corner of the window B is greater than the X-axis coordinate of the window A, and the width of the first covering area is the window A. The X-axis coordinate of the upper right corner point is subtracted from the coordinates of the upper left corner of the window B. Further, assuming that the lower left corner of the window B is located in the area of the window A, the height of the first covered area is the height of the window B.

The masked area (i.e., the second covered area) in which the window A is covered by the window C is obtained according to the width 140 of the window C and the height 100, and the coordinates of the upper left corner of the window A and the width and height of the window A. Specifically, according to the coordinates of the upper left corner of the window A and the width and height of the window A, the coordinates (X, Y) of the upper right corner of the window A are (200, 50) and the coordinates of the lower left corner (50, 150), and According to the coordinates of the upper left corner of the window C and the width and height of the window C, the coordinates (Χ, Υ) of the upper right corner of the window C are calculated as (240, 100) and the coordinates of the lower left corner (100, 210), and the window C is determined. The upper right corner point is outside the area of the window 以及 and the lower left corner of the window C is outside the area of the window D, then the width of the second covered area is determined as the X-axis coordinate of the upper right corner of the window A 200 minus the window The X-axis coordinate 100 of the upper left corner of C is 100, and the height of the second masking area is the Y-axis coordinate 150 of the lower left corner of the window A minus the Y-axis coordinate 110 of the upper left corner of the window C is 40. Therefore, the (X,Y) coordinate of the upper left corner of the second cover area of the window A is (100, 110), and the second cover area is 100 wide and 40 high. Further, assuming that the coordinate position of the upper right corner of the window C is determined to be within the area of the window ,, the width of the second covered area is the width of the window C. Further, assuming that the lower left corner of the window C is located in the area of the window A, the height of the second covered area is the height of the window C.

Further, according to the coordinates (70, 80) of the upper left corner of the joint cover area formed by the first cover area and the second cover area in the area of the window A, it is determined that the joint cover area does not completely cover the window A, and the display of the window A is determined. The area is the entire area of the window A except the joint mask area. Assume that the X-axis coordinate of the upper left corner of the joint cover area is less than or equal to the X-axis coordinate of the upper left corner of the window A, and the Y-axis coordinate of the upper left corner of the joint cover area is less than or equal to the Y-axis coordinate of the upper left corner of the window A, and the joint cover If the width of the area is greater than or equal to the width of the window A, and the height of the joint cover area is greater than or equal to the height of the window A, it is determined that the joint cover area completely covers the window A, and the display area of the window A is determined to be 0.

As shown in Fig. 4, for the window ', the coordinates of the upper left corner (Χ, Υ) according to the window 为 are (70, 80), the width is 60, the height is 120, and the window whose axis sequence is larger than the window Β The (左, Υ) coordinates of the upper left corner of C are (100, 110), the width is 140, and the height is 100. The coordinates of the upper left corner of the fixed window C are located in the window area of the window B. Further, according to the upper left corner point (X, Y) of the window B, the coordinates are (70, 80), the width is 60, and the height is 120. The coordinates of the upper right corner of the calculation window Χ (Χ, Υ) are (130, 80). And calculating the coordinates of the lower left corner point (70, 200), determining that the X-axis coordinate 100 of the upper left corner of the window C is greater than the X-axis coordinate 70 of the upper left corner of the window ,, and the X-axis coordinate 130 of the upper right corner of the window Β is smaller than 130 And determining that the pivot coordinate 110 of the upper left corner of the window C is smaller than the pivot coordinate 200 of the lower left corner of the window ,, which is greater than the pivot coordinate 80 of the upper left corner of the window ,, thereby determining that the upper left corner of the window C is located in the window Inside the window area. It is further determined that the axis sequence of the window C is larger than the window 遮 and covers the window Β.

According to the upper left corner of the window (Χ, Υ) coordinates are (70, 80), the width is 60, the height is 120, and the upper left corner of the window D (X, Υ) coordinates are (210, 160), the width is 20, the height is 20, the coordinates of the upper right corner of the calculation window Χ (Χ, Υ) are (130, 80), and the coordinates of the lower left corner point (70, 200) are calculated, and the X-axis coordinate 210 of the upper left corner of the window D is determined. It is greater than the X-axis coordinate 130 of the upper right corner of the window ,, thereby determining that the upper left corner of the window D is outside the window area of the window Β. Further confirmation window D does not cover the window Β.

Suppose that the X-axis coordinate of the upper left corner of the window C is larger than the X-axis coordinate of the upper right corner of the window ,, or the Υ-axis coordinate of the lower left corner of the window of the upper left corner of the window C, that is, the window The upper left corner of C is not located in the window area of the window ,, then it is determined that the window Β is not covered by the window (:, window D, the window 显示 window display state is full display, and the window Β display area is determined to be the entire window.

Further, the window Β is covered by the window C according to the width 140 and the height 100 of the window C, and the coordinates of the upper left corner of the window 以及 and the width and height of the window Β. Specifically, according to the coordinates of the upper left corner of the window 以及 and the width and height of the window 获取, the coordinates (Χ, Υ) of the upper right corner of the window 为 are (130, 80) and the coordinates of the lower left corner (70, 200), and According to the coordinates of the upper left corner of the window C and the width and height of the window C, the coordinates (X, Υ) of the upper right corner of the window C are calculated as (240, 100) and the coordinates of the lower left corner (100, 110), and the window C is determined. The upper right corner point is outside the area of the window ,, and the lower left corner of the window C is outside the area of the window ,, then the width of the occlusion area is determined as the X-axis coordinate 130 of the upper right corner of the window 减 minus the upper left corner of the window C The X-axis coordinate 100 is 30, and the height of the cover area is the Υ axis coordinate 200 of the window Β lower left corner point minus the Υ axis coordinate 110 of the upper left corner of the window C is 90. Therefore, the (Χ, Υ) coordinates of the upper left corner of the window Β cover area are (100, 110), the cover area is 30 wide, and the height is 90. Further, assuming that the coordinates of the upper right corner of the window C are determined to be within the area of the window ,, the width of the occlusion area is the width of the window C. Further, assuming that the lower left corner of the window C is located in the area of the window, the height of the cover area is the height of the window C.

Further, according to the coordinates (100, 110) of the upper left corner of the cover area in the area of the window ,, it is determined that the cover area does not completely cover the window Β, and the display area of the window 确定 is determined to be the entire area of the window except the cover area. Assume that the X-axis coordinate of the upper left corner of the cover area is smaller than the X-axis coordinate of the upper left corner of the window, and the coordinate of the upper left corner of the cover area is less than or equal to the pivot coordinate of the upper left corner of the window, and the width of the cover area is large. If it is equal to the width of the window ,, the height of the occlusion area is greater than or equal to the height of the window Β, then it is determined that the occlusion area completely covers the window Β, and the display area of the window Β is determined to be 0.

For window C, the Z-axis order of window C> the Z-axis order of window D> the Z-axis order of window B> the Z-axis order of window A, the window information of window C has no window of z-order in window C, window If the window display state of C is full display, it is determined that the display area of window C is the entire window (;.

As shown in FIG. 5, for the window D, according to the window information of the window D, the coordinates (X, Y) of the upper left corner of the window D are (210, 160), The width is 20 and the height is 20. The coordinates (X, Y) of the upper right corner of the calculation window D are (230, 160), and the coordinates of the lower left corner of the calculation window D (210, 180) are larger than the window D according to the z-axis order. The (X,Y) coordinates of the upper left corner of window C are (100, 110), the width is 140, and the height is 100. The coordinates of the upper right corner of the calculation window C (Χ, Υ) are (240, 100), and Calculate the coordinates of the lower left corner of the window C (100, 210), and determine that the X-axis coordinate 210 of the upper left corner of the window D is greater than the X-axis coordinate 100 of the upper left corner of the window C, and is smaller than the X-axis coordinate of the upper right corner of the window C. 240, and determining that the pivot coordinate 160 of the upper left corner of the window D is smaller than the pivot coordinate 210 of the lower left corner of the window C, is greater than the pivot coordinate 110 of the upper left corner of the window C, thereby determining that the window C covers the window D.

Further, according to the coordinates of the upper right corner of D (Χ, Υ) is (210, 160), the coordinates of the lower left corner of window D (210, 180), and the coordinates of the upper right corner of window C (Χ, Υ) are (240, 100). ), and the coordinates of the lower left corner of the window C (100, 210), the coordinates of the (X, Υ) of the upper left corner of the occlusion area of the window D are determined to be (210, 160), and the coordinates of the upper right corner of the window D are located at the window C. In the area, the width of the cover area is the width 20 of the window D, and the lower left corner of the window C is located in the area of the window C, and the height of the cover area is the height 20 of the window D.

Further, according to the (Χ, Υ) coordinate of the upper left corner of the cover area of the window D is (210, 160), the width of the cover area is the width 20 of the window D, and the height of the cover area is the height 20 of the window D, determining the cover The area completely covers the window D, and the display area of the window D is determined to be 0.

Specifically, the step of: 3, according to the display area information of the window, the image data corresponding to the display area information of the window is sent to the terminal, so that the terminal performs the display, including:

When the display field of the window is the entire window, the image data of the entire window is acquired and sent to the terminal for display by the terminal. Alternatively, when the display area of the window is an area other than the cover area of the entire window, image data of the entire window is acquired, and the cover area is filled with a solid color or a colorless, and the filled image data is sent to the terminal, and the terminal displays.

The image data of the entire window is the image data of the application running corresponding to the window; the solid color is black or white. It can be seen that for the existing image coding method, the encoded data size of the image is complicated (the color type is large, the color distribution is uneven) is larger than the image is simple (the color type is single, the color distribution is uniform), so the image coding of the solid color is The amount of data is minimal. For the video stream, that is, encoding a plurality of consecutive images, the encoding of the two images before and after, only need to encode the changed portion of the second image, so the image to be covered is filled with a solid color and encoded, because the image is not It will change, and no coding will be used to reduce the amount of data in the video stream. Moreover, when the display area of the window is 0, it is not necessary to acquire image data, and the amount of data of the video stream is reduced.

Optionally, the method for transmitting a window image in the embodiment of the present invention may further include:

Receiving a display image transmission policy sent by the terminal, where the display image transmission strategy includes: lowering the image of the window or not transmitting the image according to a Z-axis sequence of the window that is smaller than a Z-axis sequence of other windows of the terminal; An image of the window, or according to the ratio of the cover area of the window to the display area being greater than a certain threshold, determining to degrade the image of the window or not to transmit the image of the window;

According to the display image transmission strategy, the image of the window is reduced or not transmitted correspondingly.

Wherein, the image of the window is degraded, for example, the HD image is degraded to a standard definition image, etc., to reduce the network load. Optionally, if the terminal determines that the image transmission policy is displayed as not transmitting the image of the window, the terminal may directly set the window display state of the window to be completely covered by the window.

It can be seen that although the corresponding window of multiple applications is started, the user may only focus on the front end of the desktop (the Z-axis order is the largest). The window corresponding to one or two applications does not focus on the window with a small z-axis order. Therefore, it is possible to reduce the transmission of a window with a small Z-axis sequence or a window with a small z-axis sequence, thereby reducing the network load. Improve terminal operating efficiency and reduce server load.

The method for transmitting a window image according to an embodiment of the present invention is applicable to an online application scenario, and the audio and video data of the window is efficiently compressed (for example, by using a video compression technology) and then sent to the terminal, and the terminal first decodes the compressed image after receiving the compressed image. Show.

Different from the online application scenario, the online application scenario is to transfer the image data of the application window of the server running application to the terminal, and the remote desktop transmits the entire desktop of the server to the terminal, but the terminal is in a window manner. display. Therefore, if the window of the remote desktop displayed by the terminal is partially covered by other windows, the area of the covered remote desktop corresponding to the server may be monochromatically filled and then encoded and transmitted. If the cover is completely covered, the image of the entire desktop is not used. Encoding, without transmission, can also reduce network bandwidth and reduce resource consumption of servers and terminals.

The method for transmitting a window image in the embodiment of the present invention is also applicable to a remote desktop application scenario.

It can be seen that, in the case of multiple windows, when the window is fully displayed, the window image code is transmitted to the terminal, and when the window is completely covered, the window image code does not need to be transmitted to the terminal, and the window is When partially covering, the image in the covered area is filled with a solid color, and then encoded and transmitted to the terminal, thereby reducing the network load, improving the operating efficiency of the terminal, and reducing the load on the server. As shown in FIG. 6, the embodiment of the present invention provides a server, which includes: a receiving unit 61. In the process of running an application, the receiving terminal sends the application corresponding to the virtual terminal. a window information of the window; a determining unit 62, configured to determine display area information of the window according to the window information;

The transmission unit 63 transmits, by the T, image data corresponding to the display area information of the window to the terminal according to the display area information of the window, so that the terminal performs display.

The server of the embodiment of the present invention sends image data corresponding to the display area information of the window according to the display area information of the window displayed by the terminal. When the window display area is smaller than the entire window area, the image corresponding to the entire window area does not need to be sent. Data to achieve the effect of reducing network load.

In the server of the embodiment of the present invention, before the receiving unit 61 receives the window information of the window corresponding to the application, the terminal sends a request for running the application carrying the application identifier to the server, and the server selects the application requested by the virtual machine running terminal according to the identifier. And the virtual machine and the terminal establish a connection for transmitting the application running data and receiving the information sent by the terminal. Therefore, the terminal transmits the window information of the application in the terminal display window or the user's operation for the application to the virtual machine running the application through the connection. The window is the window displayed by the application running on the terminal.

Optionally, the window information sent by the terminal includes: a window display state, a coordinate of any corner point of the cover area of the window, and a width and a height of the cover area. Wherein, the coordinates include an X coordinate and a Y coordinate, and the coordinate may adopt a coordinate of a screen coordinate system or a coordinate of a window coordinate system, and the screen coordinate system refers to a point of the upper left corner of the display screen as a coordinate origin (0, 0), and a horizontal direction is an X axis. The vertical direction is the Y axis, and the window coordinate system means that the upper left corner of the window is the coordinate origin (0, 0), the horizontal direction is the X axis, and the vertical direction is the Y axis; the window covering area can be one or more. Optionally, the window information may further include: a number of cloaking windows. The message type of the terminal to send the window information is Updata_Window_infrom, and the message format can be as shown in Table 1, so that the virtual machine parses the message to obtain the window information sent by the terminal. Optionally, the window information sent by the terminal includes: a window display state, a coordinate of any corner of the window, a width and a height of the window, a coordinate of the z-axis sequence larger than the corner of the window and the window covering the window, and the z-axis The order is larger than the window and covers the window width and height of the window. Wherein, the coordinates include an X coordinate and a Y coordinate, and the coordinate may adopt a coordinate of a screen coordinate system or a coordinate of a window coordinate system; the window may be one or more of the window and the window covering the window may be one or more. Optionally, the window information may further include: a number of cloaking windows. The message type of the terminal to send the window information is Updata_Window_infrom, and the message format can be as shown in Table 2 above, so that the virtual machine parses the message to obtain the window information sent by the terminal.

Optionally, the window information sent by the terminal includes: coordinates of any corner point of the window, width and height of the window, a coordinate of the Z axis is greater than a corner of the window of the window, and a Z axis order is greater than a window width of the window and height. The coordinates include an X coordinate and a γ coordinate, and the coordinate may adopt the coordinate of the screen coordinate system or the coordinate of the window coordinate system; the window whose Z axis sequence is larger than the window may be one or more. Optionally, the window information may further include: a number of cloaking windows. The message type of the terminal to send the window information is Updata_Window_infrom, and the message format can be as shown in Table 3 above, so that the virtual machine parses the message to obtain the window information sent by the terminal.

Optionally, the receiving unit 61 is further configured to receive a window location request message sent by the terminal, where the window location request message is used to obtain the window size and location information of the virtual machine. At this time, as shown in FIG. 7, the server provided by the embodiment of the present invention may further include a sending unit 71, configured to send the window size and location information of the virtual machine to the terminal.

Alternatively, the receiving unit 61 receives the window information sent by the terminal, where the window information includes the window size and location information of the terminal, or receives a window notification message sent by the terminal, and the window notifies the message carrying terminal of the window size and location information. At this time, as shown in FIG. 7, the server provided by the embodiment of the present invention may further include a coordinate conversion unit 72, configured to convert the coordinates of the terminal into two according to the window size and location information of the terminal, the window size and the location information of the virtual machine. The corresponding coordinates of the virtual machine.

The window size of the virtual machine can be obtained through the API provided by the operating system. Coordinate conversion formula: If the window size of the virtual machine is X1*Y1, the terminal window size is Χ2*Υ2, for example, the coordinates of the upper left corner of the terminal window are (xl, yl), and the coordinates of the upper left corner of the virtual machine window are (x2, y2). ), for a window with coordinates (al, bl), width cl, and height dl at the terminal, its coordinates in the virtual machine become a2=(al-xl+x2) *X2/X1 _; b2= (bl- yl+y2) *Y2/Yl ; width is c2=cl*X2/Xl; height is d2=dl*Y2/Yl.

When the window information is in the format of Table 1: the determining unit 62 can be specifically used for:

Or, according to the window display state in the window information is partially covered, according to the X coordinate, the Y coordinate, the width and the height of the cover area of any corner of the cover area in the window information, the display area of the window is determined to be the entire window except the cover area. The area, when there are a plurality of cover areas, determine a joint cover area composed of a plurality of cover areas, thereby determining that the display area of the window is an area of the entire window except the joint cover area.

Alternatively, when the window information is in the format of Table 2: the determining unit 62 can be specifically used to determine the display area of the window as the entire window according to the window display state in the window information.

Or, according to the window display state in the window information is partially covered, according to the X coordinate, the Y coordinate, the width and the height of any corner point of the window in the window information, and any corner of the window whose z-axis order is larger than the window and covers the window X coordinate, Y coordinate, width of the point Degree and height, determine the X coordinate, Y coordinate, width and height of the cover area of any corner of the cover field;

And determining, according to the X coordinate, the Y coordinate, the width and the height of the corner of the covering area, the display area of the window is an area other than the covering area of the entire window, and when there are a plurality of covering areas, determining a plurality of covering areas The joint occlusion area determines the display field of the window as the area of the entire window except the joint occlusion area.

Alternatively, when the window information is in the format of Table 3: determining unit 62, specifically for:

If the z-axis fires over the window of the window, the X coordinate, the Y coordinate, the width and the height of any corner of the window in the window information, and the window whose z-axis order is larger than the window and covers the window The X coordinate, Y coordinate, width and height of a corner point, determine the X coordinate, Y coordinate, width and height of the cover area of any corner point of the cover area;

According to the X coordinate, the Y coordinate, the width and the height of the corner of the cover area, it is determined whether the cover area completely covers the window. When there are multiple cover areas, it is determined whether the joint cover area composed of the plurality of cover areas is completely covered. The window; if the occlusion area or the joint occlusion area completely covers the window, the window display state of the window is full occlusion, thereby determining that the display field of the window is 0, and if the occlusion area or the joint occlusion area does not completely cover the window, The window display state of the window is partially covered, and the display area of the window is determined to be an area of the entire window except the cover area, or the display area of the window is determined to be an area of the entire window except the joint cover area.

Specifically, the transmission unit 63 can be configured to acquire image data of the entire window when the display area of the window is the entire window, and send the image data to the terminal for display by the terminal. Alternatively, when the display area of the window is an area of the entire window except the cover area, image data of the entire window is acquired, and the cover area is filled with a solid color or a colorless, and the filled image data is sent to the terminal, and the terminal displays .

Among them, the solid color is black or white.

Optionally, the server of the embodiment of the present invention, the receiving unit 61 is further configured to:

At this time, the transmission unit 63 is configured to reduce or not transmit the image of the window according to the display image transmission policy.

Wherein, the image of the window is degraded, for example, the HD image is degraded to a standard definition image, etc., to reduce the network load. Optionally, if the terminal determines that the image transmission policy is not to transmit the image of the window, the window display state in the window information sent by the terminal is completely covered by the window, so that the receiving unit 61 receives the window display in the window information sent by the terminal. The status is completely covered by the window.

The server of the embodiment of the present invention can be specifically understood according to the method for transmitting the window image of the foregoing embodiment, and is not used herein. Narration. As shown in FIG. 8, an embodiment of the present invention provides a window display method, including:

Step 81: The virtual machine sends window information of the window corresponding to the application to the server, so that the virtual machine on the server determines the display area information of the window according to the window information.

Step 82: Receive image data corresponding to the display area information of the window sent by the server and display the image data.

The execution body of the display method of the window of the embodiment of the present invention is a terminal.

In the display method of the window of the embodiment of the present invention, the terminal receives the image data corresponding to the display area information of the window sent by the server, and when the window display area is smaller than the entire window area, the image data corresponding to the entire window area does not need to be received, thereby achieving Reduce the effect of network load and improve terminal operating efficiency.

In the display method of the window of the embodiment of the present invention, the terminal sends a request for running the application carrying the application identifier to the server, and the server selects the application requested by the virtual machine running terminal according to the identifier, and the transmission application running data and the receiving terminal are established by the virtual machine and the terminal. The connection to send the message. Therefore, the terminal transmits the window information of the application in the terminal display window or the user's operation for the application to the virtual machine running the application through the connection. The window is the window displayed by the application running on the terminal.

The display method of the window of the embodiment of the present invention may further include:

When the terminal establishes a window, or the Z-axis sequence of the window changes, or the coordinates of any corner of the window change, or the width or height of the window changes, the window information of the window is determined.

Specifically, referring to FIG. 2 to FIG. 5, there are four applications in the terminal corresponding to the window A, the window B, the window C, and the window D. As shown in FIG. 9, the window information of the window is determined, including:

Step 91: The terminal acquires the Z-axis sequence of all windows of the terminal, the (X, Y) coordinates of any corner of all the windows, and the width and height of all the windows.

Specifically, the terminal acquires the axis sequence of all windows, such as the axis sequence of window C > the axis sequence of window D > the axis sequence of the window 〉 the axis sequence of the window Α.

The following is an example of the upper left corner. The coordinates of the upper left corner (X, Y) of the terminal acquisition window A are (50, 50), the width is 150, and the height is 100. The upper left corner of the window Χ (Χ, Υ) The coordinates are (70,80), the width is 60, and the height is 120. The upper left corner of window C (Χ,Υ) has coordinates (100,110), width 140, and height 100; as in the upper left corner of window D (Χ , Υ) The coordinates are (210, 160), the width is 20, and the height is 20.

Step 92: The terminal determines whether the axis sequence of the window is maximum, and if yes, the window display state is full display, otherwise, the process proceeds to step 93 to determine (Χ, Υ) coordinates and window of any corner of the window of the axis window Width and height.

Specifically, the terminal determines that the Z-axis order of the window C is the largest according to the axis sequence of the window C, the axis sequence of the window D, and the axis sequence of the window ,, that is, the window C is the top of the Z-axis sequence. The window, according to the window at the top of the Z-axis sequence, covers other windows in the Z-axis sequence, and determines that the window display state of the window C is full display.

Since the window Π Α, window D, and window Β Ζ axis is not the maximum sequence proceeds to step 93 further determines _Ζ axis of order greater than any corner of the window the window points (Χ, Υ) coordinates and the width and height of the window.

Step 93: The terminal is greater than (Χ, Υ) coordinates of any corner of the window of the window according to the axis sequence, and the width and height of the window. Determining whether the window is covered by a window whose z-axis order is larger than the window. If not, the window display state of the window is full display, otherwise it is determined that the axis sequence is larger than the window and covers any corner of the window of the window (Χ, Υ) The coordinates and the width and height of the window.

Specifically, the axis sequence of the window C, the axis sequence of the window D, and the axis sequence of the window 大于 are larger than the axis sequence of the window ,, and the terminal is based on the horizontal and vertical axes (Χ, Υ) of the upper left corner of the window A and the window. The width and height of the window are confirmed to be larger than the window Α. The upper left corner of the window is located in the window area of window A, and it is determined that window A is covered by window B. Specifically, as shown in FIG. 3, for window A, the terminal is based on the Z-axis sequence of window B> the Z-axis order of window A, and the horizontal and vertical axis (X, Y) coordinates of window A are (50, 50), width 150, height 100; The upper left corner of the window Χ (Χ, Υ) coordinates are (70, 80), and the coordinates of the upper left corner of the window 确定 are determined to be within the window area of the window Α. Further, the terminal can calculate the coordinates (Χ, Υ) of the upper right corner of the window 根据 according to the XY axis (Χ, Υ) coordinates of the window A as (50, 50), the width 150, and the height of 100 degrees (200, 50), And calculating the coordinates of the lower left corner point (50, 150), determining that the X-axis coordinate 70 of the upper left corner of the window 大于 is greater than the X-axis coordinate 50 of the upper left corner of the window ,, and less than the X-axis coordinate 200 of the upper right corner of the window A, And determining that the Y-axis coordinate 80 of the upper left corner of the window B is smaller than the Y-axis coordinate 150 of the lower left corner of the window A, and larger than the Y-axis coordinate 50 of the upper left corner of the window A, thereby determining that the upper left corner of the window B is located in the window A. Within the window area.

As another alternative, the terminal confirms according to the horizontal and vertical axes (Χ, Υ) of the upper left corner of the window A, the width and height of the window, and the coordinates of the upper left corner of the window 和 and/or the height of the window Β. 1 axis sequence is larger than the window Α window Β window Β the upper left corner of the point Y axis is sitting on the small D · window A lower left corner point Y coordinate, or the window B lower left corner point Y coordinate is greater than the window A The Y-axis coordinate of the upper left corner point determines that window A is not covered by window B. Further, the terminal determines the upper left corner of the window 根据 according to the horizontal and vertical axes (Χ, Υ) of the upper left corner of the window A and the width and height of the window and the coordinates of the upper left corner of the window 和 and/or the height of the window Β. The X coordinate of the point is larger than the X-axis coordinate of the upper left corner of the window ,, smaller than the X-axis coordinate of the upper right corner of the window A, and the Y-axis coordinate of the lower left corner of the window B is larger than the Y-axis coordinate of the upper left corner of the window A. , then it is determined that window A is covered by window B. And the first cover area in which the window A is covered can be further determined.

According to the window information of the window A, the (Χ, Υ) coordinates of the upper left corner of the window A are (50, 50), the width is 150, the height is 100, and the top left corner of the window C whose axis sequence is larger than the window Α is (Χ, Υ) The coordinates are (100, 110), the width is 140, and the height is 100. It is determined that the coordinates of the upper left corner of the window C are located in the window area of the window Α. Further, according to the upper left corner point (Χ, Υ) of the window A, the coordinates (50, 50), the width 150, and the height of 100 degrees are calculated as the coordinates (Χ, Υ) of the upper right corner of the window 为 (200, 50), and Calculating the coordinates of the lower left corner point (50, 150), determining that the X-axis coordinate 100 of the upper left corner of the window C is greater than the X-axis coordinate 50 of the upper left corner of the window ,, less than the X-axis coordinate 200 of the upper right corner of the window A, and It is determined that the Y-axis coordinate 110 of the upper left corner of the window C is smaller than the Y-axis coordinate 150 of the lower left corner of the window A, and is greater than the Y-axis coordinate 50 of the upper left corner of the window A, thereby determining that the upper left corner of the window C is located in the window A. Within the window area. Further, the z-axis order of the window C is larger than the window A and the window A is covered.

According to the window information of window A, the coordinates of (左, Υ) of the upper left corner of window A are (50, 50), and the coordinates of the upper left corner of window D ((, Υ) are (210, 160), and the width is 20 , the height is 20, the coordinates of the upper right corner of the calculation window ( (X, Υ) are (200, 50), and the coordinates of the lower left corner point (50, 150) are calculated, and the X-axis coordinate 210 of the upper left corner of the window D is determined to be greater than The X-axis coordinate 200 of the upper right corner of the window A, the Y-axis coordinate 160 of the upper left corner of the window D is greater than the Y-axis coordinate 150 of the upper right corner of the window A, thereby determining that the upper left corner of the window D is located in the window area of the window A Outside. Further determining that window D does not obscure window A.

In addition, it is assumed that the X-axis coordinate of the upper left corner of the window B is larger than the X-axis coordinate of the upper right corner of the window A, or the Y-axis coordinate of the upper left corner of the window B is larger than the Y-axis coordinate of the lower left corner of the window A, that is, The top left corner of window B is not located in window A. In the window area, it is determined that the window A is not covered by the window B; it is assumed that the X-axis coordinate of the upper left corner of the window C is larger than the X-axis coordinate of the upper right corner of the window A, or the Y-axis coordinate of the upper left corner of the window C If the Y-axis coordinate of the lower left corner of the window A is larger, that is, the upper left corner of the window C is not located in the window area of the window A, it is determined that the window A is not covered by the window C; therefore, it is determined that the window A is not covered by the window B or the window ( The window D is covered, and the window display state of the window A is full display.

As shown in FIG. 4, for window B, the Z-axis order of window C and the Z-axis order of window D are larger than the Z-axis order of window B, and the terminal coordinates (70, 80) according to the upper left corner of window B (Χ, Υ). ), the width is 60, the height is 120, and the "Y" coordinate of the upper left corner of the window C whose axis sequence is larger than the window 为 is (100, 110), the width is 140, and the height is 100. The upper left of the window C is determined. The corner coordinates are in the window area of the window Β. Further, according to the upper left corner of the window Β (Χ, Υ) coordinates are (70, 80), the width is 60, the height is 120, and the coordinates of the upper right corner of the window Β (Χ, Υ) are calculated as (130, 80) And calculating the coordinates of the lower left corner point (70, 200), determining that the X-axis coordinate 100 of the upper left corner of the window C is greater than the X-axis coordinate 70 of the upper left corner of the window ,, and less than the X-axis coordinate 130 of the upper right corner of the window Β And determining that the axis coordinate 110 of the upper left corner of the window C is smaller than the pivot coordinate 200 of the lower left corner of the window ,, which is greater than the 坐标 axis coordinate 80 of the upper left corner of the window ,, thereby determining that the upper left corner of the window C is located in the window Inside the window area. Further, it is determined that the axis of the window C is in the window and the window is closed.

According to the upper left corner of the window (X, Υ), the coordinates are (70, 80), the width is 60, the height is 120, and the upper left corner of the window D (X, Υ) coordinates are (210, 160), and the width is 20, the height is 20, the coordinates of the upper right corner of the calculation window Χ (Χ, Υ) are (130, 80), and the coordinates of the lower left corner point (70, 200) are calculated, and the X-axis coordinate 210 of the upper left corner of the window D is determined. It is greater than the X-axis coordinate 130 of the upper right corner of the window ,, thereby determining that the upper left corner of the window D is outside the window area of the window Β. Further confirmation window D does not cover the window Β.

Suppose that the X-axis coordinate of the upper left corner of the window C is larger than the X-axis coordinate of the upper right corner of the window ,, or the 坐标 coordinate of the upper left corner of the window C is the Υ coordinate of the lower left corner of the window ,, that is, the window The upper left corner of C is not in the window area of the window ,, then it is determined that the window Β is not covered by the window (;, window D, and the window 显示 window display state is full display.

As shown in FIG. 5, for the D-window D, the Z-axis order of the window C is larger than the Z-axis order of the window D, and the terminal coordinates (X, Y) of the upper left corner of the window D according to the window information of the window D is (210, 160), width is 20, height is 20, the coordinates (X, Y) of the upper right corner of the calculation window D are (230, 160), and the coordinates of the lower left corner of the calculation window D (210, 180), according to the z-axis order The (Χ, Υ) coordinates of the upper left corner of the window C larger than the window D are (100, 110), the width is 140, and the height is 100. The coordinates of the upper right corner of the calculation window C (Χ, Υ) are (240, 100). And calculating the coordinates of the lower left corner of the window C (100, 210), determining that the X-axis coordinate 210 of the upper left corner of the window D is greater than the X-axis coordinate 100 of the upper left corner of the window C, which is smaller than the upper right corner of the window C. The X-axis coordinate 240, and the paraxial coordinate 160 of the upper left corner of the window D is determined to be smaller than the paraxial coordinate 210 of the lower left corner of the window C, and greater than the paraxial coordinate 110 of the upper left corner of the window C, thereby determining the axis of C The sequence is larger than window D and covers window D.

Step 94: The terminal is larger than the (Χ, Υ) coordinate of any corner of the window according to the z-axis sequence and covering the window, and the width and height of the window, and the (Χ, Υ) coordinates of any corner of the window and the window. Width and height, determine the (Χ, Υ) coordinates of any corner of the window's covered area and the width and height of the covered area.

As shown in FIG. 3, for the window Α, according to the width 60 and the height 120 of the window ,, and the coordinates of the upper left corner of the window 以及 and the width and height of the window A, the occlusion area of the window A covered by the window B is obtained (ie, A covered area). Specifically, the coordinates of the upper left corner of the terminal window A and the width and height of the window A obtain the coordinates (Χ, Υ) of the upper right corner of the window A as (200, 50) And the coordinates of the lower left corner point (50, 150), and the coordinates of the upper right corner of the window B (X, Y) are (130, 80) and the lower left corner according to the coordinates of the upper left corner of the window B and the width and height of the window B. Point coordinates (70, 200), the upper right corner of the terminal determining window 位于 is located in the area of the window ,, determining that the width of the first covering area is the width 60 of the window B, and the height of the first covering area is the lower left corner of the window A The Y-axis coordinate 150 of the point minus the Y-axis coordinate 80 of the upper left corner of the window B is 70. Therefore, the (X, Y) coordinate of the upper left corner of the first cover area of the window A is (70, 80), the first cover The area is 60 wide and 70 high. Further, assuming that the terminal determines that the coordinates of the upper corner of the window 位于 are outside the area of the window ,, that is, the X-axis coordinate of the upper right corner of the window B is greater than the X-axis coordinate of the window A, the width of the first covering area is the window. The X-axis coordinate of the upper right corner of A is subtracted from the coordinates of the upper left corner of window B. Further, assuming that the lower left corner of the terminal window B is located in the area of the window A, the height of the first covered area is the height of the window B.

The masked area (i.e., the second covered area) in which the window A is covered by the window C is obtained according to the width 140 of the window C and the height 100, and the coordinates of the upper left corner of the window A and the width and height of the window A. Specifically, according to the coordinates of the upper left corner of the window A and the width and height of the window A, the coordinates (Χ, Υ) of the upper right corner of the window A are (200, 50) and the coordinates of the lower left corner (50, 150), and According to the coordinates of the upper left corner of the window C and the width and height of the window C, the coordinates (X, Υ) of the upper right corner of the window C are calculated as (240, 100) and the coordinates of the lower left corner (100, 210), and the window C is determined. The upper right corner point is outside the area of the window 以及 and the lower left corner of the window C is outside the area of the window A, and then the width of the second covered area is determined as the X-axis coordinate 200 of the upper right corner of the window A minus the window C The X-axis coordinate 100 of the upper left corner point is 100, and the height of the second masking area is the Y-axis coordinate 150 of the lower left corner of the window A minus the Y-axis coordinate 110 of the upper left corner of the window C is 40. Therefore, the (X, Y) coordinate of the upper left corner of the second cover area of the window A is (100, 110), and the second cover area is 100 wide and 40 high. Further, assuming that the coordinate position of the upper right corner of the window C is determined within the area of the window ,, the width of the second mask area is the width of the window C. Further, assuming that the lower left corner of the window C is located in the area of the window A, the height of the second covered area is the height of the window C.

As shown in Fig. 4, for the window B, the window B is covered by the window C according to the width 140 and the height 100 of the window C, and the coordinates of the upper left corner of the window B and the width and height of the window B. Specifically, according to the coordinates of the upper left corner of the window B and the width and height of the window B, the coordinates (X, Y) of the upper right corner of the window B are (130, 80) and the coordinates of the lower left corner (70, 200), and Calculate the coordinates of the upper right corner of the window C (Χ, Υ) as (240, 100) and the coordinates of the lower left corner (100, 110) according to the coordinates of the upper left corner of the window C and the width and height of the window C, and determine the window C. The upper right corner point is outside the area of the window ,, and the lower left corner of the window C is outside the area of the window ,, then the width of the occlusion area is determined as the X-axis coordinate 130 of the upper right corner of the window 减 minus the upper left corner of the window C The X-axis coordinate 100 is 30, and the height of the cover area is the Υ axis coordinate 200 of the window Β lower left corner point minus the Υ axis coordinate 110 of the upper left corner of the window C is 90. Therefore, the (Χ, Υ) coordinates of the upper left corner of the window area are (100, 110), the cover area is 30 wide, and the height is 90. Further, assuming that the coordinates of the upper right corner of the window C are determined to be within the area of the window ,, the width of the occlusion area is the width of the window C. Further, assuming that the lower left corner of the window C is located in the area of the window, the height of the cover field is the height of the window C.

As shown in Fig. 5, for window D, the coordinates (X, Υ) of the upper right corner of D are (210, 160), the coordinates of the lower left corner of window D (210, 180), and the coordinates of the upper right corner of window C ( Χ, Υ) is (240, 100), and the coordinates of the lower left corner of the window C (100, 210), determine the coordinates of the upper left corner of the window D (Χ, Υ) (210, 160), window D The coordinates of the upper right corner point are located in the area of the window C, and the width of the cover area is the width 20 of the window D, and the lower left corner of the window C is located in the area of the window C. The height of the covered area is the height 20 of the window D.

Step 95: When the terminal determines that the cover area of the window completely covers the window, determine that the window display state of the window is full cover. When the cover area of the window does not completely cover the window, the window display state of the window is partially covered.

According to the coordinates (70, 80) of the upper left corner of the joint cover area formed by the first cover area and the second cover area in the area of the window A, it is determined that the joint cover area does not completely cover the window A, that is, the window display of the window A The status is partially covered. Assume that the X-axis coordinate of the upper left corner of the joint cover area is less than or equal to the X-axis coordinate of the upper left corner of the window A, and the Y-axis coordinate of the upper left corner of the joint cover area is smaller than the Y-axis coordinate of the upper left corner of the window A, and the joint cover The width of the area is greater than or equal to the width of the window A, and the height of the joint cover area is high, and the height of the window A is determined to completely cover the window A, that is, the window display state of the window A is fully covered.

According to the coordinates (100, 110) of the upper left corner of the cover area in the area of the window B, it is determined that the cover area does not completely cover the window B, that is, the window display state of the window B is partially covered. Assume that the X-axis coordinate of the upper left corner of the cover area is less than or equal to the X-axis coordinate of the upper left corner of the window B. The Y-axis coordinate of the upper left corner of the cover area is less than or equal to the Y-axis coordinate of the upper left corner of the window B, and the width of the cover area is large. For the width of the window B, the height of the cover area is greater than or equal to the height of the window B, it is determined that the cover area completely covers the window B, that is, the window display state of the window B is fully covered.

According to the (X, Y) coordinate of the upper left corner of the covering area of the window D is (210, 160), the width of the covering area is the width 20 of the window D, and the height of the covering area is the height 20 of the window D, and the covering area is completely determined. The window D is covered, that is, the window display state of the window D is fully covered.

Optionally, in the display method of the window of the embodiment of the present invention, the window information includes a window size and location information of the terminal, or the terminal sends a window notification message, where the window notifies the message carrying the window size and location information of the terminal, so that the server The window size and location information of the terminal, the window size and location information of the virtual machine convert the coordinates of the terminal into the corresponding coordinates of the virtual machine.

Alternatively, the display method of the window of the embodiment of the present invention may further include:

Sending a window location request message to the server, the window location request message is used to request the window size and location information of the virtual machine.

Receiving window size and location information of the virtual machine sent by the server;

The coordinates of the terminal are converted into corresponding coordinates of the virtual machine according to the window size and position information of the terminal, the window size and position information of the virtual machine.

The window size of the terminal can be obtained through an API provided by the operating system. Coordinate conversion formula: If the window size of the virtual machine is X1*Y1, the terminal window size is Χ2*Υ2, for example, the coordinates of the upper left corner of the terminal window are (xl, yl), and the coordinates of the upper left corner of the virtual machine window are (x2, y2). ), for a window with coordinates (al, bl), width cl, and height dl at the terminal, its coordinates in the virtual machine become a2=( al- xl+x2) *X2/X1; b2= (bl-yl+y2) *Y2/Yl; width is c2=cl*X2/Xl; high is d2=dl*Y2/Yl o Optionally, the display method of the window of the embodiment of the present invention may further include:

Determining that the display image transmission strategy is to degrade the image of the window or not to transmit the image of the window according to the Z-axis order of the window, and according to the cover area and the display area of the window When the ratio is greater than a certain threshold, determining that the display image transmission policy is to degrade the image of the window or not to transmit the image of the window;

The determined display image transmission policy is sent to the server. Wherein, the image of the window is degraded, for example, the HD image is degraded to a standard definition image, etc., to reduce the network load. Optionally, if the terminal determines that the image transmission policy is not to transmit the image of the window, the window display state in the window information sent by the terminal is completely covered by the window. As shown in FIG. 10, the embodiment of the present invention provides a terminal, including: a sending unit 101, configured to send a window of a window corresponding to the application in a process of running the application by the virtual machine. The information is sent to the server, so that the virtual machine on the server determines the display area information of the window according to the window information.

The receiving unit 102 is configured to receive image data corresponding to the display area information of the window sent by the server and display the image data. The terminal of the embodiment of the present invention receives the image data corresponding to the display area information of the window sent by the server. When the window display area is smaller than the entire window area, the image data corresponding to the entire window area does not need to be received, thereby reducing the network load. And improve terminal operating efficiency.

In the embodiment of the present invention, the terminal sends a request for running the application carrying the application identifier to the server, and the server selects the application requested by the virtual machine running terminal according to the identifier, and the virtual machine and the terminal establish a connection for transmitting the application running data and transmitting the information by the receiving terminal. Therefore, the terminal transmits the window information of the application in the terminal display window or the user's operation for the application to the virtual machine running the application through the connection. The window is the window displayed by the application running on the terminal.

As shown in FIG. 11, the terminal of the embodiment of the present invention further includes:

The determining unit 111 is configured to determine the window information of the window when the terminal establishes a window, or the z-axis sequence of the window changes, or the coordinate of any corner of the window changes, or the width or height of the window changes.

The determination unit 11 body includes:

The obtaining subunit 1111 is configured to acquire the Z-axis sequence of all windows of the terminal, the (Χ, Υ) coordinates of any corner point of all the windows, and the width and height of all the windows;

The first determining sub-unit 1112 is configured to determine whether the axis sequence of the window corresponding to the application is the maximum, and if yes, the window display state of the window corresponding to the application is a full display;

a second determining sub-unit 1113, configured to determine, by the first determining sub-unit 1112, that the axis sequence of the window corresponding to the application is not the maximum, according to any one of the corners of the window corresponding to the window corresponding to the application (Χ, Υ The coordinates and the width and height of the window determine whether the window corresponding to the application is covered by the window whose axis sequence is larger than the window corresponding to the application. If not, the window display state of the corresponding window of the application is full display, otherwise the axis is determined. The (序,Υ) coordinates of the corners of the window corresponding to the window of the application and the width and height of the window;

a third determining subunit 1114, configured to: (Χ, Υ) coordinates of any corner of the window corresponding to the window corresponding to the application and cover the window corresponding to the application, and the width and height of the window, and the application corresponding to the application The (Χ, Υ) coordinates of any corner of the window and the width and height of the window, determine the (Χ, Υ) coordinates of any corner of the covered area of the window corresponding to the application, and the width and height of the occluded area;

The fourth determining sub-unit 1115 is configured to determine that the cover area of the window corresponding to the application completely covers the window corresponding to the application, and determine that the window display state of the window corresponding to the application is full coverage, and the cover area of the corresponding window of the application is not When the window corresponding to the application is completely covered, it is determined that the window display state of the window corresponding to the application is partially covered. Optionally, the window information sent by the sending unit 101 may include the window size and location information of the terminal, or the sending unit 101 sends a window notification message, where the window notifies the message carrying the window size and location information of the terminal, so that the server is configured according to the terminal. The window size and location information, the window size and location information of the virtual machine convert the coordinates of the terminal to the corresponding coordinates of the virtual machine.

Alternatively, the sending unit 101 may be further configured to send a window location request message to the server, where the window location request message is used to request to obtain the window size and location information of the virtual machine; at this time, the receiving unit 102 may further be configured to receive The window size and location information of the virtual machine sent by the server; at this time, the terminal of the embodiment of the present invention further includes a coordinate conversion 112, which is used to set the coordinates of the terminal according to the window size and location information of the terminal, the window size and the location information of the virtual machine. Convert to the corresponding coordinates of the virtual machine.

The window size of the terminal can be obtained through an API provided by the operating system. Coordinate conversion formula: If the window size of the virtual machine is X1*Y1, the terminal window size is Χ2*Υ2, for example, the coordinates of the upper left corner of the terminal window are (xl, yl), and the coordinates of the upper left corner of the virtual machine window are (x2, y2). ), for a window with coordinates (al, bl), width cl, and height dl at the terminal, its coordinates in the virtual machine become a2=( al- xl+x2) *X2/X1; b2= (bl-yl+y2) *Y2/Yl _; width is c2=cl*X2/Xl; height is d2=dl*Y2/Yl.

Optionally, the determining unit 111 is further configured to determine, according to the Z-axis sequence of the window that is smaller than the Z-axis sequence of the other windows of the terminal, that the display image transmission policy is to degrade the image of the window or not to transmit the image of the window. Or determining, according to the ratio of the cover area of the window to the display area, that the display image transmission policy is to degrade the image of the window or not to transmit the image of the window;

The sending unit 101 is further configured to send the determined display image transmission policy to the server.

Specifically, it may be determined by the first determining subunit 1112 whether the Z-axis order of the window is smaller than the Z-axis order of other windows of the terminal. The fourth determining sub-unit 1115 may determine that the ratio of the covered area of the window to the display area is greater than a certain threshold.

Wherein, the image of the window is degraded, for example, the HD image is degraded to a standard definition image, etc., to reduce the network load. Optionally, if the terminal determines that the image transmission policy is displayed as not transmitting the image of the window, the window display state in the window information sent by the sending unit 101 is full coverage of the window.

The terminal of the embodiment of the present invention can be specifically understood by referring to the display method of the window in the foregoing embodiment, and details are not described herein. As shown in FIG. 12, an application schematic diagram of a display system of a window according to an embodiment of the present invention includes a terminal 121 and a server 122.

Specifically, the terminal 121 includes a determining unit 1211, a coordinate converting unit 1212, an input device unit 1213, a transmitting unit 1214, a receiving unit 1215, and a display unit 1216. The server 122 includes a receiving unit 1221, an application running unit 1222, an image capturing unit 1223, an image filling unit 1224, an image position processing unit 1225, an encoding unit 1226, and a transmitting unit 1226.

The determining unit 1211 is configured to determine window information of the window.

The coordinate conversion unit 1212 is configured to convert the window coordinates of the terminal into corresponding coordinates of the window on the server-side virtual machine.

The input device unit 1213 is configured to generate an input event. The input events operated by the user may specifically refer to keyboard and mouse events (terminals of the PC model), button or touch screen events (mobile phone or PDA type terminals), remote control or gamepad operation events (television type terminals), and the like.

The sending unit 1214 is configured to send window information to the receiving unit 1221.

The receiving unit 1221 is configured to receive the window information sent by the sending unit 1214, and send the information to the application running unit 1222 and the image. Location processing unit 1225.

The application running unit 1222 controls the running of the application on the virtual machine, including starting, suspending, and aborting the application.

The processing unit 1225 is configured to determine the window covering area information when the window portion is covered, or calculate the window covering area information according to the window covering position information.

The image intercepting unit 1223 is configured to intercept an image of the window according to the running of the application running unit 1222, and send the captured image to the encoding unit 1226 when the window is fully displayed.

The image filling unit 1224 is configured to fill a masked area in the image of the cut window with a solid color when the window portion is covered, and send the clipped image with the fill to the encoding unit 726.

The encoding unit 1226 is configured to encode the received image and send it to the sending unit 1226.

The sending unit 226 is configured to send the encoded image to the receiving unit 1215.

The receiving unit 1215 is configured to receive the encoded image from the sending unit 1226.

The display unit 1216 is configured to perform decoding display on the encoded image received by the receiving unit 1215.

The terminal 121 and the server 122 in the display system of the embodiment of the present invention can be specifically understood by referring to the terminal and the server in the above embodiments, and details are not described herein. A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. In execution, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Claims

Rights request

1. A window image transmission method, characterized by: including:

When the virtual machine is running Yingchuan, the receiving terminal sends the window information of the window corresponding to the application.

Determine the display area of the window according to the window information;

According to the display area of the window, image data corresponding to the display area information of the window is sent to the terminal, so that the terminal performs display.

2. The window image transmission method according to claim 1, characterized in that the window information includes the coordinates of any corner point of the window, the width and height of the window, and any corner point of the window whose Z-axis sequence is greater than the window. The coordinates and Z-axis sequence of are greater than the window width and height of the window, and the specific determination of the display area of the window is:

According to the X coordinate, Y coordinate, width and height of any corner point of the window in the window information, and the X coordinate, Y coordinate, width and height of any corner point of the window whose z-axis order is greater than the window, determine the z-axis order is greater than the Whether the window of the window covers the window. If the window with a z-axis sequence greater than the window does not cover the window, the window display status of the window is full display, and the display area of the window is determined to be the entire window;

If the window whose z-axis sequence is equal to the window covers the window, then the X coordinate, Y coordinate, width and height of any corner point of the window in the window information, and any window whose z-axis sequence is greater than the window and covers the window. The X coordinate, Y coordinate, width and height of a corner point, determine the X coordinate, Y coordinate, width and height of any corner point of the covered area, and the width and height of the covered area;

Based on the X coordinate, Y coordinate of any corner point of the covered area, and the width and height of the covered area, determine whether the covered area completely covers the window. When there are multiple covered areas, determine whether the joint covered area composed of multiple covered areas is completely covered. This window; if the covering area or the combined covering area completely covers the window, then the window display status of the window is fully covered, thereby determining that the display field of the window is 0, if the covering area or the combined covering area does not completely cover the window, then The window display status of the window is partially covered, and the display area of the window is determined to be the entire area of the window except the covering area, or the display area of the window is determined to be the entire area of the window except the joint covering area.

3. The window image transmission method according to claim 1, wherein the window information includes the window display status, coordinates of any corner point of the covered area of the window, and the width and height of the covered area, and the determined The display area of the above window is specifically:

According to the window display status in the window information, when the window display status is full display, determine that the display area of the window is the entire window:

Or, when the window display status in the window information is fully covered, determine the display area of the window to be 0;

Or, when the window display status in the window information is partially covered, determine the display area of the window to be the entire window except the covered area based on the X coordinate, Y coordinate of any corner point of the covered area in the window information, and the width and height of the covered area. area, when there are multiple covering areas, determine the joint covering area composed of multiple covering areas, thereby determining the display area of the window to be the area of the entire window except the joint covering area.

4. The window image transmission method according to claim 1, characterized in that, the window information includes: The window information includes: window display status, coordinates of any corner point of the window, width and height of the window, Z-axis sequence greater than The coordinates of any corner point of the window and covering the window, and the width and height of the window whose Z-axis sequence is greater than the window and covering the window, the determination of the display area of the window is specifically: When the window display status in the window information is full display, the display area of the window is determined to be the entire window; When the window display status in the window information is full display, the display area of the window is determined to be the entire window;

Or, when the window display status is partially covered according to the window information, according to the X coordinate, Y coordinate, width and height of any corner point of the window in the window information, and any corner of the window whose z-axis order is greater than the window and covers the window. The X coordinate, Y coordinate, width and height of the point determine the X coordinate, Y coordinate, width and height of any corner point of the covered area, and the width and height of the covered area;

And based on the X coordinate, Y coordinate of any corner point of the covering area, and the width and height of the covering area, determine the display area of the window to be the area of the entire window except the covering area. When there are multiple covering areas, determine the composition of multiple covering areas. joint covering area, thereby determining the display area of the window to be the area of the entire window except the joint covering area.

5. The window image transmission method according to any one of claims 1 to 4, characterized in that the method further includes: determining the image data corresponding to the display area according to the display area.

6. The window image transmission method according to claim 5, characterized in that, according to the display area of the window, image data corresponding to the display area information of the window is sent to the terminal, so that the The terminal displays, including:

When the display area of the window is the entire window, obtain the image data of the entire window, send it to the terminal, and display it by the terminal; or, when the display area of the window is the area of the entire window except the covered area, obtain the image data of the entire window , fill the covered area with a solid color or no color, send the filled image data to the terminal, and display it on the terminal.

7. The window image transmission method according to any one of claims 1 to 6, characterized in that the method further includes: receiving a display image transmission strategy sent by the terminal, and the display image transmission strategy includes the display image transmission strategy according to the The Z-axis sequence of the window is the Z-axis sequence of other windows of the terminal. The image of the window is transmitted with reduced quality or the image of the window is not transmitted, or the ratio of the covered area to the display area of the window is greater than a certain When the threshold is reached, determine whether to transmit the image of the window at a reduced quality or not to transmit the image of the window;

According to the display image transmission strategy, the image of the window is transmitted or not transmitted corresponding to reduced image quality.

8. A server, characterized in that it includes:

The receiving unit is configured to receive the terminal to send window information of the window corresponding to the application while the virtual machine is running the application; the determining unit is configured to determine the display area information of the window based on the window information;

The transmission unit is configured to send image data corresponding to the display area information of the window to the terminal according to the display area information of the window, so that the terminal performs display.

9. The server according to claim 8, wherein the window information received by the receiving unit includes the coordinates of any corner point of the window, the width and height of the window, and the Z-axis sequence of the window greater than the window. The coordinates and Z-axis sequence of any corner point are greater than the window width and height of the window, and the determination unit is specifically used to:

If the z-axis sequence is higher than the window covering the window, then the X coordinate and Y coordinate of any corner point of the window in the window information are The mark, width and height, as well as the X coordinate, Y coordinate, width and height of any corner point of the window whose z-axis sequence is greater than the window and covers the window, determine the X coordinate, Y coordinate of any corner point of the covered area, and the width and height;

Based on the X coordinate, Y coordinate of any corner point of the covered area, and the width and height of the covered area, determine whether the covered area completely covers the window. When there are multiple covered areas, determine whether the joint covered area composed of multiple covered areas is completely covered. This window; if the covering area or the combined covering area completely covers the window, then the window display status of the window is fully covered, thus determining the display field of the window to be 0, if the covering area or the combined covering area does not completely cover the window, then The window display status of the window is partially covered, and the display area of the window is determined to be the entire area of the window except the covering area, or the display area of the window is determined to be the entire area of the window except the joint covering area.

10. The server according to claim 8, wherein the window information received by the receiving unit includes window display status, coordinates of any corner point of the covered area of the window, and the width and height of the covered area, so The above determination unit is specifically used for:

11. The server according to claim 8, characterized in that, the window information received by the receiving unit includes: window information includes: window display status, coordinates of any corner point of the window, width and height of the window, The Z-axis sequence is greater than the coordinates of any corner point of the window and covers the window, and the Z-axis sequence is greater than the window and covers the width and height of the window. The determination unit is specifically used:

According to the window information, when the window display status is full display, the display area of the window is determined to be the entire window; according to the window information, when the window display status is full display, the window display area is determined to be the entire window;

Or, when the window display status is partially covered according to the window information, according to the X coordinate, Y coordinate, width and height of any corner point of the window in the window information, and any corner of the window whose z-axis order is greater than the window and covers the window. The X coordinate, Y coordinate, width and height of the point determine the X coordinate, Y coordinate of any corner point of the covered area, the width and height of the covered area; According to the X coordinate, Y coordinate of any corner point of the covered area, the width and height of the covered area height, the display area of the window is determined to be the area of the entire window except the covering area. When there are multiple covering areas, the joint covering area composed of multiple covering areas is determined, so that the display area of the window is determined to be the area of the entire window except the joint covering area. area.

12. The server according to any one of claims 8-11, characterized in that, the transmission unit is specifically used when the display area of the window is the entire window, obtains the image data of the entire window, and sends it to the terminal. display on the above terminal;

Or, when the display area of the window is the area of the entire window except the covered area, the image data of the entire window is obtained, the covered area is filled with solid color or colorless, and the filled image data is sent to the terminal, and the terminal displays it.

13. The server according to any one of claims 8-11, characterized in that the receiving unit is also used to receive the terminal The display image transmission strategy sent by the terminal, the display image transmission strategy includes transmitting the image of the window at a reduced quality or not transmitting the image of the window according to the z-axis order of the window is smaller than the z-axis order of other windows of the terminal. , or determine whether to transmit the image of the window in a degraded manner or not to transmit the image of the window according to the ratio of the covering area to the display area of the window when it reaches a certain threshold; the transmission unit is also used to transmit the image of the window according to the display The image transmission strategy corresponds to reducing the image quality and transmitting or not transmitting the image of the window.

14. A window display method, which is characterized by including:

During the process of running the application, the virtual machine sends the window information of the window corresponding to the application to the server, so that the virtual machine on the server determines the display area information of the window based on the window information;

Receive image data corresponding to the display area information of the window sent by the server and display it.

15. The window display method according to claim 14, characterized in that the method further includes determining the window information of the window to be sent, specifically including:

Obtain the Z-axis sequence of all windows of the terminal, the (Χ, Y) coordinates of any corner point of all windows, and the width and height of all windows; determine whether the Z-axis sequence of the window corresponding to the application is the largest, and if so, the application corresponds to The window display status of the window is full display, otherwise according to the (Χ, Υ) coordinates of any corner point of the window whose ζ-axis order is greater than the window corresponding to the application and the width and height of the window, determine whether the window corresponding to the application is ζ-axis The window cover is larger than the window corresponding to the application;

If the window whose _Z- axis order is greater than the window corresponding to the application does not cover the window corresponding to the application, the window display state of the window corresponding to the application is full display. If the window whose Z-axis order is greater than the window corresponding to the application covers the window corresponding to the application of the window, determine the ( _X , Y) coordinates of any corner point of the window that is greater than the window corresponding to the application and cover the window, and the width and height of the window; determine the window corresponding to the application according to the ζ-axis order and cover the window The (X, Y) coordinates of any corner point of the window corresponding to the application and the width and height of the window, and the (X, Y) coordinates of any corner point of the window corresponding to the application and the width and height of the window, determine the Apply the (Χ, Y) coordinates of any corner point of the covered area of the corresponding window and the width and height of the covered area;

When it is determined that the covering area of the window corresponding to the application completely covers the window corresponding to the application, it is determined that the window display status of the window corresponding to the application is fully covered. When the covering area of the window corresponding to the application does not completely cover the window corresponding to the application, Make sure that the window display status of the window corresponding to the application is partially covered.

16. The window display method according to claim 14 or 15, characterized in that the method further includes: determining the display image transmission according to the Z-axis order of the window corresponding to the application being smaller than the Z-axis order of other windows of the terminal. The strategy is to transmit the image of the window in degraded quality or not to transmit the image of the window, or when the ratio of the covered area to the display area of the window corresponding to the application is greater than a certain threshold, it is determined that the display image transmission strategy is to transmit the image in degraded quality. an image of a window or not transmitting an image of said window;

Send the determination display image transfer policy to the server.

17. A terminal, characterized in that it includes:

The sending unit, while the virtual machine is running the application, sends the window information of the window corresponding to the application to the server, so that the virtual machine on the server determines the display area information of the window based on the window information;

The receiving unit is configured to receive image data corresponding to the display area information of the window sent by the server and display the image data.

18. The terminal according to claim 17, characterized in that, the terminal further includes a determining unit, and the determining unit specifically include:

Obtain sub-units, used to obtain the Z-axis sequence of all windows of the terminal, the (X, Y) coordinates of any corner point of all windows, and the width and height of all windows;

The first determination subunit is used to determine whether the ζ-axis sequence of the window corresponding to the application is the largest, and if so, the window display status of the window corresponding to the application is full display;

The second determination sub-unit is used when the first determination sub-unit determines that the ζ-axis order of the window corresponding to the application is not the maximum, based on the ζ-axis order greater than (Χ, any corner point of the window corresponding to the application), Υ) coordinates and the width and height of the window, determine whether the window corresponding to the application is covered by a window whose ζ-axis sequence is greater than the window corresponding to the application. If not, the window display status of the window corresponding to the application is full display, otherwise determine ζ The axis sequence is greater than the window corresponding to the application and covers the (X, Y) coordinates of any corner point of the window as well as the width and height of the window;

The third determination subunit is used to determine the (X, Y) coordinates of any corner point of the window that is larger than the window corresponding to the application in _Z- axis order and covers the window corresponding to the application, as well as the width and height of the window, and the window corresponding to the application. The (X, Y) coordinates of any corner point of the window and the width and height of the window, determine the (X, Y) coordinates of any corner point of the covered area of the window corresponding to the application and the width and height of the covered area;

The fourth determination subunit is used to determine that when the covered area of the window corresponding to the application completely covers the window corresponding to the application, it is determined that the window display status of the window corresponding to the application is fully covered, and the covered area of the window corresponding to the application is not completely covered. When covering the window corresponding to the application, it is determined that the window display status of the window corresponding to the application is partially covered.

19. The terminal according to claim 17 or 18, characterized in that,

The determination unit is further configured to determine that the display image transmission strategy is to transmit the image of the window with reduced quality or not to transmit the image of the window based on the Z-axis order of the window corresponding to the application being smaller than the Z-axis order of other windows of the terminal, Or when the ratio of the covered area to the display area of the window corresponding to the application is greater than a certain threshold, the display image transmission strategy is determined to be to transmit the image of the window with reduced quality or not to transmit the image of the window;

The sending unit is also used to send the determined display image transmission strategy to the server.

20. A window display system, characterized in that it includes the server according to any one of claims 8-13 and the terminal according to any one of claims 17-19.