CN116804907A - Touch screen redirection method and device, electronic equipment and storage medium - Google Patents
Touch screen redirection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116804907A CN116804907A CN202311032012.8A CN202311032012A CN116804907A CN 116804907 A CN116804907 A CN 116804907A CN 202311032012 A CN202311032012 A CN 202311032012A CN 116804907 A CN116804907 A CN 116804907A
- Authority
- CN
- China
- Prior art keywords
- touch
- touch screen
- data
- interface
- virtual machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000000875 corresponding effect Effects 0.000 claims abstract description 42
- 238000004891 communication Methods 0.000 claims abstract description 20
- 235000013599 spices Nutrition 0.000 claims description 41
- 230000004044 response Effects 0.000 claims description 15
- 238000004806 packaging method and process Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 description 15
- 230000015654 memory Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000004088 simulation Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000001276 controlling effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
- G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/131—Protocols for games, networked simulations or virtual reality
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The application provides a touch screen redirection method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: reading touch data of a touch screen; judging whether the operation display form of the current touch operation map meets a preset gesture or not through touch data; if yes, switching the current display interface to the interface to be displayed; if not, judging whether the current display interface is a virtual machine interface, if so, sending touch data to a server through a preset communication protocol so that the virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the specified local program in the thin client, so that the specified local program responds to the touch operation to execute corresponding actions. The application can solve the problem that the virtual machine interface and the thin client graphical interface cannot be freely switched when the touch screen device is used, so that the touch screen device can control both the virtual machine and the thin client and can be freely switched.
Description
Technical Field
The present application relates to the field of electronic digital data processing technologies, and in particular, to a touch screen redirection method, a device, an electronic device, and a storage medium.
Background
When the virtual cloud desktop is realized by using the SPICE protocol and the qemu+kvm technology by using the thin client, if the touch screen device is used as a display operation device of the thin client, the following technical problems are encountered when the touch screen directly redirects data into the virtual machine through the U port: the touch screen device is redirected to the virtual machine through the U port, and the universal USB protocol is used for directly interacting, so that the data of the touch screen device are not convenient to intercept and process.
Therefore, after the thin client is redirected to the virtual machine, the local graphical interface of the thin client cannot acquire the data of the touch screen device, and therefore when the graphical interface of the thin client needs to be operated, the touch screen device cannot correspondingly control the thin client. Corresponding solutions are lacking in the prior art.
Disclosure of Invention
In view of the above, embodiments of the present application provide a method, an apparatus, an electronic device, and a storage medium for redirecting a touch screen, which can solve the problem that a virtual machine interface and a thin client graphical interface cannot be freely switched when the touch screen device is used, so that the touch screen device can control both a virtual machine and a thin client, and can be switched at will.
The technical scheme of the embodiment of the application is realized as follows:
In a first aspect, an embodiment of the present application provides a touch screen redirection method, including the following steps:
responding to touch operation on a touch screen, and reading touch data of the touch screen, wherein the touch screen is connected with a thin client through a USB interface;
judging whether the operation display form of the current touch operation mapping meets a preset gesture or not through the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
if yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
In one possible implementation, the preset communication protocol is an extended SPICE protocol, the extended SPICE protocol is used for packaging the touch data, and the extended SPICE protocol is established by the following ways:
defining a target command and a target data structure in SPICE protocol, wherein the target command is used for identifying and sending the target data structure, and the target data structure comprises the touch data;
the SPICE protocol defining the target command and the target data structure is referred to as the extended SPICE protocol.
In a possible implementation manner, a touch screen simulation module is preset in the server, the touch screen simulation module simulates the touch screen based on a USB device frame in the QEMU, a touch screen driver is preset in the virtual machine, and the touch data is sent to the server through a preset communication protocol, so that the virtual machine in the server can perform corresponding actions based on the touch data, and the method includes:
packaging the touch data through a SPICE client in the thin client to obtain packaged data;
sending the encapsulated data to a SPICE server in the server through an extended SPICE protocol;
Analyzing the encapsulated data through the simulated touch screen module to obtain a touch instruction;
executing the touch screen instruction through the touch screen drive, so that the virtual machine performs corresponding actions based on the touch screen instruction.
In one possible implementation manner, before the touch data of the touch screen is read in response to a touch operation on the touch screen, the method further includes:
in response to a start operation of the virtual machine, mounting the simulated touch screen module into the server so that the virtual machine can identify the touch screen;
and loading the touch screen driver so that the virtual machine can interact with the simulated touch screen module.
In one possible implementation manner, before the touch data of the touch screen is read in response to a touch operation on the touch screen, the method further includes:
judging the type of equipment connected with the USB interface through the thin client;
and if the equipment type is the touch screen, intercepting the touch data so as to enable the thin client to forward the touch data.
In a possible implementation manner, determining, according to the touch data, whether the operation presentation form of the current touch operation map meets a preset gesture includes:
And comparing the touch data with gesture data corresponding to the preset gesture, and determining the operation display form of the current touch operation map as the preset gesture if the similarity of the touch data and the gesture data is greater than or equal to a preset matching threshold value.
In a possible implementation manner, the touch screen is provided with a multi-touch function, and the method further includes:
responding to multi-touch operation aiming at the touch screen, and reading multi-touch data of the touch screen;
and operating a local interface and a local program of the virtual machine and/or the thin client based on the multi-touch data.
In a second aspect, an embodiment of the present application further provides a touch screen redirecting device, where the device includes:
the touch screen comprises a reading module, a touch screen and a display module, wherein the reading module is used for responding to touch operation of the touch screen and reading touch data of the touch screen, and the touch screen is connected with a thin client through a USB interface;
the judging module is used for judging whether the operation display form of the current touch operation mapping meets a preset gesture or not according to the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
If yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a storage medium, and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor in communication with the storage medium via the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the touch screen redirection method of any of the first aspects.
In a fourth aspect, embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the touch screen redirection method of any of the first aspects.
The embodiment of the application has the following beneficial effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of steps S101-S102 provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart of steps S201-S202 provided in an embodiment of the present application;
FIG. 3 is a schematic flow chart of steps S301-S304 provided in an embodiment of the present application;
FIG. 4 is a schematic flow chart of steps S401-S402 provided by an embodiment of the present application;
FIG. 5 is a schematic flow chart of steps S501-S502 provided in an embodiment of the present application;
FIG. 6 is a flow chart of steps S601-S602 provided by an embodiment of the present application;
FIG. 7 is a schematic block diagram of a touch screen redirection provided by an embodiment of the present application;
FIG. 8 is a touch screen redirection flowchart provided by an embodiment of the present application;
fig. 9 is a schematic structural diagram of a touch screen redirecting device according to an embodiment of the present application;
fig. 10 is a schematic diagram of a composition structure of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for the purpose of illustration and description only and are not intended to limit the scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this disclosure, illustrates operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to or removed from the flow diagrams by those skilled in the art under the direction of the present disclosure.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
In addition, the described embodiments are only some, but not all, embodiments of the application. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present application.
In the following description, the terms "first", "second", "third" and the like are merely used to distinguish similar objects and do not represent a particular ordering of the objects, it being understood that the "first", "second", "third" may be interchanged with a particular order or sequence, as permitted, to enable embodiments of the application described herein to be practiced otherwise than as illustrated or described herein.
It should be noted that the term "comprising" will be used in embodiments of the application to indicate the presence of the features stated hereafter, but not to exclude the addition of other features.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the application and is not intended to be limiting of the application.
Before describing embodiments of the present application in further detail, the terms and terminology involved in the embodiments of the present application will be described, and the terms and terminology involved in the embodiments of the present application will be used in the following explanation.
QEMU: is a universal open source computer emulator and virtualizer.
SPICE: is a virtual desktop transport protocol.
KVM: is an open-source system virtual machine module.
Redirecting: in a private cloud desktop, a device of a client is transferred to a virtual machine to which it is being connected through a network.
Thin client: is a desktop computer without a hard disk supported by a thin client operating system rather than other operating systems such as Windows or Linux. These fanless computers do not store sensitive data and applications locally, but rather store them locally or in the cloud.
Referring to fig. 1, fig. 1 is a schematic flow chart of steps S101 to S102 of a touch screen redirecting method according to an embodiment of the present application, and will be described with reference to steps S101 to S102 shown in fig. 1.
Step S101, responding to touch operation of a touch screen, and reading touch data of the touch screen, wherein the touch screen is connected with a thin client through a USB interface;
step S102, judging whether the operation display form of the current touch operation mapping meets a preset gesture or not according to the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
if yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
The touch screen redirection method has the following beneficial effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
The above-described exemplary steps of the embodiments of the present application are described below, respectively.
In step S101, touch data of a touch screen is read in response to a touch operation on the touch screen, wherein the touch screen is connected with a thin client through a USB interface.
In step S102, judging whether the operation display form of the current touch operation map meets a preset gesture according to the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
If yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
Referring to fig. 7 and 8, fig. 7 is a schematic block diagram of a touch screen redirection provided by an embodiment of the present application, and fig. 8 is a flow chart of a touch screen redirection provided by an embodiment of the present application, as shown in fig. 7, the system is divided into a server Host and a thin client, the thin client is externally connected with a physical touch screen device through a USB interface, a touch screen reading module and a gesture processing and forwarding module are disposed in the thin client, the gesture processing and forwarding module can directly control a local graphical interface (which can be used for switching a display interface or operating a local program), or after switching to a virtual machine interface, touch data is sent to a SPICE client in the server Host through an extended SPICE protocol, and then an application in a client OS system (i.e. a virtual machine) is operated through a simulated touch screen module and a touch screen driver.
As shown in fig. 8, after the touch screen data (i.e., the touch data) is read, gesture determination is performed, and if the gesture is a gesture (the gesture refers to "switching the display interface", for example, sliding rightward), a corresponding action is performed to switch the display interface; if not, judging whether the virtual machine interface is the virtual machine interface again, if so, controlling the virtual machine OS (namely the client OS system) through simulating the touch screen module, and if not, considering the virtual machine interface as an operation on the local interface, correspondingly, sending touch data to a local graphical interface receiving module (namely the graphical interface in fig. 7), and executing corresponding graphical interface control.
In some embodiments, referring to fig. 2, fig. 2 is a schematic flow chart of steps S201-S202 provided in the embodiment of the present application, where the preset communication protocol is an extended SPICE protocol, and the extended SPICE protocol is used for packaging the touch data, and the extended SPICE protocol is established through the steps S201-S202, and will be described in connection with the steps.
In step S201, a target command and a target data structure are defined in SPICE protocol, wherein the target command is used for identifying and transmitting the target data structure, and the target data structure includes the touch data.
In step S202, the SPICE protocol defining the target command and the target data structure is taken as the extended SPICE protocol.
Here, a new command is defined in SPICE protocol, and a new data structure enables transmission of encapsulated touch data between thin client and server Host through Input channel. For example:
new command Id: spice_msgc_input_touch_action (115).
New data structure:
here, a unit indicates an unsigned integer, a number after the unit indicates a number of bits, for example, 32 indicates 32 bits, num_of_actions indicates the number of operations of a touch operation, data_size indicates the size of touch data, data indicates specific touch data (touch data), touch indicates whether or not touch is made for specific touch data (touch data) (1 indicates yes, 0 indicates no), valid indicates whether or not valid (1 indicates yes, 0 indicates no), range indicates whether or not in range (1 indicates yes, 0 indicates no), reserve indicates a reserved number, id is id of the touch data, x indicates a touch value on the x axis, and y indicates a touch value on the y axis.
In some embodiments, referring to fig. 3, fig. 3 is a schematic flow chart of steps S301 to S304 provided by the embodiment of the present application, a touch screen simulation module is preset in the server, the touch screen simulation module simulates the touch screen based on a USB device frame in a QEMU, a touch screen driver is preset in the virtual machine, and the touch data is sent to the server through a preset communication protocol, so that the virtual machine in the server can perform corresponding actions based on the touch data, and the steps S301 to S304 can be implemented, which will be described in connection with each step.
In step S301, packaging processing is performed on the touch data by the SPICE client in the thin client, so as to obtain packaged data.
In step S302, the encapsulated data is sent to a SPICE server in the server through an extended SPICE protocol.
In step S303, the encapsulated data is analyzed by the simulated touch screen module, so as to obtain a touch instruction.
In step S304, the touch screen command is executed through the touch screen driver, so that the virtual machine performs a corresponding action based on the touch screen command.
As shown in FIG. 7, the encapsulated data is sent to the SPICE server by extending the SPICE protocol, and then the analog touch screen module can read the data and then pass it to the touch screen driver, thereby performing the corresponding actions in the guest OS system (virtual machine).
In some embodiments, referring to fig. 4, fig. 4 is a schematic flow chart of steps S401 to S402 provided in the embodiment of the present application, where steps S401 to S402 may be further performed before touch data of a touch screen is read in response to a touch operation on the touch screen, and will be described with reference to specific steps.
In step S401, in response to the start operation of the virtual machine, the simulated touch screen module is mounted in the server, so that the virtual machine can recognize the touch screen.
In step S402, the touch screen driver is loaded to enable the virtual machine to interact with the simulated touch screen module.
Here, the HID touch screen device based on the standard is simulated by the USB device frame in the QEMU (i.e. the simulated touch screen module is obtained), and the HID touch screen device is mounted in the system when the virtual machine is started, so that the virtual machine can identify the simulated touch screen, and automatically load the standard touch screen drive to interact with the simulated touch screen module.
In some embodiments, referring to fig. 5, fig. 5 is a schematic flow chart of steps S501 to S502 provided in the embodiment of the present application, where steps S501 to S502 may be further performed before touch data of a touch screen is read in response to a touch operation on the touch screen, and specific steps will be described in connection with the steps.
In step S501, the device type connected to the USB interface is determined by the thin client.
In step S502, if the device type is the touch screen, the touch data is intercepted, so that the thin client forwards the touch data.
The thin client intercepts devices such as a touch screen by judging the type of the USB specific device, so that the thin client reads local data of the touch screen, and then forwards the touch data according to the current display content.
In some embodiments, determining, according to the touch data, whether the operation presentation form of the current touch operation map satisfies a preset gesture includes:
and comparing the touch data with gesture data corresponding to the preset gesture, and determining the operation display form of the current touch operation map as the preset gesture if the similarity of the touch data and the gesture data is greater than or equal to a preset matching threshold value.
Here, the preset gesture may be a screen switch gesture, a page-turning gesture, or an zoom-in/zoom-out gesture, where the preset gesture corresponds to a set of standard gesture data, and when it is determined whether the operation display form of the current touch operation map satisfies the preset gesture, the standard gesture data is compared with the touch data to determine whether the current touch operation satisfies the corresponding preset gesture, for example, the similarity between the gesture data and the touch data is greater than 90%, and the operation display form of the current touch operation map is considered to satisfy the preset gesture.
In some embodiments, the touch data includes X-axis data and Y-axis data, and the gesture is defined by the following rules:
detecting the X-axis data and the Y-axis data in the touch data;
And if the fluctuation range of the Y-axis data is smaller than a preset fluctuation threshold value and the X-axis data is in an increasing state, determining the touch data as the gesture.
Here, operating the touch screen device, the touch screen reading module in the thin client can read corresponding touch data, wherein the touch data consists of some point coordinates, pressing marks, data point ids and other information;
the specific gesture is formed by a set of regular touch data, multiple sets of touch data are recorded, then the rules of the gesture are compared, and if all the rules are met, a gesture, such as a right sliding gesture, can be understood as a set of touch data with Y values being basically equal and X values being increased.
In some embodiments, referring to fig. 6, fig. 6 is a schematic flow chart of steps S601 to S602 provided in the embodiment of the present application, the touch screen has a multi-touch function, and the method may further execute the steps S601 to S602, which will be described with reference to specific steps.
In step S601, in response to a multi-touch operation for the touch screen, multi-touch data of the touch screen is read.
In step S602, a local interface and a local program of the virtual machine and/or the thin client are operated based on the multi-touch data.
Here, since the analog touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
In summary, the embodiment of the application has the following beneficial effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
Based on the same inventive concept, the embodiment of the present application further provides a touch screen redirecting device corresponding to the touch screen redirecting method in the first embodiment, and since the principle of solving the problem of the device in the embodiment of the present application is similar to that of the above touch screen redirecting method, implementation of the device may refer to implementation of the method, and repeated parts are not repeated.
As shown in fig. 9, fig. 9 is a schematic structural diagram of a touch screen redirecting device 900 according to an embodiment of the present application. The touch screen redirection device 900 includes:
the reading module 901 is configured to read touch data of a touch screen in response to a touch operation on the touch screen, where the touch screen is connected to a thin client through a USB interface.
A judging module 902, configured to judge, according to the touch data, whether an operation display form of the current touch operation map meets a preset gesture; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
if yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
Those skilled in the art will appreciate that the implementation functions of the units in the touch screen redirection device 900 shown in fig. 9 may be understood with reference to the foregoing description of the touch screen redirection method. The functions of the units in the touch screen redirection device 900 shown in fig. 9 may be implemented by a program running on a processor, or may be implemented by specific logic circuits.
In one possible implementation, the preset communication protocol is an extended SPICE protocol, which is used to encapsulate the touch data, and the determining module 902 establishes the extended SPICE protocol by:
defining a target command and a target data structure in SPICE protocol, wherein the target command is used for identifying and sending the target data structure, and the target data structure comprises the touch data;
the SPICE protocol defining the target command and the target data structure is referred to as the extended SPICE protocol.
In a possible implementation manner, a touch screen simulation module is preset in the server, the touch screen simulation module simulates the touch screen based on a USB device frame in the QEMU, a touch screen driver is preset in the virtual machine, and the judging module 902 sends the touch data to the server through a preset communication protocol, so that the virtual machine in the server can perform corresponding actions based on the touch data, including:
Packaging the touch data through a SPICE client in the thin client to obtain packaged data;
sending the encapsulated data to a SPICE server in the server through an extended SPICE protocol;
analyzing the encapsulated data through the simulated touch screen module to obtain a touch instruction;
executing the touch screen instruction through the touch screen drive, so that the virtual machine performs corresponding actions based on the touch screen instruction.
In a possible implementation manner, before the reading module 901 reads touch data of the touch screen in response to a touch operation on the touch screen, the method further includes:
in response to a start operation of the virtual machine, mounting the simulated touch screen module into the server so that the virtual machine can identify the touch screen;
and loading the touch screen driver so that the virtual machine can interact with the simulated touch screen module.
In a possible implementation manner, before the reading module 901 reads touch data of the touch screen in response to a touch operation on the touch screen, the method further includes:
judging the type of equipment connected with the USB interface through the thin client;
And if the equipment type is the touch screen, intercepting the touch data so as to enable the thin client to forward the touch data.
In a possible implementation manner, the determining module 902 determines, according to the touch data, whether the operation presentation form of the current touch operation map meets a preset gesture, including:
and comparing the touch data with gesture data corresponding to the preset gesture, and determining the operation display form of the current touch operation map as the preset gesture if the similarity of the touch data and the gesture data is greater than or equal to a preset matching threshold value.
In a possible implementation manner, the touch screen is provided with a multi-touch function, and the judging module 902 further includes:
responding to multi-touch operation aiming at the touch screen, and reading multi-touch data of the touch screen;
and operating a local interface and a local program of the virtual machine and/or the thin client based on the multi-touch data.
The touch screen redirection device has the following beneficial effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
As shown in fig. 10, fig. 10 is a schematic diagram of a composition structure of an electronic device 1000 according to an embodiment of the present application, where the electronic device 1000 includes:
a processor 1001, a storage medium 1002 and a bus 1003, wherein the storage medium 1002 stores machine readable instructions executable by the processor 1001, when the electronic device 1000 is running, the processor 1001 communicates with the storage medium 1002 through the bus 1003, and the processor 1001 executes the machine readable instructions to perform the steps of the touch screen redirection method according to the embodiment of the present application.
In actual use, the various components of the electronic device 1000 are coupled together via the bus 1003. It is understood that the bus 1003 is used to enable connected communication between these components. The bus 1003 includes a power bus, a control bus, and a status signal bus in addition to a data bus. But for clarity of illustration, the various buses are labeled as bus 1003 in fig. 10.
The electronic equipment has the following beneficial effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
The embodiment of the application also provides a computer readable storage medium, which stores executable instructions that, when executed by the at least one processor 1001, implement the touch screen redirection method according to the embodiment of the application.
In some embodiments, the storage medium may be a magnetic random Access Memory (FRAM, ferromagneticRandom Access Memory), read Only Memory (ROM), programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read Only Memory (EPROM, erasableProgrammable Read-Only Memory), electrically erasable programmable Read Only Memory (EEPROM, electricallyErasable Programmable Read-Only Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk Read Only Memory (CD-ROM, compact Disc Read-Only Memory), or the like; but may be a variety of devices including one or any combination of the above memories.
In some embodiments, the executable instructions may be in the form of programs, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and they may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.
As an example, the executable instructions may, but need not, correspond to files in a file system, may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a hypertext markup Language (HTML, hyperTextMarkup Language) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
As an example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices located at one site or, alternatively, distributed across multiple sites and interconnected by a communication network.
The above computer-readable storage medium has the following advantageous effects:
1. the local graphical interface and the remote virtual machine interface can be freely switched and controlled through the touch screen, and an external mouse or a keyboard is not needed.
2. Since a general touch screen device is simulated by software, various virtual machines can be compatible.
3. Support multi-touch operation: as the simulated touch screen supports the transmission of multi-touch data, the corresponding specific content is that the device descriptor marks the multi-touch device, and meanwhile, the data supports batch transmission. Therefore, the physical touch screen accessed in the way supports multiple points, and the multi-point touch operation is supported after redirection.
In the several embodiments provided in the present application, it should be understood that the disclosed method and electronic device may be implemented in other manners. The above described device embodiments are only illustrative, e.g. the division of the units is only one logical function division, and there may be other divisions in practice, such as: multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily appreciate variations or alternatives within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. A method for redirecting a touch screen, comprising the steps of:
responding to touch operation on a touch screen, and reading touch data of the touch screen, wherein the touch screen is connected with a thin client through a USB interface;
judging whether the operation display form of the current touch operation mapping meets a preset gesture or not through the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
if yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
2. The method of claim 1, wherein the pre-set communication protocol is an extended SPICE protocol for encapsulating the touch data, the extended SPICE protocol established by:
defining a target command and a target data structure in SPICE protocol, wherein the target command is used for identifying and sending the target data structure, and the target data structure comprises the touch data;
the SPICE protocol defining the target command and the target data structure is referred to as the extended SPICE protocol.
3. The method according to claim 2, wherein a simulated touch screen module is preset in the server, the simulated touch screen module simulates the touch screen based on a USB device frame in the QEMU, a touch screen driver is preset in the virtual machine, and the touch data is sent to the server through a preset communication protocol, so that the virtual machine in the server can perform corresponding actions based on the touch data, including:
packaging the touch data through a SPICE client in the thin client to obtain packaged data;
Sending the encapsulated data to a SPICE server in the server through an extended SPICE protocol;
analyzing the encapsulated data through the simulated touch screen module to obtain a touch instruction;
executing the touch screen instruction through the touch screen drive, so that the virtual machine performs corresponding actions based on the touch screen instruction.
4. A method according to claim 3, wherein, in response to a touch operation on a touch screen, before reading touch data of the touch screen, the method further comprises:
in response to a start operation of the virtual machine, mounting the simulated touch screen module into the server so that the virtual machine can identify the touch screen;
and loading the touch screen driver so that the virtual machine can interact with the simulated touch screen module.
5. The method of claim 1, wherein prior to reading touch data for a touch screen in response to a touch operation for the touch screen, the method further comprises:
judging the type of equipment connected with the USB interface through the thin client;
and if the equipment type is the touch screen, intercepting the touch data so as to enable the thin client to forward the touch data.
6. The method of claim 1, wherein determining, from the touch data, whether an operation presentation morphology of a current touch operation map satisfies a preset gesture comprises:
and comparing the touch data with gesture data corresponding to the preset gesture, and determining the operation display form of the current touch operation map as the preset gesture if the similarity of the touch data and the gesture data is greater than or equal to a preset matching threshold value.
7. The method of claim 1, wherein the touch screen is multi-touch enabled, the method further comprising:
responding to multi-touch operation aiming at the touch screen, and reading multi-touch data of the touch screen;
and operating a local interface and a local program of the virtual machine and/or the thin client based on the multi-touch data.
8. A touch screen redirection device, the device comprising:
the touch screen comprises a reading module, a touch screen and a display module, wherein the reading module is used for responding to touch operation of the touch screen and reading touch data of the touch screen, and the touch screen is connected with a thin client through a USB interface;
the judging module is used for judging whether the operation display form of the current touch operation mapping meets a preset gesture or not according to the touch data; the preset gesture is used for switching the display interface of the thin client; the display interface comprises a local interface or a virtual machine interface, and the virtual machine interface is a cloud desktop;
If yes, determining an interface to be displayed according to the current display interface of the touch screen, and switching the current display interface into the interface to be displayed;
if not, judging whether the current display interface is a virtual machine interface, if so, sending the touch data to a server through a preset communication protocol, so that a virtual machine in the server responds to the touch operation to execute corresponding actions; if not, the touch data is sent to the local program appointed in the thin client machine, so that the appointed local program responds to the touch operation to execute corresponding actions.
9. An electronic device, comprising: a processor, a storage medium, and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor in communication with the storage medium via the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the touch screen redirection method of any of claims 1 to 7.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, performs the touch screen redirection method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311032012.8A CN116804907A (en) | 2023-08-16 | 2023-08-16 | Touch screen redirection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311032012.8A CN116804907A (en) | 2023-08-16 | 2023-08-16 | Touch screen redirection method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116804907A true CN116804907A (en) | 2023-09-26 |
Family
ID=88079620
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311032012.8A Pending CN116804907A (en) | 2023-08-16 | 2023-08-16 | Touch screen redirection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116804907A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105378634A (en) * | 2013-05-17 | 2016-03-02 | 思杰系统有限公司 | Remoting or localizing touch gestures at a virtualization client agent |
| US20190129680A1 (en) * | 2017-11-01 | 2019-05-02 | American Megatrends, Inc. | Techniques of remotely providing user input to thin client |
| CN113973133A (en) * | 2021-09-30 | 2022-01-25 | 西安万像电子科技有限公司 | USB equipment redirection method and device, electronic equipment and storage medium |
| CN116107648A (en) * | 2022-12-28 | 2023-05-12 | 湖南麒麟信安科技股份有限公司 | Physical CD driver redirection method, system and device in cloud desktop environment |
-
2023
- 2023-08-16 CN CN202311032012.8A patent/CN116804907A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105378634A (en) * | 2013-05-17 | 2016-03-02 | 思杰系统有限公司 | Remoting or localizing touch gestures at a virtualization client agent |
| US20190129680A1 (en) * | 2017-11-01 | 2019-05-02 | American Megatrends, Inc. | Techniques of remotely providing user input to thin client |
| CN113973133A (en) * | 2021-09-30 | 2022-01-25 | 西安万像电子科技有限公司 | USB equipment redirection method and device, electronic equipment and storage medium |
| CN116107648A (en) * | 2022-12-28 | 2023-05-12 | 湖南麒麟信安科技股份有限公司 | Physical CD driver redirection method, system and device in cloud desktop environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101620058B1 (en) | Apparatus for switching screen between virtual machines and method thereof | |
| CN106843715B (en) | Touch support for remoted applications | |
| KR101751936B1 (en) | Apparatus and method for virtualization of i/o device using shared memory in host-based terminal virtualization environment | |
| CN102207886B (en) | Virtual machine high-speed simulation is assisted | |
| US7698706B2 (en) | Methods and apparatus for implementing an integrated user interface for managing multiple virtual machines operative in a computing system | |
| US8966477B2 (en) | Combined virtual graphics device | |
| CN107977141B (en) | Interaction control method and device, electronic equipment and storage medium | |
| US20100319008A1 (en) | Parallel processing method for dual operating system | |
| CN107329690B (en) | Virtual object control method and device, storage medium and electronic equipment | |
| WO2012047997A1 (en) | Display management for native user experiences | |
| CN108776544B (en) | Interaction method and device in augmented reality, storage medium and electronic equipment | |
| WO2016061018A1 (en) | Input signal emulation | |
| US10311715B2 (en) | Smart device mirroring | |
| WO2014172710A1 (en) | Methods, systems, and media for binary compatibility | |
| US20080215852A1 (en) | System and Device Architecture For Single-Chip Multi-Core Processor Having On-Board Display Aggregator and I/O Device Selector Control | |
| US20210326151A1 (en) | Methods, devices and computer-readable storage media for processing a hosted application | |
| EP3633533B1 (en) | Electronic apparatus and controlling method thereof | |
| CN112579187A (en) | Optimization method and device for cold start of application program | |
| CN109542323B (en) | Interaction control method and device based on virtual scene, storage medium and electronic equipment | |
| CN114691300A (en) | Hot migration method of virtual machine instance | |
| CN110516447A (en) | A kind of method and apparatus of identification terminal simulator | |
| CN110597597B (en) | Method, system, device and storage medium for virtualization of hardware | |
| CN113868174B (en) | Verification platform building method and device and storage medium | |
| CN105051714A (en) | System and method of web-based virtual media redirection | |
| US20130030786A1 (en) | Emulating input/output components |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |