WO2021248788A1 - 一机多控方法、装置、系统及电子设备 - Google Patents

一机多控方法、装置、系统及电子设备 Download PDF

Info

Publication number
WO2021248788A1
WO2021248788A1 PCT/CN2020/125574 CN2020125574W WO2021248788A1 WO 2021248788 A1 WO2021248788 A1 WO 2021248788A1 CN 2020125574 W CN2020125574 W CN 2020125574W WO 2021248788 A1 WO2021248788 A1 WO 2021248788A1
Authority
WO
WIPO (PCT)
Prior art keywords
control terminal
slave
master
toolkit
terminal
Prior art date
Application number
PCT/CN2020/125574
Other languages
English (en)
French (fr)
Inventor
余泽希
刘洪歧
Original Assignee
网易（杭州）网络有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 网易（杭州）网络有限公司 filed Critical 网易（杭州）网络有限公司
Priority to US18/001,077 priority Critical patent/US20230214243A1/en
Publication of WO2021248788A1 publication Critical patent/WO2021248788A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/452—Remote windowing, e.g. X-Window System, desktop virtualisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03547—Touch pads, in which fingers can move on a surface
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033—Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038—Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • G06F3/0383—Signal control means within the pointing device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038—Indexing scheme relating to G06F3/038
    • G06F2203/0383—Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/041—Indexing scheme relating to G06F3/041 - G06F3/045
    • G06F2203/04104—Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048—Indexing scheme relating to G06F3/048
    • G06F2203/04808—Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/54—Indexing scheme relating to G06F9/54
    • G06F2209/545—Gui

Definitions

  • the present disclosure relates to the technical field of intelligent control, and in particular to a method, device, system and electronic equipment for one machine with multiple controls.
  • each terminal brand also has many different models of terminals.
  • each brand will launch new products at regular intervals, which will result in hundreds of Android system terminals in the market.
  • Kind of, the models, resolutions, and system versions of these terminals are different, which brings a lot of trouble to mobile application and game development and testing personnel.
  • the purpose of the present disclosure is to provide a one-machine multi-control method, device, system and electronic equipment to alleviate the above technical problems.
  • the embodiments of the present disclosure provide a one-machine multi-control method, which is applied to a master control terminal, a master toolkit is deployed on the master control terminal, the master control terminal is connected to at least one slave control terminal, and the master control terminal Both the slave control terminal and the slave control terminal are mobile terminals installed with the Android system, and a slave toolkit is deployed on the slave control terminal.
  • the method includes: responding to the touch operation of the graphical user interface of the master control terminal, and performing the touch operation through the master toolkit The corresponding operation event is converted into an operation protocol package; a message is sent to the slave control terminal, and the message carries the operation protocol package to trigger the slave control terminal to perform a touch operation based on the operation protocol package through the slave tool package.
  • the above method further includes: in response to a start instruction for the main toolkit, monitoring the touch operation of the graphical user interface of the main control terminal.
  • the step of monitoring the touch operation of the graphical user interface of the main control terminal includes: monitoring the character device file of the main control terminal; if the character device file changes, determining the control terminal's A touch operation occurs on the graphical user interface.
  • the step of converting the operation event corresponding to the touch operation into an operation protocol package through the main toolkit includes: determining whether the multi-touch protocol type of the main control terminal is the second protocol type; if Yes, the operation protocol package is generated based on the main toolkit and the operation event corresponding to the touch operation, where the second protocol type is a stateful multi-touch protocol type.
  • the step of converting the operation event corresponding to the touch operation into an operation protocol package through the master toolkit further includes: if the multi-touch protocol type of the master control terminal is the first protocol type , Convert the operation event corresponding to the touch operation into the operation event corresponding to the second protocol type; generate an operation protocol package based on the main toolkit and the operation event corresponding to the second protocol type, where the first protocol type is stateless multipoint Touch protocol type.
  • the step of generating the operation protocol package based on the operation event corresponding to the master toolkit and the touch operation includes: extracting the operation event from the character device file of the master control terminal through the master toolkit; The coordinate points and pressure values contained in the received operation event are normalized; an operation protocol package is generated according to the normalized operation event.
  • the above-mentioned message sent to the slave control terminal is a ZeroMQ message carrying an operation protocol packet.
  • the embodiments of the present disclosure provide a one-machine multi-control method, which is applied to a slave terminal.
  • a slave toolkit is deployed on the slave terminal.
  • the slave terminal is connected to the master terminal.
  • the master terminal and the slave terminal The control terminals are all mobile terminals installed with the Android system, and the main toolkit is deployed on the main control terminal.
  • the method includes: receiving a message sent by the main control terminal; wherein, the message carries the corresponding touch operation on the main control terminal Operation protocol package; parse out the operation protocol package from the message; simulate operation on the graphic user interface of the slave control terminal based on the operation protocol package.
  • the step of performing a simulation operation on the graphical user interface of the slave terminal based on the operation protocol package includes: extracting the operation event carried in the operation protocol package, and writing the operation event to the slave terminal
  • the character device file is used to update the character device file of the slave control terminal; based on the slave toolkit and the updated character device file, the graphical user interface of the slave control terminal is simulated.
  • the step of extracting the operation event carried in the operation protocol packet includes: obtaining the multi-touch protocol type of the slave terminal; if the multi-touch protocol type is the second protocol type, then The operation event corresponding to the second protocol type is extracted from the operation protocol packet, where the second protocol type is a stateful multi-touch protocol type.
  • the step of extracting the operation event carried in the operation protocol packet further includes: if the multi-touch protocol type is the first protocol type, extracting the first protocol type corresponding to the operation protocol packet The operation event of, wherein the first protocol type is a stateless multi-touch protocol type.
  • the step of performing a simulation operation on the graphical user interface of the slave control terminal based on the operation protocol package further includes: if the version of the Android system of the slave control terminal is an operating system that satisfies the preset version, obtaining The rotation direction of the graphic user interface of the slave control terminal; according to the rotation direction, the received operation protocol packet is converted to coordinates, and the preset application program is called to perform the conversion of the operation protocol package on the graphic user interface of the slave control terminal. Simulation operation.
  • the above method further includes: in response to a start instruction directed to the slave toolkit, monitoring the message sent by the master control terminal.
  • the received message sent by the master control terminal is a ZeroMQ message carrying an operation protocol packet.
  • the embodiments of the present disclosure provide a one-machine multi-control device, which is applied to a master control terminal.
  • a master toolkit is deployed on the master control terminal.
  • the master control terminal is connected to at least one slave control terminal.
  • Both the slave control terminal and the slave control terminal are mobile terminals installed with the Android system, and the slave tool kit is deployed on the slave control terminal.
  • the package converts the operation event corresponding to the touch operation into an operation protocol package; the sending module is used to send a message to the slave control terminal, and the message carries the operation protocol package to trigger the slave control terminal to execute the touch operation based on the operation protocol package through the slave tool kit. Control operation.
  • the embodiments of the present disclosure provide a one-machine multi-control device, which is applied to a slave control terminal.
  • a slave toolkit is deployed on the slave control terminal.
  • the slave control terminal is connected to the master control terminal.
  • the master control terminal and the slave control terminal The control terminals are all mobile terminals installed with the Android system.
  • the main toolkit is deployed on the main control terminal.
  • the device includes: a receiving module for receiving a message sent by the main control terminal; wherein, the message carries the message on the main control terminal.
  • the operation protocol package corresponding to the touch operation; the analysis module is used to parse the operation protocol package from the message; the simulation operation module is used to simulate the operation on the graphical user interface of the slave terminal based on the operation protocol package.
  • the embodiments of the present disclosure provide a one-machine multi-control system, the system includes a service device, and also includes a master terminal and a slave terminal, wherein the service device is in communication connection with the master terminal and the slave terminal;
  • the master terminal is configured with the device described in the third aspect, and the slave terminal is configured with the device described in the fourth aspect;
  • the service device is used to push the master toolkit to the master control terminal and the slave toolkit to the slave control terminal; and start the master toolkit and the slave toolkit to realize the slave control terminal Simultaneously simulate the touch operation of the master terminal.
  • the embodiments of the present disclosure provide an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor.
  • the processor implements the computer program when the computer program is executed. The method described in the first to second aspects above.
  • embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and when the computer program is run by a processor, the above-mentioned first to second aspects are executed method.
  • the one-machine multi-control method, device, system and electronic equipment can respond to the touch operation of the graphical user interface of the master control terminal and convert the operation event corresponding to the touch operation into Operation protocol package; and then send a message carrying the operation protocol package to the slave control terminal to trigger the slave control terminal to perform touch operations based on the operation protocol package.
  • the master control terminal and the slave control terminal directly pass through the master toolkit Establish a communication connection with the slave toolkit. Therefore, no additional hardware cost is required, and the deployment is simple and convenient.
  • the master toolkit when the master toolkit is deployed on the master terminal, there is no need to restrict the model, resolution, system version, etc. of the master terminal. Therefore, it has better expansibility and compatibility, avoiding testers to test separately on different versions, effectively reducing labor costs, and at the same time improving the efficiency of testing.
  • FIG. 1 is a flowchart of a one-machine multi-control method provided by an embodiment of the present disclosure
  • FIG. 2 is a schematic diagram of the connection of a one-machine multi-control solution provided by an embodiment of the present disclosure
  • FIG. 3 is a schematic diagram of an operation protocol package provided by an embodiment of the disclosure.
  • FIG. 5 is a flowchart of another one-machine multi-control method provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of a one-machine multi-control device provided by an embodiment of the disclosure.
  • FIG. 7 is a schematic structural diagram of another one-machine multi-control device provided by an embodiment of the present disclosure.
  • FIG. 8 is a schematic structural diagram of another one-machine multi-control device provided by an embodiment of the disclosure.
  • FIG. 9 is a schematic structural diagram of another one-machine multi-control device provided by an embodiment of the disclosure.
  • FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the disclosure.
  • the current one-machine multi-control solutions mainly include the Solopi solution and the Total Control solution.
  • the design and implementation of the Solopi solution is based on the recording and playback function.
  • the user is recorded on one device first. Then send the user's operation record to any other device through the socket to realize the playback of the operation, so as to achieve the effect of controlling multiple devices through one device.
  • the master control machine needs to establish a Socket connection with the slave control machine, and then send the user's operation record to each slave control machine in real time on the master control machine, and the slave control machine will analyze the operation record after receiving the operation record, and then on the slave control machine Simulate the user's operation to complete the playback of the operation record.
  • the Total Control solution is to realize the interconnection and intercommunication between the computer and the terminal through USB/WIFI, and the terminal is controlled uniformly on the computer side.
  • the hub is also connected to the computer through a USB cable, and then the corresponding client is installed on the terminal and the computer, and the terminal screen is mapped to the computer, and then the terminal is operated on the computer through the mouse to achieve the effect of multi-control.
  • the Solopi scheme is based on the principle of recording and playback. Therefore, the operation delay is relatively large. There will be no response from the host computer. Only after the master control machine completes an operation, the operation record is sent to the slave control machine. After the slave control machine is parsed, there will be a corresponding response. The whole process takes a long time, so for some complicated For multi-touch operations, the resolution is slower and the delay will be greater, which will seriously affect the actual use effect and user experience. Moreover, the number of devices that can be supported is limited, which also seriously affects the use of large-scale device clusters.
  • the Total Control solution requires external hardware costs, including computers, HUB hubs, USB cables and other hardware. As the number of terminals increases, the computer configuration required is better. At the same time, the compatibility between the terminal and the computer There are also certain requirements. Furthermore, because the multi-control solution uses a computer to perform multi-machine operations, and a computer mouse can only simulate some single-touch operations and cannot simulate multi-touch operations, this solution does not support multi-touch operations. For some operations that require multi-touch, it cannot be tested.
  • the one-machine multi-control method, device, system and electronic equipment provided by the embodiments of the present disclosure can effectively alleviate the above-mentioned technical problems.
  • an embodiment of the present disclosure provides a one-machine multi-control method, which is applied to a master control terminal, and a master toolkit is deployed on the master control terminal.
  • the master control terminal may be Smart phones, tablets, palmtops, and other smart terminals with Android operating system, and the master terminal is connected to at least one slave terminal. Both the master terminal and the slave terminal are mobile terminals installed with the Android system. A slave kit is deployed on the terminal.
  • a flow chart of a one-machine multi-control method includes the following steps:
  • Step S102 responding to the touch operation of the graphical user interface of the master control terminal, and converting the operation event corresponding to the touch operation into an operation protocol package through the master toolkit;
  • Step S104 sending a message to the slave control terminal, the message carrying the above-mentioned operation protocol package, to trigger the slave control terminal to perform a touch operation based on the operation protocol package;
  • the above-mentioned master control terminal and slave control terminal are both mobile terminals installed with the Android system, and in order to realize the one-machine multi-control method, it is usually necessary to establish a connection between the master control terminal and the slave control terminal.
  • the master toolkit is deployed on the master terminal in advance, and the slave toolkit is deployed on the slave terminal, and then the communication connection between the master terminal and the slave terminal is established through the master toolkit and the slave toolkit Among them, the establishment of a communication connection is usually realized by configuring the corresponding functional modules in the master toolkit and the slave toolkit.
  • both the master toolkit and the slave toolkit can be provided with an interface to realize communication.
  • the interface may be a ZeroMQ function module.
  • the operating protocol is sent through the ZeroMQ message sending module configured in the main toolkit, and correspondingly, the operating protocol is received through the ZeroMQ message receiving module configured in the toolkit.
  • the above-mentioned message sent by the master terminal to the slave terminal is a ZeroMQ message carrying the operation protocol package, so as to realize the ZeroMQ connection between the master terminal and the slave terminal, so as to realize the conciseness between the master terminal and the slave terminal.
  • high-performance communication in addition to the above-mentioned ZeroMQ function module, other interface protocol modules can be set in the master toolkit and the slave toolkit, which can be specifically set according to actual use conditions, which are not limited in the embodiments of the present disclosure.
  • the master toolkit and the slave toolkit also include other functional modules, such as the ability to obtain device information of the master terminal or the slave terminal, and so on. Therefore, when the user performs a touch operation on the master terminal, the master terminal can execute the above-mentioned steps S102 and S104, and the slave terminal is controlled by the master terminal, thereby achieving the purpose of one machine with multiple controls.
  • the one-machine multi-control method can respond to the touch operation of the graphical user interface of the master control terminal, and convert the operation event corresponding to the touch operation into an operation protocol package through the master toolkit;
  • the control terminal sends a message carrying the operation protocol package to trigger the slave control terminal to perform touch operations based on the operation protocol package.
  • the master control terminal and the slave control terminal directly establish a communication connection through the master toolkit and the slave toolkit Therefore, there is no need for additional hardware costs, simple and convenient deployment, and at the same time, when deploying the main toolkit on the main control terminal, there is no need to restrict the model, resolution, system version, etc. of the main control terminal, so it has better expansion
  • the compatibility and compatibility prevent testers from performing separate tests on different versions, effectively reducing labor costs, and at the same time improving the efficiency of testing.
  • connection between the master terminal and the slave terminal through the master toolkit and the slave toolkit is a wireless connection.
  • the master terminal and the slave terminal be in the same WiFi environment , Or, in a different wifi environment but can communicate with each other in a network environment, so as to facilitate the implementation of the above-mentioned one-machine multi-control scheme.
  • FIG. 2 shows a schematic connection diagram of a one-machine multi-control solution. As shown in FIG. 2, it includes a master terminal and multiple slave terminals. Among them, the block in the master terminal in FIG. 2 Represents the master toolkit, the box in the slave terminal represents the slave toolkit, and the dashed line connecting the master terminal and the slave terminal represents the wireless connection.
  • a limited number of slave terminals is used as an example for illustration in Figure 2. In actual use, the number of slave control terminals can be set according to actual use conditions, which is not limited in the embodiment of the present disclosure.
  • the master toolkit and slave toolkit are pre-deployed to the master terminal and slave terminal through the service device. Therefore, for the master terminal, the above method It also includes: monitoring the touch operation of the graphical user interface of the main control terminal in response to the start instruction for the main toolkit.
  • the main control terminal usually responds to the start instruction after receiving the main toolkit pushed by the service device.
  • the service device and the main control terminal can be connected via USB, and then the main toolkit can be pushed to the main control terminal.
  • the above-mentioned main toolkit deployed on the main control terminal is generally implemented based on the C language.
  • the service device pushes the main toolkit to the main control terminal, and after the main toolkit is started, the main control terminal can monitor and act on the main control terminal. Control the touch operation of the graphical user interface of the terminal, so that the master control terminal responds to the touch operation, and then executes the above-mentioned process of step S102 and step S104.
  • the above-mentioned main toolkit monitors the touch operation of the graphical user interface of the main control terminal by monitoring the character device file of the main control terminal. This is because when the user performs a touch operation on the graphical user interface The operation event generated by the touch operation can cause the character device file of the master terminal to change, so that the operation event can be recorded. Therefore, if the character device file changes, it can be determined that the graphical user interface of the master terminal is touched. Control operation.
  • the above-mentioned character device files are usually stored in the system directory /dev/input/ of the Android system, and this directory contains various types of character device files.
  • this directory contains various types of character device files.
  • multi-touch character devices are monitored.
  • the operation event generated when the master terminal responds to the touch operation will also meet the format corresponding to the corresponding multi-touch protocol.
  • the multi-touch protocol is mainly divided into two types, one is the TypeA protocol and the other is the TypeB protocol.
  • the TypeA protocol is a stateless multi-touch protocol
  • the TypeB protocol is a stateful protocol.
  • TypeA protocol and TypeB protocol are also called Type A protocol and Type B protocol.
  • Type B protocol uses ID to identify contacts, which reduces the amount of data reported to the user space.
  • Type B protocols need to use ABS_MT_TRACKING_ID, and an ABS_MT_SLOT event will be generated. Therefore, when determining the multi-touch protocol type of the master terminal, it can be determined whether the multi-touch protocol of the master terminal is the TypeA protocol or the TypeB protocol according to whether the multi-touch character device file supports the ABS_MT_SLOT command.
  • the step of converting the operation event corresponding to the touch operation into an operation protocol package through the main toolkit includes: determining whether the multi-touch protocol type of the main control terminal is the second protocol type ; If yes, generate an operation protocol package based on the main toolkit and the operation event corresponding to the touch operation, where, in the embodiment of the present disclosure, the second protocol type is a stateful multi-touch protocol type, that is, the above-mentioned TypeB protocol , Or the protocol type corresponding to the Type B protocol.
  • the multi-touch protocol type of the master control terminal is the first protocol type, that is, the protocol type corresponding to the TypeA protocol or the Type A protocol, it is necessary to convert the operation event corresponding to the touch operation to the one corresponding to the second protocol type Operational events; and then based on the main toolkit and the operation events corresponding to the second protocol type to generate an operation protocol package.
  • the step of generating the operation protocol package includes: extracting operation events from the character device file of the master control terminal through the master toolkit; normalizing the coordinate points and pressure values contained in the extracted operation events; The operation event after the unified processing generates an operation protocol package.
  • the process of judging the multi-touch protocol type of the master terminal is usually implemented in the startup process of the master toolkit. After the service device pushes the master toolkit to the master terminal, it also needs to start the master toolkit , So that when the main control terminal responds to a touch operation, in addition to rendering the graphical user interface according to the touch operation, the main toolkit can also convert the operation event corresponding to the touch operation into an operation protocol package.
  • the operation instruction set supported by the multi-touch protocol type of the main control terminal can also be obtained, such as: ABS_MT_SLOT, ABS_MT_TRACKING_ID, BTN_TOUCH, ABS_MT_TOUCH_MAJOR, ABS_MT_WIDTH_MAJOR, ABS_MT_PRESSURE, etc.
  • monitor the multi-touch character device file If the content of the multi-touch character device file changes, it means that the user has performed some touch operations on the graphical user interface of the master terminal. Next, whenever the user executes During the touch operation, the main toolkit will generate the operation protocol package corresponding to the operation event of the touch operation, and then send the operation protocol package through the message.
  • FIG. 3 shows a schematic diagram of an operation protocol package
  • FIG. 3 shows an operation protocol package generated by operation events after normalization, including single-finger click events, single-finger sliding events, and multi-finger Sliding events, where the normalization parameters used in the normalization process can be parsed from the character device file, such as the maximum height, maximum width, minimum pressure value, and maximum pressure value of the aforementioned graphical user interface.
  • d 0 0.2 0.3 0.5 means: the first item d represents down, which represents the operation of finger pressing, the second item 0 represents the first finger, and the third item 0.2 represents the normalized On the abscissa, the fourth term 0.3 represents the normalized ordinate, and the fifth term 0.5 represents the normalized pressure value.
  • m 0 0.4 0.4 0.5 means: the first item m stands for move, which means the operation of sliding the finger, and the other items refer to the same as the down operation (down), so I won’t repeat them again.
  • u 0 means: first The item u represents up, which means the operation of lifting the finger, and the second item 0 represents the first finger.
  • the embodiment of the present disclosure also provides a one-machine multi-control method, which is applied to a slave control terminal, and a slave toolkit is deployed on the slave control terminal, specifically, the slave control terminal Connected to the master terminal, the master terminal and the slave terminal are both mobile terminals installed with the Android system, and the master toolkit is deployed on the master terminal, as shown in the flowchart of another one-machine multi-control method as shown in Figure 4.
  • the method includes:
  • Step S402 receiving a message sent by the master terminal
  • the message carries an operation protocol package corresponding to the touch operation on the master terminal
  • the master terminal and the slave terminal can establish a ZeroMQ connection, and the message sent by the master terminal is the ZeroMQ message carrying the operation protocol package.
  • the message sent by the master control terminal received by the control terminal is also the ZeroMQ message carrying the operating protocol package.
  • the slave control terminal After the slave control terminal receives the message, the following steps can be executed to parse the ZeroMQ message.
  • Step S404 parse out the operation protocol package from the message
  • step S406 a simulation operation is performed on the graphical user interface of the slave control terminal based on the operation protocol package.
  • the simulation operation on the graphical user interface of the slave control terminal includes: extracting the operation event carried in the operation protocol package, and writing the operation event to the character device file of the slave control terminal to update the character device file of the slave control terminal ; Based on the slave toolkit and updated character device files to simulate operation on the graphical user interface of the slave terminal.
  • the operation protocol package is generated by the master terminal in conjunction with the multi-touch protocol, when extracting the operation events carried in the operation protocol package, it is also necessary to combine the current slave terminal’s multi-touch protocol type Therefore, when extracting the operation event carried in the operation protocol package, it is necessary to obtain the current multi-touch protocol type of the slave terminal; if the multi-touch protocol type is the second protocol type, extract the first from the operation protocol package Operation events corresponding to two protocol types, where the second protocol type is the stateful multi-touch protocol type described in the foregoing embodiment.
  • the operation event corresponding to the first protocol type is extracted from the operation protocol packet, where the first protocol type is the stateless multi-touch described in the foregoing embodiment. Control protocol type.
  • the slave terminal when the slave terminal receives the operation protocol sent by the master terminal through the ZeroMQ message, it can be based on the multi-touch protocol type (TypeA or TypeB) of the slave terminal, Convert the received operation protocol packet into different types of operation events, and then write the operation events into the character device file of the slave terminal to simulate the operation of the user, and realize the operation of the slave terminal and the master terminal are consistent, so as to achieve In this way, the effect of the master terminal controlling the slave terminal is realized, so that the one-machine multi-control scheme is realized.
  • TypeA or TypeB multi-touch protocol type
  • the slave control terminal can usually be divided into two implementation methods.
  • the C language and driver are used.
  • the underlying multi-touch character device file and other technologies are implemented; for systems of Android 10 and above, the Java language is used to call Android standard multi-touch API (Application Programming Interface) and other technologies.
  • the processing process of the slave control terminal below the Android version 10.
  • the version of the Android system of the slave control terminal is an operating system that satisfies the preset version, for example, Android Operating system version 10 or higher
  • the operation protocol package after the conversion can be simulated on the graphic user interface of the slave control terminal.
  • Android 10 and above have strict permission control on the underlying character device files, and cannot write to character device files.
  • the slave toolkit of the slave control terminal of Android 10 and above The operation event will not be written to the character device file of the slave control terminal.
  • the API of the obtain event can simulate user operations, and its implementation principle is relatively simple.
  • the above-mentioned slave toolkit is also pre-deployed through the service device, and when the slave toolkit is deployed, the service device will automatically obtain the Android system version of the slave terminal, so as to push the matched version to the slave terminal. From the kit.
  • the above-mentioned one-machine multi-control method applied to the slave control terminal also includes the following process: in response to the start instruction for the slave toolkit, monitoring the message sent by the master control terminal, wherein the slave control terminal also needs to receive the service device push first
  • the slave toolkit is so that when a message sent by the master terminal is received, the method shown in Figure 4 is executed.
  • the service device and the slave terminal can also be connected via USB, and then the toolkit will be pushed to the slave terminal.
  • the embodiment of the present disclosure also provides another one-machine multi-control method, which is applied to the service Device, the service device is connected to the main control terminal and the slave control terminal via USB;
  • Figure 5 shows the flowchart of another one-machine multi-control method, from the perspective of the service device, the one-machine multi-control method under the Android system
  • the deployment of, specifically, as shown in Figure 5, includes the following steps:
  • Step S502 push the master toolkit to the master control terminal, and push the slave toolkit to the slave control terminal;
  • the master toolkit and the slave toolkit are used to establish communication connections;
  • step S504 the master toolkit and the slave toolkit are started, so that the slave control terminal can synchronously simulate the touch operation of the master control terminal.
  • the service device can be any configuration of terminal devices such as desktop computers, notebooks, palmtop computers, etc.
  • the windows operating system is required, and the corresponding application software is usually run on the service device to facilitate the control of the terminal or slave
  • an effective communication connection can be established.
  • the device list can be displayed on the service device.
  • the user can select one of the terminal devices as the master terminal.
  • the rest of the terminal devices are As a slave terminal, after determining the master terminal and slave terminal, the service device can push the pre-stored master toolkit and slave toolkit to the corresponding master terminal and slave terminal respectively to deploy one machine with multiple controls environment of.
  • the main toolkit when pushing the main toolkit to the main control terminal, it is necessary to obtain the device information and IP address of the main control terminal; save the IP address of the main control terminal, and obtain the main toolkit of the main control terminal according to the device information, and The main toolkit is pushed to the main control terminal.
  • the IP address of the master terminal is used to establish a communication connection between the master terminal and the slave terminal, such as establishing a ZeroMQ connection and so on.
  • the main toolkit is usually packaged into a binary file package.
  • the service device can push the file package to the preset directory of the main control terminal.
  • the main toolkit can be pushed to the /data of the Android system. /local/tmp/ directory.
  • the aforementioned device information refers to the serial number of the master terminal and ADB (Android Debug Bridge) status information.
  • the master can also be obtained Information such as the hardware information and operating system version of the control terminal, and specific device information can be obtained according to actual requirements, which are not limited in the embodiments of the present disclosure.
  • the slave toolkit when the slave toolkit is pushed to the slave terminal, the device information and IP address of the slave terminal can be obtained; the IP address of the slave terminal is saved, and the system version corresponding to the slave terminal is obtained according to the device information of the slave terminal.
  • the matched slave toolkit will be pushed to the slave control terminal.
  • real-time running logs can also be displayed on the service device.
  • the running logs are used to record the information of each stage of the master terminal and slave terminal deployment, so that users can view each terminal If you encounter errors during the deployment process, you can also easily troubleshoot the errors.
  • the USB connection between the master terminal and slave terminal and the service device can be disconnected, and the master terminal and slave terminal are in the same local area network.
  • the user can follow the above figure 1 Operate the master terminal with the method shown in Figure 4, and then check the display effect of the slave terminal, so as to realize the automatic process of one machine with multiple controls.
  • the deployment process of the one-machine multi-control solution under the Android system on the service device does not require additional hardware costs, and only a number of terminal device mobile phones with the Android system and an ordinary windows computer are sufficient. For users, the scalability of the whole process is good. At the same time, it can be compatible with terminal devices of different models, resolutions, and system versions. In addition, it can be fully compatible with TypeA protocol and TypeB when performing one machine with multiple controls.
  • the multi-touch protocol of the protocol supports all multi-touch operations without additional hardware costs. One-key operation can be achieved through the service device, and the deployment of the master terminal and all slave terminals can be deployed.
  • the above-mentioned one-machine multi-control method also includes the following advantages:
  • the high-performance ZeroMQ publish-subscribe mode is used as the data transmission method, which has good scalability and can expand the number of slave control terminals infinitely without any impact on performance.
  • the main toolkit directly converts the TypeA protocol to the TypeB protocol, so this solution is fully compatible with TypeA and TypeB multi-touch protocols, supports all multi-touch operations, and is very important to users. It is completely transparent, and there is no need for the user to care about the multi-touch protocol of the master terminal during actual use.
  • an embodiment of the present disclosure also provides a one-machine multi-control device, which is applied to a master control terminal, and a master toolkit is deployed on the master control terminal.
  • the terminal is connected to at least one slave control terminal.
  • Both the master control terminal and the slave control terminal are mobile terminals installed with the Android system, and the slave toolkit is deployed on the slave control terminal, as shown in Figure 6 for a one-machine multi-control device
  • the structure diagram, as shown in Figure 6, the device includes:
  • the conversion module 60 is configured to respond to the touch operation of the graphical user interface of the master control terminal, and convert the operation event corresponding to the touch operation into an operation protocol package through the master toolkit;
  • the sending module 62 is configured to send a message to the slave control terminal, the message carrying the operation protocol package to trigger the slave control terminal to perform the touch operation based on the operation protocol package through the slave tool package.
  • FIG. 7 also shows a schematic structural diagram of another one-machine multi-control device.
  • the device also includes:
  • the first activation module 64 is configured to monitor the touch operation of the graphical user interface of the main control terminal in response to the activation instruction for the main toolkit.
  • the first activation module 64 is further configured to monitor the character device file of the main control terminal; if the character device file changes, determine that a touch operation occurs on the graphical user interface of the main control terminal.
  • the above conversion module is further configured to: determine whether the multi-touch protocol type of the master control terminal is the second protocol type; if so, generate an operation based on the master toolkit and the operation event corresponding to the touch operation Protocol package; wherein, the second protocol type is a stateful multi-touch protocol type.
  • the conversion module is further configured to: if the multi-touch protocol type of the master control terminal is the first protocol type, convert the operation event corresponding to the touch operation into the operation event corresponding to the second protocol type, where
  • the first protocol type is a stateless multi-touch protocol type; an operation protocol package is generated based on the operation event corresponding to the main toolkit and the second protocol type.
  • the step of generating an operation protocol package based on the main toolkit and the operation event corresponding to the touch operation includes: extracting the operation event from the character device file of the main control terminal through the main toolkit; The coordinate points and pressure values included in the operation event are normalized; an operation protocol package is generated according to the operation event after the normalization processing.
  • the embodiment of the present disclosure also provides another one-machine multi-control device, which is applied to a slave control terminal.
  • a slave toolkit is deployed on the slave control terminal.
  • Both the master control terminal and the slave control terminal are mobile terminals installed with the Android system, and the master toolkit is deployed on the master control terminal, as shown in Figure 8.
  • the device includes:
  • the receiving module 80 is configured to receive a message sent by the master control terminal; wherein the message carries an operation protocol package corresponding to the touch operation on the master control terminal;
  • the parsing module 82 is configured to parse the operation protocol packet from the message
  • the simulation operation module 84 is configured to perform a simulation operation on the graphical user interface of the slave control terminal based on the operation protocol package.
  • the simulation operation module 84 is also used to: extract the operation event carried in the operation protocol package, and write the operation event to the character device file of the slave control terminal, so as to check the character device of the slave control terminal.
  • the file is updated; based on the slave toolkit and the updated character device file, a simulation operation is performed on the graphical user interface of the slave terminal.
  • the simulation operation module 84 is further configured to: obtain the multi-touch protocol type of the slave control terminal; if the multi-touch protocol type is the second protocol type, extract the multi-touch protocol type from the operation protocol packet The operation event corresponding to the second protocol type, where the second protocol type is a stateful multi-touch protocol type.
  • the simulation operation module 84 is further configured to: if the multi-touch protocol type is the first protocol type, extract the operation event corresponding to the first protocol type from the operation protocol packet, wherein the first protocol type A protocol type is a stateless multi-touch protocol type.
  • the simulation operation module 84 is further configured to: if the version of the Android system of the slave control terminal is an operating system that satisfies the preset version, obtain the rotation direction of the graphical user interface of the slave control terminal; The received operation protocol package performs coordinate conversion, and a preset application program is invoked to perform a simulation operation on the converted operation protocol package on the graphical user interface of the slave control terminal.
  • FIG. 9 also shows a schematic structural diagram of another one-machine multi-control device.
  • the device also includes:
  • the second activation module 86 is configured to monitor the message sent by the master control terminal in response to the activation instruction for the slave toolkit.
  • the embodiment of the present disclosure also provides a one-machine multi-control system, which includes a service device, and also includes a master terminal and a slave terminal, wherein the service device is in communication connection with the master terminal and the slave terminal;
  • the master control terminal is configured with the device shown in FIG. 6 or FIG. 7, and the slave control terminal is configured with the device shown in FIG. 8 or FIG. 9.
  • the service device is used to push the master toolkit to the master control terminal and the slave toolkit to the slave control terminal; and to start the master toolkit and the slave toolkit to realize the slave control terminal to synchronize the touch operation of the master control terminal.
  • the service device can obtain the device information and IP address of the main control terminal; save the IP address of the main control terminal, and obtain the main toolkit of the main control terminal according to the device information, and push the main toolkit to the Master terminal; wherein the IP address of the master terminal is used to establish a connection between the master terminal and the slave terminal.
  • the service device can also obtain the device information and IP address of the slave control terminal; save the IP address of the slave control terminal, and obtain the slave toolkit matching the system version of the slave control terminal according to the device information of the slave control terminal, And push the slave toolkit to the slave control terminal.
  • an embodiment of the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor, and the processor implements the foregoing when the computer program is executed.
  • an electronic device including a memory, a processor, and a computer program stored on the memory and capable of running on the processor, and the processor implements the foregoing when the computer program is executed.
  • the embodiment of the present disclosure also provides a schematic structural diagram of an electronic device. As shown in FIG. 10, it is a schematic structural diagram of the electronic device.
  • the electronic device includes a processor 111 and a memory 110.
  • the electronic device further includes a bus 112 and a communication interface 113, where the processor 111, the communication interface 113 and the memory 110 are connected through the bus 112.
  • the memory 110 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
  • the communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 113 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
  • the bus 112 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnection standard) bus, or an EISA (Extended Industry Standard Architecture) bus or the like.
  • the bus 112 can be divided into an address bus, a data bus, a control bus, and so on. For ease of presentation, only one bidirectional arrow is used to indicate in FIG. 10, but it does not mean that there is only one bus or one type of bus.
  • the processor 111 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the foregoing method can be completed by an integrated logic circuit of hardware in the processor 111 or instructions in the form of software.
  • the aforementioned processor 111 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; it may also be a Digital Signal Processor (DSP for short). ), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present disclosure may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory, and the processor 111 reads the information in the memory, and completes the steps of the aforementioned one-machine multi-control method in combination with its hardware.
  • the embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and the computer program executes the above-mentioned one-machine multi-control method when the computer program is run by a processor.
  • the computer program product of the one-machine multi-control method, device, system and electronic equipment provided by the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the foregoing method embodiments
  • the instructions included in the program code can be used to execute the foregoing method embodiments
  • the terms “installed”, “connected”, and “connected” should be understood in a broad sense, for example, they may be fixed connections or detachable connections. , Or integrally connected; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication between two components.
  • installed e.g., they may be fixed connections or detachable connections. , Or integrally connected; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication between two components.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present disclosure essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present disclosure.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Telephone Function (AREA)
  • Selective Calling Equipment (AREA)

Abstract

一种一机多控方法、装置、系统及电子设备，涉及智能控制的技术领域，应用于主控终端的方法包括，响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包（S102）；向从控终端发送消息，该消息携带有操作协议包，以触发从控终端通过从工具包基于操作协议包执行触控操作（S104）。

Description

一机多控方法、装置、系统及电子设备
相关申请的交叉引用
本申请要求于2020年06月10日提交的申请号为202010526846.4、名称为“一机多控方法、装置、系统及电子设备”的中国专利申请的优先权，该中国专利申请的全部内容通过引用并入本文。
技术领域
本公开涉及智能控制的技术领域，尤其是涉及一种一机多控方法、装置、系统及电子设备。
背景技术
目前，国内Android系统的终端品牌众多，每个终端的品牌旗下还拥有许多不同型号的终端，加之各个品牌每隔一段时间就会推出新品，就会导致Android系统的终端在市场拥有成百上千种，这些终端的型号、分辨率、系统版本等各不相同，给移动应用和游戏开发测试人员带来了诸多的麻烦。
如果单纯的靠测试人员在每种型号，每种分辨率，每种系统版本上都分别去做测试的话，工作量是巨大的，不仅需要较高的人力成本，也严重制约了测试人员的工作效率。
发明内容
有鉴于此，本公开的目的在于提供一种一机多控方法、装置、系统及电子设备，以缓解上述技术问题。
第一方面，本公开实施例提供了一种一机多控方法，该方法应用于主控终端，主控终端上部署有主工具包，主控终端与至少一从控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，从控终端上部署有从工具包，该方法包括：响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包；向从控终端发送消息，该消息携带有操作协议包，以触发从控终端通过从工具包基于操作协议包执行触控操作。
在一种较佳的实施方式中，上述方法还包括：响应针对于主工具包的启动指令，监听主控终端的图形用户界面的触控操作。
在一种较佳的实施方式中，上述监听主控终端的图形用户界面的触控操作的步骤，包括：监听主控终端的字符设备文件；如果字符设备文件发生变化，则确定主控终端的图形用户界面发生触控操作。
在一种较佳的实施方式中，通过主工具包将触控操作对应的操作事件转换成操作协议包的步骤包括：判断主控终端的多点触控协议类型是否为第二协议类型；如果是，基于主工具包和触控操作对应的操作事件生成操作协议包，其中，第二协议类型为有状态 的多点触控协议类型。
在一种较佳的实施方式中，上述通过主工具包将触控操作对应的操作事件转换成操作协议包的步骤，还包括：如果主控终端的多点触控协议类型为第一协议类型，将触控操作对应的操作事件转换为第二协议类型对应的操作事件；基于主工具包和第二协议类型对应的操作事件生成操作协议包，其中，第一协议类型为无状态的多点触控协议类型。
在一种较佳的实施方式中，上述基于主工具包和触控操作对应的操作事件生成操作协议包的步骤包括：通过主工具包从主控终端的字符设备文件中提取操作事件；对提取到的操作事件中包含的坐标点和压力值进行归一化处理；根据归一化处理之后的操作事件生成操作协议包。
在一种较佳的实施方式中，上述向从控终端发送的消息为携带有操作协议包的ZeroMQ消息。
第二方面，本公开实施例提供了一种一机多控方法，该方法应用于从控终端，从控终端上部署有从工具包，从控终端与主控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，主控终端上部署有主工具包，该方法包括：接收主控终端发送的消息；其中，该消息携带有主控终端上的触控操作对应的操作协议包；从消息解析出操作协议包；基于操作协议包在从控终端的图形用户界面进行模拟操作。
在一种较佳的实施方式中，上述基于操作协议包在从控终端的图形用户界面进行模拟操作的步骤包括：提取操作协议包中携带的操作事件，将操作事件写入到从控终端的字符设备文件，以对从控终端的字符设备文件进行更新；基于从工具包和更新的字符设备文件在从控终端的图形用户界面进行模拟操作。
在一种较佳的实施方式中，上述提取操作协议包中携带的操作事件的步骤包括：获取从控终端的多点触控协议类型；如果多点触控协议类型为第二协议类型，则从操作协议包中提取第二协议类型对应的操作事件，其中，所述第二协议类型为有状态的多点触控协议类型。
在一种较佳的实施方式中，上述提取操作协议包中携带的操作事件的步骤还包括：如果多点触控协议类型为第一协议类型，则从操作协议包中提取第一协议类型对应的操作事件，其中，所述第一协议类型为无状态的多点触控协议类型。
在一种较佳的实施方式中，上述基于操作协议包在从控终端的图形用户界面进行模拟操作的步骤还包括：如果从控终端的安卓系统的版本为满足预设版本的操作系统，获取从控终端的图形用户界面的旋转方向；根据旋转方向对接收到的操作协议包进行坐标转换，并调用预设的应用程序以在从控终端的图形用户界面上对转换之后的操作协议包进行模拟操作。
在一种较佳的实施方式中，上述方法还包括：响应针对于从工具包的启动指令，监听主控终端发送的消息。
在一种较佳的实施方式中，上述接收的主控终端发送的消息为携带有操作协议包的ZeroMQ消息。
第三方面，本公开实施例提供了一种一机多控装置，该装置应用于主控终端，主控终端上部署有主工具包，主控终端与至少一从控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，从控终端上部署有从工具包，该装置包括：转换模块，用于响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包；发送模块，用于向从控终端发送消息，该消息携带有操作协议包，以触发从控终端通过从工具包基于操作协议包执行触控操作。
第四方面，本公开实施例提供了一种一机多控装置，该装置应用于从控终端，从控终端上部署有从工具包，从控终端与主控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，主控终端上部署有主工具包，该装置包括：接收模块，用于接收主控终端发送的消息；其中，该消息携带有主控终端上的触控操作对应的操作协议包；解析模块，用于从消息解析出操作协议包；模拟操作模块，用于基于操作协议包在从控终端的图形用户界面进行模拟操作。
第五方面，本公开实施例提供了一种一机多控系统，该系统包括服务设备，还包括主控终端和从控终端，其中，服务设备与主控终端和从控终端通信连接；
所述主控终端配置有第三方面所述的装置，所述从控终端配置有第四方面所述的装置；
所述服务设备用于向所述主控终端推送主工具包，向所述从控终端推送从工具包；以及，启动所述主工具包和所述从工具包，以实现所述从控终端同步模拟所述主控终端的触控操作。
第六方面，本公开实施例提供了一种电子设备，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，该处理器执行所述计算机程序时实现上述第一至第二方面所述的方法。
第七方面，本公开实施例提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行上述第一至第二方面所述的方法。
本公开实施例带来了以下有益效果：
本公开实施例提供的一种一机多控方法、装置、系统及电子设备，能够响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包；进而向从控终端发送携带有操作协议包的消息，以触发从控终端基于操作协议包执行触控操作，在控制过程中，由于主控终端和从控终端直接通过主工具包和从工具包建立通信连接，因此，无需额外的硬件成本，部署简单方便，同时，在主控终端上部署主工具包时，无需对主控终端的型号、分辨率、系统版本等进行限制，因此，具有较好的拓展性和兼容性，避免了测试人员在不同版本上分别进行测试，有效减少了人力成 本，同时，也提高了测试的工作效率。
本公开的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本公开而了解。本公开的目的和其他优点在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本公开的一些实施方式，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种一机多控方法的流程图；
图2为本公开实施例提供的一种一机多控方案的连接示意图；
图3为本公开实施例提供的一种操作协议包的示意图；
图4为本公开实施例提供的另一种一机多控方法的流程图；
图5为本公开实施例提供的另一种一机多控方法的流程图；
图6为本公开实施例提供的一种一机多控装置的结构示意图；
图7为本公开实施例提供的另一种一机多控装置的结构示意图；
图8为本公开实施例提供的另一种一机多控装置的结构示意图；
图9为本公开实施例提供的另一种一机多控装置的结构示意图；
图10为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合附图对本公开的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是全部的实施例。基于本公开中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
目前的一机多控方案中，主要有Solopi方案和Total Control方案，其中，Solopi方案的设计和实现是基于录制回放功能实现的，在录制回放的场景中，先在一台设备上记录了用户的操作，然后将该用户的操作记录通过socket发送到其它任意一台设备上实现操作的回放，从而实现通过一台设备操控多台设备的效果。在实现时需要主控机与从控机建立Socket连接，然后在主控机上将用户的操作记录实时发送到各个从控机，从控机在接收到操作记录后进行解析，然后在从控机上模拟用户的操作，从而完成操作记录的回放。
而Total Control方案则是通过USB/WIFI实现电脑与终端的互联互通，在电脑端对终端进行统一的控制。支持终端信息查看、消息提示、一键进入消息所在应用，支持脚本 录制、编辑和自动化执行、批量一键安装应用等功能，在实现时，需要使终端与HUB集线器之间通过USB线连通，HUB集线器也通过USB线与电脑连通，然后在终端和电脑上安装相应的客户端，通过将终端画面映射到电脑上，然后通过鼠标在电脑上对终端进行操作，从而达到多控的效果。
上述两种常用的方案，虽然都能够实现一机多控方案，但是，Solopi方案由于是基于录制回放原理实现的，因此，其操作延迟较大，用户在主控机上未完成操作之前，从控机不会有任何响应，只有待主控机一个操作完成，将操作记录发送到从控机，从控机经过解析后，才会有相应的响应，整个过程耗时较长，因此针对一些复杂的多点触控操作，解析更慢，其延迟将会更大，将严重影响实际使用的效果和用户体验，并且，可支持的设备数量有限，也严重影响了大规模设备集群的使用。
Total Control方案则需要往外的硬件成本，包括电脑，HUB集线器，USB线等硬件，且随着终端的数量越多，需要的电脑配置就要越好，同时，对终端与电脑之间的兼容性也有一定的要求。进一步，由于该多控方案是通过电脑来进行多机操作的，而电脑鼠标只能模拟一些单点触控的操作，无法模拟多点触控操作，因此该方案不支持多点触控操作，对于一些需要多点触控操作也无法进行测试。
基于此，本公开实施例提供的一种一机多控方法、装置、系统及电子设备，可以有效缓解上述技术问题。
为便于对本实施例进行理解，首先对本公开实施例所公开的一种一机多控方法进行详细介绍。
在一种可能的实施方式中，本公开实施例提供了一种一机多控方法，该方法应用于主控终端，主控终端上部署有主工具包，具体地，该主控终端可以是智能手机、平板电脑、掌上电脑等具有安卓操作系统的智能终端，并且，该主控终端与至少一从控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，从控终端上部署有从工具包。
如图1所示的一种一机多控方法的流程图，该方法包括以下步骤：
步骤S102，响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包；
步骤S104，向从控终端发送消息，该消息携带有上述操作协议包，以触发从控终端基于该操作协议包执行触控操作；
具体实现时，上述主控终端和从控终端均为安装有安卓系统的移动终端，且，为了能够实现一机多控方法，通常需要在主控终端和从控终端之间建立连接，本公开实施例中，则是预先在主控终端上部署主工具包，以及在从控终端上部署从工具包，然后通过主工具包和从工具包建立主控终端和从控终端之间的通信连接，其中，建立通信连接通常是主工具包和从工具包的中配置相应的功能模块来实现的。在一优选实施方式中，主工具包和从工具包中均可以设置一接口实现通信。优选地，所述接口可以为ZeroMQ功 能模块，例如，操作协议通过主工具包中配置的ZeroMQ消息发送模块将其发送，相应地，通过从工具包中配置的ZeroMQ消息接收模块接收操作协议。即，上述主控终端向从控终端发送的消息为携带有所述操作协议包的ZeroMQ消息，以实现主控终端和从控终端以ZeroMQ连接，从而实现主控终端与从控终端之间简洁和高性能的通信。在实际使用时，除上述ZeroMQ功能模块，在主工具包和从工具包中还可以设置其他接口协议模块，具体可以根据实际使用情况进行设置，本公开实施例对此不进行限制。
进一步，主工具包和从工具包还包括其他功能模块，如能够获取主控终端或者从控终端的设备信息等等功能。因此，当用户在主控终端上执行触控操作时，主控终端可以执行上述步骤S102和步骤S104的过程，通过主控终端来控制从控终端，从而实现一机多控的目的。
因此，本公开实施例提供的一机多控方法，能够响应主控终端的图形用户界面的触控操作，并通过主工具包将触控操作对应的操作事件转换成操作协议包；进而向从控终端发送携带有操作协议包的消息，以触发从控终端基于操作协议包执行触控操作，在控制过程中，由于主控终端和从控终端直接通过主工具包和从工具包建立通信连接，因此，无需额外的硬件成本，部署简单方便，同时，在主控终端上部署主工具包时，无需对主控终端的型号、分辨率、系统版本等进行限制，因此，具有较好的拓展性和兼容性，避免了测试人员在不同版本上分别进行测试，有效减少了人力成本，同时，也提高了测试的工作效率。
在实际使用时，主控终端和从控终端之间通过主工具包和从工具包进行的连接是一种无线连接，具体实现时，通常要求主控终端和从控终端均处于同一个WiFi环境，或者，不同wifi环境但能够互通的网络环境下，以便于执行上述一机多控方案。
进一步，为了便于理解，图2示出了一种一机多控方案的连接示意图，如图2所示，包括主控终端和多个从控终端，其中，图2中主控终端中的方块代表主工具包，从控终端中的方块代表从工具包，且，连接主控终端和从控终端的虚线代表无线连接，为了便于说明，图2中以有限个从控终端为例进行说明，在实际使用时，从控终端的数量可以根据实际使用情况进行设置，本公开实施例对此不进行限制。
进一步，为了实现主控终端与从控终端能够建立通信，上述主工具包和从工具包，是通过服务设备预先部署到主控终端和从控终端上的，因此，对于主控终端，上述方法还包括：响应针对于主工具包的启动指令，监听主控终端的图形用户界面的触控操作，其中，主控终端通常在接收到服务设备推送的主工具包之后，可以响应上述启动指令。例如，服务设备与主控终端可以通过USB连接，然后将主工具包推送至主控终端上。
具体实现时，上述部署于主控终端的主工具包一般是基于C语言实现的，服务设备将主工具包推送至主控终端，并启动主工具包之后，可以使主控终端监听作用于主控终端的图形用户界面的触控操作，以使主控终端响应该触控操作，进而执行上述步骤S102和步骤S104的过程。
在实际使用时，上述主工具包监听主控终端的图形用户界面的触控操作是通过监听主控终端的字符设备文件实现的，这是因为，当用户在图形用户界面上执行触控操作时，该触控操作产生的操作事件能够引起主控终端的字符设备文件发生变化，以便于记录该操作事件，因此，如果该字符设备文件发生变化，则可以确定主控终端的图形用户界面发生触控操作。
具体地，上述字符设备文件通常在安卓系统的系统目录/dev/input/中保存，该目录下包含多种类型的字符设备文件，本公开实施例中，监听的是多点触控的字符设备文件，以便于对该字符设备文件进行解析，获取主控终端的相关属性，如图形用户界面的最大高度、最大宽度、最小压力值和最大压力值等等，以便于生成后续的操作协议包。
进一步，考虑到安卓系统的智能终端通常是满足一定的多点触控协议的，因此，在主控终端响应触控操作时产生的操作事件也会满足相应的多点触控协议对应的格式。
目前，多点触控协议主要分为两种，一种是TypeA协议，一种是TypeB协议，其中，TypeA协议是一种无状态的多点触控协议，而TypeB协议是一种有状态的多点触控协议。TypeA协议和TypeB协议也称为A类协议和B类协议，其主要区别是B类协议使用ID来标识触点，减少了上报到用户空间的数据量。B类协议需要使用ABS_MT_TRACKING_ID，会产生一个ABS_MT_SLOT事件。因此，在判断主控终端的多点触控协议类型时，可以根据多点触控字符设备文件是否支持ABS_MT_SLOT指令来判断主控终端的多点触控协议是TypeA协议还是TypeB协议。
进一步，本公开实施例中上述步骤S102中，通过主工具包将触控操作对应的操作事件转换成操作协议包的步骤包括：判断主控终端的多点触控协议类型是否为第二协议类型；如果是，则基于主工具包和触控操作对应的操作事件生成操作协议包，其中，本公开实施例中，第二协议类型为有状态的多点触控协议类型，即，上述TypeB协议，或者B类协议对应的协议类型。
进一步，如果主控终端的多点触控协议类型为第一协议类型，即，TypeA协议或者A类协议对应的协议类型，则需要将触控操作对应的操作事件转换为第二协议类型对应的操作事件；然后再基于主工具包和第二协议类型对应的操作事件生成操作协议包。
具体地，生成操作协议包的步骤包括：通过主工具包从主控终端的字符设备文件中提取操作事件；对提取到的操作事件中包含的坐标点和压力值进行归一化处理；根据归一化处理之后的操作事件生成操作协议包。
具体实现时，判断主控终端的多点触控协议类型的过程，通常是在主工具包的启动过程实现的，服务设备将主工具包推送至主控终端后，还需要启动该主工具包，以便于主控终端响应触控操作时，除根据触控操作渲染图形用户界面外，还可以通过该主工具包将触控操作对应的操作事件转换成操作协议包。
进一步，在主工具包启动时，还可以获取该主控终端的多点触控协议类型所支持的操作指令集，如：ABS_MT_SLOT，ABS_MT_TRACKING_ID，BTN_TOUCH， ABS_MT_TOUCH_MAJOR，ABS_MT_WIDTH_MAJOR，ABS_MT_PRESSURE等。同时，监控多点触控字符设备文件，如果当多点触控字符设备文件内容发生变化时，说明用户在该主控终端的图形用户界面进行了一些触控操作，接下来，每当用户执行触控操作时，主工具包都会将触控操作对应的操作事件生成操作协议包，然后通过消息将操作协议包发送出去。
进一步，图3示出了一种操作协议包的示意图，且，图3中示出的是归一化之后的操作事件生成的操作协议包，包括单指点击事件，单指滑动事件和多指滑动事件，其中，归一化处理时用到的归一化参数，可以从字符设备文件中解析得到，如前述图形用户界面的最大高度、最大宽度、最小压力值和最大压力值等参数。进一步，图3中，d 0 0.2 0.3 0.5表示的意思是：第一项d代表down，表示手指按下的操作，第二项0代表第一根手指，第三项0.2表示归一化后的横坐标，第四项0.3表示归一化后的纵坐标，第五项0.5表示归一化后的压力值。m 0 0.4 0.4 0.5表示的意思是：第一项m代表move，表示手指滑动的操作，其它项与按下操作(down)的指代相同，不再赘述，u 0表示的意思是：第一项u代表up，表示手指抬起的操作，第二项0代表第一根手指。
上述图3所示的操作协议包通过消息发送至从控终端后，主控终端的相关流程结束。
进一步，在上述图1的基础上，本公开实施例还提供了一种一机多控方法，该方法应用于从控终端，且从控终端上部署有从工具包，具体地，从控终端与主控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，主控终端上部署有主工具包，如图4所示的另一种一机多控方法的流程图，该方法包括：
步骤S402，接收主控终端发送的消息；
其中，该消息携带有主控终端上的触控操作对应的操作协议包；
以上述主工具包和从工具包均设置ZeroMQ功能模块为例，此时，主控终端和从控终端可以建立ZeroMQ连接，主控终端发送的消息即为携带有操作协议包的ZeroMQ消息，从控终端接收的主控终端发送的消息也为该携带有操作协议包的ZeroMQ消息，从控终端接收到该消息之后，可以执行下述步骤，对ZeroMQ消息进行解析。
步骤S404，从消息解析出操作协议包；
步骤S406，基于操作协议包在从控终端的图形用户界面进行模拟操作。
其中，在从控终端的图形用户界面进行模拟操作包括：提取操作协议包中携带的操作事件，将操作事件写入到从控终端的字符设备文件，以对从控终端的字符设备文件进行更新；基于从工具包和更新的字符设备文件在从控终端的图形用户界面进行模拟操作。
在实际使用时，由于操作协议包是主控终端结合多点触控协议生成的，因此，在提取操作协议包中携带的操作事件时，也需要结合当前从控终端的多点触控协议类型，因此，在提取操作协议包中携带的操作事件时，需要获取当前从控终端的多点触控协议类 型；如果多点触控协议类型为第二协议类型，则从操作协议包中提取第二协议类型对应的操作事件，其中，第二协议类型为前述实施例所述的有状态的多点触控协议类型。
进一步，如果多点触控协议类型为第一协议类型，则从操作协议包中提取第一协议类型对应的操作事件，其中，第一协议类型为前述实施例所述的无状态的多点触控协议类型。
例如，以对ZeroMQ消息解析为例进行说明，当从控终端通过ZeroMQ消息接收到主控终端发送过来的操作议包后，可以根据从控终端的多点触控协议类型(TypeA还是TypeB)，将接收到的操作协议包转换成不同类型的操作事件，然后将该操作事件写入到从控终端的字符设备文件，模拟用户的操作，实现从控终端与主控终端的操作一致，从而达到了主控终端控制从控终端的效果，这样就实现了一机多控方案。
在实际使用时，由于安卓系统，特别是安卓10版本的系统对系统权限做了严格的限制，因此从控终端通常可以分为两种实现方式，针对安卓10以下的系统，采用C语言和驱动底层多点触控字符设备文件等技术实现；针对安卓10及以上的系统，采用java语言调用安卓标准多点触控API(Application Programming Interface，应用程序接口)等技术来实现。
其中，图4所示的一机多控方法中，通常是至安卓10版本以下的从控终端的处理过程，如果从控终端的安卓系统的版本为满足预设版本的操作系统，例如，安卓10版本以上的操作系统，则在接收到操作协议包时，可以获取从控终端的图形用户界面的旋转方向；根据旋转方向对接收到的操作协议包进行坐标转换，并调用预设的应用程序以在从控终端的图形用户界面上对转换之后的操作协议包进行模拟操作。具体地，这是由于安卓10及以上的版本对底层的字符设备文件有严格的权限控制，无法对字符设备文件进行写操作，因此，安卓10及以上的版本的从控终端的从工具包则不会将操作事件写入到从控终端的字符设备文件，只需监控从控终端的屏幕旋转，然后对接收到的操作协议包根据屏幕旋转方向进行坐标转换，最后调用Android API，如MotionEvent.obtain事件的API，即可模拟用户操作，其实现原理要相对简单。
在实际使用时，上述从工具包，也是通过服务设备预先部署的，且，在部署从工具包时，服务设备会自动获取到从控终端的安卓系统版本，以便于向从控终端推送匹配的从工具包。
因此，上述应用于从控终端的一机多控方法，还包括以下过程：响应针对于从工具包的启动指令，监听主控终端发送的消息，其中，从控终端也需要先接收服务设备推送的从工具包，以便于在接收到主控终端发送的消息时，执行图4所示的方法。其中，服务设备与从控终端也可以通过USB连接，然后将从工具包推送至从控终端。
进一步，为了便于对本公开实施例提供的一机多控方法进行理解，在上述图1和图4的基础上，本公开实施例还提供了另一种一机多控方法，该方法应用于服务设备，该服务设备通过USB与主控终端和从控终端连接；如图5所示的另一种一机多控方法的流程 图，从服务设备角度，对安卓系统下的一机多控方法的部署进行说明，具体地，如图5所示，包括以下步骤：
步骤S502，向主控终端推送主工具包，以及向从控终端推送从工具包；
其中，主工具包和从工具包用于建立通信连接的工具包；
步骤S504，启动主工具包和从工具包，以实现从控终端同步模拟主控终端的触控操作。
具体实现时，服务设备可以是任意配置的台式机、笔记本、掌上电脑等终端设备，一般要求windows操作系统即可，且通常在服务设备上运行有相应的应用软件，以便于主控终端或者从控终端通过USB接入服务设备时，能够建立有效的通信连接。
在实际使用时，对于通过USB接入服务设备的终端设备，可以在服务设备上显示设备列表，用户可以选取其中的一个终端设备作为主控终端，当选取主控终端之后，其余的终端设备则作为从控终端，当确定出主控终端和从控终端之后，服务设备可以将预先存储的主工具包和从工具包分别推送至相应的主控终端和从控终端，以部署一机多控的环境。
具体地，向主控终端推送主工具包时，需要获取主控终端的设备信息和IP地址；保存该主控终端的IP地址，以及，根据设备信息获取主控终端的主工具包，并将主工具包推送至所述主控终端。其中，主控终端的IP地址用于在主控终端与从控终端之间建立通信连接，如建立ZeroMQ连接等等。
其中，主工具包通常是打包成二进制的文件包，服务设备可以将该文件包推送至主控终端的预设目录下，通常，对于安卓系统，可以将主工具包推送到安卓系统的/data/local/tmp/目录下。
进一步，本公开实施例中，上述设备信息指的是主控终端的序列号和ADB(Android Debug Bridge，安卓调试桥)状态信息，此外，除上述序列号和ADB状态信息，还可以获取到主控终端的硬件信息和操作系统版本等信息，具体的设备信息，可以根据实际的需求情况进行获取，本公开实施例对此不进行限制。
进一步，向从控终端推送从工具包时，可以获取从控终端的设备信息和IP地址；保存从控终端的IP地址，以及，根据从控终端的设备信息获取与从控终端的系统版本相匹配的从工具包，并将从工具包推送至从控终端。
此外，在部署主工具包和从工具包时，还可以在服务设备上显示实时的运行日志，该运行日志用于记录主控终端和从控终端部署的各个阶段信息，方便用户查看每个终端的部署情况，如遇部署过程中的报错等等，也可以方便对错误进行排查。
当分别部署完主控终端和从控终端之后，可以断开主控终端和从控终端与服务设备的USB连接，并使主控终端和从控终端处于同一局域网中，用户可以按照上述图1和图4所示的方法操作主控终端，然后查看从控终端的显示效果，从而实现一机多控的自动化过程。
因此，在服务设备进行安卓系统下的一机多控方案的部署过程，无需额外的硬件成本，只需若干台安卓系统的终端设备手机和一台普通的windows电脑即可。对于用户来说，整个过程的可扩展性较好，同时，能够兼容不同型号、不同分辨率、不同系统版本的终端设备，并且，在进行一机多控时，还可以完全兼容TypeA协议和TypeB协议的多点触控协议，支持所有的多点触控操作，同时无需额外的硬件成本，通过服务设备即可实现一键操作，部署好主控终端和所有从控终端的部署。
综上，上述一机多控方法，还包括以下优点：
(1)以高性能ZeroMQ的发布订阅模式作为数据传输方式，可扩展性好，可无限扩展从控终端的数量，且对性能无任何影响。
(2)基于C语言和标准API实现的操作指令解析，直接对安卓系统的字符设备文件进行读写操作，性能极高，延迟极小(大约1-5ms)，用户体验好。
(3)由于在主控终端内部，主工具包直接将TypeA协议转换为TypeB协议，因此该方案完全兼容TypeA和TypeB的多点触控协议，支持所有的多点触控操作，且对用户来说完全透明，在实际使用过程中无需用户关心该主控终端的多点触控协议。
对应于图1所示的一机多控方法，本公开实施例还提供了一种一机多控装置，该装置应用于主控终端，且，主控终端上部署有主工具包，主控终端与至少一从控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，从控终端上部署有从工具包，如图6所示的一种一机多控装置的结构示意图，如图6所示，该装置包括：
转换模块60，用于响应主控终端的图形用户界面的触控操作，并通过所述主工具包将所述触控操作对应的操作事件转换成操作协议包；
发送模块62，用于向从控终端发送消息，该消息携带有所述操作协议包，以触发所述从控终端通过从工具包基于操作协议包执行所述触控操作。
进一步，在图6的基础上，图7还示出了另一种一机多控装置的结构示意图，除图6所示的结构外，该装置还包括：
第一启动模块64，用于响应针对于所述主工具包的启动指令，监听所述主控终端的图形用户界面的触控操作。
进一步，第一启动模块64还用于：监听所述主控终端的字符设备文件；如果所述字符设备文件发生变化，则确定所述主控终端的图形用户界面发生触控操作。
进一步，上述转换模块还用于：判断所述主控终端的多点触控协议类型是否为第二协议类型；如果是，基于所述主工具包和所述触控操作对应的操作事件生成操作协议包；其中，所述第二协议类型为有状态的多点触控协议类型。
进一步，上述转换模块还用于：如果所述主控终端的多点触控协议类型为第一协议类型，将所述触控操作对应的操作事件转换为第二协议类型对应的操作事件，其中，所述第一协议类型为无状态的多点触控协议类型；基于所述主工具包和所述第二协议类型对应的操作事件生成操作协议包。
其中，基于所述主工具包和所述触控操作对应的操作事件生成操作协议包的步骤包括：通过所述主工具包从所述主控终端的字符设备文件中提取操作事件；对提取到的所述操作事件中包含的坐标点和压力值进行归一化处理；根据归一化处理之后的所述操作事件生成操作协议包。
对应于图4所示的一机多控方法，本公开实施例还提供了另一种一机多控装置，该装置应用于从控终端，从控终端上部署有从工具包，从控终端与主控终端连接，主控终端和从控终端均为安装有安卓系统的移动终端，主控终端上部署有主工具包，如图8所示的另一种一机多控装置的结构示意图，该装置包括：
接收模块80，用于接收主控终端发送的消息；其中，该消息携带有所述主控终端上的触控操作对应的操作协议包；
解析模块82，用于从所述消息解析出所述操作协议包；
模拟操作模块84，用于基于所述操作协议包在所述从控终端的图形用户界面进行模拟操作。
其中，模拟操作模块84还用于：提取所述操作协议包中携带的操作事件，将所述操作事件写入到所述从控终端的字符设备文件，以对所述从控终端的字符设备文件进行更新；基于所述从工具包和更新的所述字符设备文件在所述从控终端的图形用户界面进行模拟操作。
进一步，模拟操作模块84还用于：获取所述从控终端的多点触控协议类型；如果所述多点触控协议类型为第二协议类型，则从所述操作协议包中提取所述第二协议类型对应的操作事件，其中，所述第二协议类型为有状态的多点触控协议类型。
进一步，模拟操作模块84还用于：如果所述多点触控协议类型为第一协议类型，则从所述操作协议包中提取所述第一协议类型对应的操作事件，其中，所述第一协议类型为无状态的多点触控协议类型。
进一步，模拟操作模块84还用于：如果所述从控终端的安卓系统的版本为满足预设版本的操作系统，获取所述从控终端的图形用户界面的旋转方向；根据所述旋转方向对接收到的所述操作协议包进行坐标转换，并调用预设的应用程序以在所述从控终端的图形用户界面上对转换之后的所述操作协议包进行模拟操作。
进一步，在图8的基础上，图9还示出了另一种一机多控装置的结构示意图，除图8所示的结构外，该装置还包括：
第二启动模块86，用于响应针对于所述从工具包的启动指令，监听所述主控终端发送的消息。
本公开实施例提供的一机多控装置，其实现原理及产生的技术效果和前述方法实施例相同，为简要描述，装置的实施例部分未提及之处，可参考前述方法实施例中相应内容，在此不再赘述。
进一步，本公开实施例还提供了一种一机多控系统，该系统包括服务设备，还包括 主控终端和从控终端，其中，服务设备与主控终端和从控终端通信连接；所述主控终端配置有图6或图7所示的装置，从控终端配置有图8或图9所示的装置。
服务设备用于向主控终端推送主工具包，向从控终端推送从工具包；以及，启动主工具包和从工具包，以实现从控终端同步模拟主控终端的触控操作。
进一步，服务设备可以获取主控终端的设备信息和IP地址；保存该主控终端的IP地址，以及，根据所述设备信息获取主控终端的主工具包，并将主工具包推送至所述主控终端；其中，所述主控终端的IP地址用于在所述主控终端与所述从控终端之间建立连接。
进一步，服务设备还可以获取从控终端的设备信息和IP地址；保存该从控终端的IP地址，以及，根据从控终端的设备信息获取与从控终端的系统版本相匹配的从工具包，并将从工具包推送至所述从控终端。
进一步，本公开实施例还提供了一种电子设备，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，该处理器执行所述计算机程序时实现上述一机多控方法。
本公开实施例还提供了一种电子设备的结构示意图，如图10所示，为该电子设备的结构示意图，其中，该电子设备包括处理器111和存储器110，该存储器110存储有能够被该处理器111执行的计算机可执行指令，该处理器111执行该计算机可执行指令以实现上述一机多控方法。
在图10示出的实施方式中，该电子设备还包括总线112和通信接口113，其中，处理器111、通信接口113和存储器110通过总线112连接。
其中，存储器110可能包含高速随机存取存储器(RAM，Random Access Memory)，也可能还包括非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。通过至少一个通信接口113(可以是有线或者无线)实现该系统网元与至少一个其他网元之间的通信连接，可以使用互联网，广域网，本地网，城域网等。总线112可以是ISA(Industry Standard Architecture，工业标准体系结构)总线、PCI(Peripheral Component Interconnect，外设部件互连标准)总线或EISA(Extended Industry Standard Architecture，扩展工业标准结构)总线等。所述总线112可以分为地址总线、数据总线、控制总线等。为便于表示，图10中仅用一个双向箭头表示，但并不表示仅有一根总线或一种类型的总线。
处理器111可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器111中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器111可以是通用处理器，包括中央处理器(Central Processing Unit，简称CPU)、网络处理器(Network Processor，简称NP)等；还可以是数字信号处理器(Digital Signal Processor，简称DSP)、专用集成电路(Application Specific Integrated Circuit，简称ASIC)、现场可编程门阵列(Field-Programmable Gate Array，简称FPGA)或者其他可编程 逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本公开实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器111读取存储器中的信息，结合其硬件完成前述一机多控方法的步骤。
本公开实施例还提供一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行上述一机多控方法。
本公开实施例所提供的一机多控方法、装置、系统及电子设备的计算机程序产品，包括存储了程序代码的计算机可读存储介质，所述程序代码包括的指令可用于执行前面方法实施例中所述的方法，具体实现可参见方法实施例，在此不再赘述。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统和装置的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
另外，在本公开实施例的描述中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或一体地连接；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，可以是两个元件内部的连通。对于本领域技术人员而言，可以具体情况理解上述术语在本公开中的具体含义。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
在本公开的描述中，需要说明的是，术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本公开和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本公开的限制。此外，术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。
最后应说明的是：以上实施例，仅为本公开的具体实施方式，用以说明本公开的技术方案，而非对其限制，本公开的保护范围并不局限于此，尽管参照前述实施例对本公开进行了详细的说明，本领域技术人员应当理解：任何熟悉本技术领域的技术人员在本 公开揭露的技术范围内，其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化，或者对其中部分技术特征进行等同替换；而这些修改、变化或者替换，并不使相应技术方案的本质脱离本公开实施例技术方案的精神和范围，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以权利要求的保护范围为准。

Claims (19)

  1. 一种一机多控方法，所述方法应用于主控终端，所述主控终端上部署有主工具包，所述主控终端与至少一从控终端连接，所述主控终端和所述从控终端均为安装有安卓系统的移动终端，所述从控终端上部署有从工具包，所述方法包括：
    响应所述主控终端的图形用户界面的触控操作，并通过所述主工具包将所述触控操作对应的操作事件转换成操作协议包；
    向从控终端发送消息，所述消息携带有所述操作协议包，以触发所述从控终端通过所述从工具包基于所述操作协议包执行所述触控操作。
  2. 根据权利要求1所述的方法，其中，所述方法还包括：
    响应针对于所述主工具包的启动指令，监听所述主控终端的图形用户界面的触控操作。
  3. 根据权利要求2所述的方法，其中，所述监听所述主控终端的图形用户界面的触控操作的步骤，包括：
    监听所述主控终端的字符设备文件；
    如果所述字符设备文件发生变化，则确定所述主控终端的图形用户界面发生触控操作。
  4. 根据权利要求1所述的方法，其中，通过所述主工具包将所述触控操作对应的操作事件转换成操作协议包的步骤包括：
    判断所述主控终端的多点触控协议类型是否为第二协议类型；
    如果是，基于所述主工具包和所述触控操作对应的操作事件生成操作协议包；
    其中，所述第二协议类型为有状态的多点触控协议类型。
  5. 根据权利要求4所述的方法，其中，通过所述主工具包将所述触控操作对应的操作事件转换成操作协议包的步骤，还包括：
    如果所述主控终端的多点触控协议类型为第一协议类型，将所述触控操作对应的操作事件转换为所述第二协议类型对应的操作事件，其中，所述第一协议类型为无状态的多点触控协议类型；
    基于所述主工具包和所述第二协议类型对应的操作事件生成操作协议包。
  6. 根据权利要求4或5所述的方法，其中，基于所述主工具包和所述触控操作对应的操作事件生成操作协议包的步骤包括：
    通过所述主工具包从所述主控终端的字符设备文件中提取操作事件；
    对提取到的所述操作事件中包含的坐标点和压力值进行归一化处理；
    根据归一化处理之后的所述操作事件生成操作协议包。
  7. 根据权利要求1所述的方法，其中，向所述从控终端发送的消息为携带有所述操作协议包的ZeroMQ消息。
  8. 一种一机多控方法，所述方法应用于从控终端，所述从控终端上部署有从工具包， 所述从控终端与主控终端连接，所述主控终端和所述从控终端均为安装有安卓系统的移动终端，所述主控终端上部署有主工具包，所述方法包括：
    接收主控终端发送的消息；所述消息携带有所述主控终端上的触控操作对应的操作协议包；
    从所述消息解析出所述操作协议包；
    基于所述操作协议包在所述从控终端的图形用户界面进行模拟操作。
  9. 根据权利要求8所述的方法，其中，基于所述操作协议包在所述从控终端的图形用户界面进行模拟操作的步骤包括：
    提取所述操作协议包中携带的操作事件，将所述操作事件写入到所述从控终端的字符设备文件，以对所述从控终端的字符设备文件进行更新；
    基于所述从工具包和更新的所述字符设备文件在所述从控终端的图形用户界面进行模拟操作。
  10. 根据权利要求9所述的方法，其中，提取所述操作协议包中携带的操作事件的步骤包括：
    获取所述从控终端的多点触控协议类型；
    如果所述多点触控协议类型为第二协议类型，则从所述操作协议包中提取所述第二协议类型对应的操作事件，其中，所述第二协议类型为有状态的多点触控协议类型。
  11. 根据权利要求10所述的方法，其中，提取所述操作协议包中携带的操作事件的步骤还包括：
    如果所述多点触控协议类型为第一协议类型，则从所述操作协议包中提取所述第一协议类型对应的操作事件，其中，所述第一协议类型为无状态的多点触控协议类型。
  12. 根据权利要求8所述的方法，其中，基于所述操作协议包在所述从控终端的图形用户界面进行模拟操作的步骤还包括：
    如果所述从控终端的安卓系统的版本为满足预设版本的操作系统，获取所述从控终端的图形用户界面的旋转方向；
    根据所述旋转方向对接收到的所述操作协议包进行坐标转换，并调用预设的应用程序以在所述从控终端的图形用户界面上对转换之后的所述操作协议包进行模拟操作。
  13. 根据权利要求8所述的方法，其中，所述方法还包括：
    响应针对于所述从工具包的启动指令，监听所述主控终端发送的所述消息。
  14. 根据权利要求8所述的方法，其中，接收的所述主控终端发送的消息为携带有所述操作协议包的ZeroMQ消息。
  15. 一种一机多控装置，所述装置应用于主控终端，所述主控终端上部署有主工具包，所述主控终端与至少一从控终端连接，所述主控终端和所述从控终端均为安装有安卓系统的移动终端，所述从控终端上部署有从工具包，所述装置包括：
    转换模块，用于响应所述主控终端的图形用户界面的触控操作，并通过所述主工具 包将所述触控操作对应的操作事件转换成操作协议包；
    发送模块，用于向从控终端发送消息，所述消息携带有所述操作协议包，以触发所述从控终端通过所述从工具包基于所述操作协议包执行所述触控操作。
  16. 一种一机多控装置，所述装置应用于从控终端，所述从控终端上部署有从工具包，所述从控终端与主控终端连接，所述主控终端和所述从控终端均为安装有安卓系统的移动终端，所述主控终端上部署有主工具包，所述装置包括：
    接收模块，用于接收主控终端发送的消息；其中，所述消息携带有所述主控终端上的触控操作对应的操作协议包；
    解析模块，用于从所述消息解析出所述操作协议包；
    模拟操作模块，用于基于所述操作协议包在所述从控终端的图形用户界面进行模拟操作。
  17. 一种一机多控系统，所述系统包括服务设备，还包括主控终端和从控终端，其中，所述服务设备与所述主控终端和所述从控终端通信连接；
    所述主控终端配置有权利要求15所述的装置，所述从控终端配置有权利要求16所述的装置；
    所述服务设备用于向所述主控终端推送主工具包，向所述从控终端推送从工具包；以及，启动所述主工具包和所述从工具包，以实现所述从控终端同步模拟所述主控终端的触控操作。
  18. 一种电子设备，包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现上述权利要求1-14任一项所述的方法。
  19. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器运行时执行上述权利要求1-14任一项所述的方法。
PCT/CN2020/125574 2020-06-10 2020-10-30 一机多控方法、装置、系统及电子设备 WO2021248788A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/001,077 US20230214243A1 (en) 2020-06-10 2020-10-30 One-machine multi-control method, apparatus, system and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010526846.4A CN111651232A (zh) 2020-06-10 2020-06-10 一机多控方法、装置、系统及电子设备
CN202010526846.4 2020-06-10

Publications (1)

Publication Number Publication Date
WO2021248788A1 true WO2021248788A1 (zh) 2021-12-16

Family

ID=72351699

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/125574 WO2021248788A1 (zh) 2020-06-10 2020-10-30 一机多控方法、装置、系统及电子设备

Country Status (3)

Country Link
US (1) US20230214243A1 (zh)
CN (1) CN111651232A (zh)
WO (1) WO2021248788A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111651232A (zh) * 2020-06-10 2020-09-11 网易（杭州）网络有限公司 一机多控方法、装置、系统及电子设备
CN112636966A (zh) * 2020-12-18 2021-04-09 中国人民解放军国防科技大学 一种基于远程调试协议的应用批量部署方法
CN113467694A (zh) * 2021-07-09 2021-10-01 网易（杭州）网络有限公司 设备控制方法、装置、控制终端、主受控设备及介质
CN113760750B (zh) * 2021-08-13 2024-04-02 深圳Tcl新技术有限公司 一种智能设备测试方法、装置、电子设备和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102123132A (zh) * 2010-01-08 2011-07-13 阿尔卡特朗讯 为用户终端提供通信的方法及装置
US20140055360A1 (en) * 2012-08-21 2014-02-27 Hon Hai Precision Industry Co., Ltd. Input system and method for computer
CN105808425A (zh) * 2016-02-24 2016-07-27 北京奇虎科技有限公司 一种实现移动端应用测试的方法、装置和系统
CN105975367A (zh) * 2016-04-29 2016-09-28 腾讯科技（深圳）有限公司 移动设备的测试方法和装置
CN111651232A (zh) * 2020-06-10 2020-09-11 网易（杭州）网络有限公司 一机多控方法、装置、系统及电子设备

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2203803A1 (en) * 2007-09-14 2010-07-07 Panasonic Avionics Corporation Portable user control device and method for vehicle information systems

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102123132A (zh) * 2010-01-08 2011-07-13 阿尔卡特朗讯 为用户终端提供通信的方法及装置
US20140055360A1 (en) * 2012-08-21 2014-02-27 Hon Hai Precision Industry Co., Ltd. Input system and method for computer
CN105808425A (zh) * 2016-02-24 2016-07-27 北京奇虎科技有限公司 一种实现移动端应用测试的方法、装置和系统
CN105975367A (zh) * 2016-04-29 2016-09-28 腾讯科技（深圳）有限公司 移动设备的测试方法和装置
CN111651232A (zh) * 2020-06-10 2020-09-11 网易（杭州）网络有限公司 一机多控方法、装置、系统及电子设备

Also Published As

Publication number Publication date
CN111651232A (zh) 2020-09-11
US20230214243A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
WO2021248788A1 (zh) 一机多控方法、装置、系统及电子设备
US20180365133A1 (en) Continuous integration testing for network-based applications
CN105487966B (zh) 程序测试方法、装置及系统
WO2019127476A1 (zh) 虚拟系统蓝牙通信方法及装置、虚拟系统、存储介质及电子设备
US20150236799A1 (en) Method and system for quick testing and detecting mobile devices
CN108959199B (zh) 一种日志的突出显示方法、装置、存储介质及安卓终端
CN104063305A (zh) 智能终端软件的压力测试方法、装置和系统
TW201518933A (zh) 測試裝置及其測試方法
CN108733403B (zh) 一种bios设置方法和装置
WO2021129335A1 (zh) 操作监控方法及装置、操作分析方法及装置
CN112231206A (zh) 应用程序测试的脚本编辑方法、计算机可读存储介质及测试平台
CN114448823A (zh) Nfs服务测试方法、系统及电子设备
WO2014000299A1 (zh) 串口重定向处理方法、设备和系统
CN112154417A (zh) 经由仿真网络通信信道在应用的单机版本和基于Web的版本之间共享代码库
CN109144586A (zh) 一种驱动适配方法、装置、计算机设备及存储介质
KR20220060429A (ko) 원격지의 네트워크 스위치에 대한 로그 수집 시스템 및 이의 빅데이터 구축 방법
CN112416698A (zh) 监控系统的扩展方法及装置、存储介质及电子设备
CN111651308A (zh) Dp转hdmi芯片的调试数据获取方法、装置和智能设备
WO2023098015A1 (zh) 芯片平台升级方法及相关装置
CN114171107A (zh) 固态硬盘vpd信息的检测方法、装置、设备及存储介质
CN114003443A (zh) 一种硬盘性能自动测试方法、装置、终端及存储介质
US20150143012A1 (en) Method for configuring maximum transmission unit (mtu), terminal and usb data card
WO2019127475A1 (zh) 虚拟sim卡的实现方法及装置、存储介质、电子设备
TWI855846B (zh) 伺服器測試系統、方法、電子設備及電腦可讀儲存介質
CN114371978B (zh) 安卓获取数据的方法、设备、装置和系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20939887

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20939887

Country of ref document: EP

Kind code of ref document: A1