WO2014090016A1

WO2014090016A1 - Device control method and mobile communication terminal

Info

Publication number: WO2014090016A1
Application number: PCT/CN2013/084437
Authority: WO
Inventors: 陈虞付; 李超; 刘晓健; 吴军
Original assignee: 北京奇虎科技有限公司; 奇智软件（北京）有限公司
Priority date: 2012-12-12
Filing date: 2013-09-27
Publication date: 2014-06-19
Also published as: CN103051693A; CN103051693B

Abstract

Provided in the present invention are a device control method and a mobile communication terminal. The device control method comprises: the mobile communication terminal receives a control operation command of a user for performing a control operation on a designated device via a set HTML page, where the HTML page is for use in providing an interface for the user to perform the control operation on the designated device via the mobile communication terminal, receives the control operation command generated by the user by operating the interface, and transmits the control operation command; and the mobile communication device transmits the control operation command to the designated device via the HTML page, thus allowing the designated device to execute the control operation indicated by the control operation command. The effects of remote control of a device and enhanced user operating experience are achieved via the present invention.

Description

Device control method and mobile communication terminal

Technical field

The present invention relates to the field of computer technologies, and in particular, to a device control method and a mobile communication terminal. Background technique

With the development and development of the Internet, the electric computer brain plays a role in the life and life of the people, playing the role of the more and more important role. The living and living of people and the work and work are becoming more and more dependent on the computer brain. . In order to carry out the control system for the computer brain, the computer brains all have equipment for input and output, for example, one type of equipment is often used for input and input equipment. The keyboard keyboard and the mouse mouse are used to operate the computer brain. Among them, the keyboard keyboard is usually used as the input and input of the information, and the mouse mouse is used. Used as a choice or a part of the operation is often used;; Another type of input and use equipment is often used to make the use of the keyboard and touch touch panel The operation of the computer brain, a lot of used for portable portable computer brain, among which, the key keyboard communication is usually used as input and input of information, touch touch The slabs are used as an alternative or some of the often used operations. .

However, when using the first type of input and input device described above to perform the control system for the computer brain, the key keyboard and the mouse mouse are both physical physics. Li Jie Jie.. Quality, the space occupied by the space and the placement of the position of the placement is also required, and only can be close to the distance from the operation An electric computer brain;; when using the second and second types of input and input devices described above for the control of the computer brain, the key keyboard and the touch panel are both internal The built-in electric corroof is only able to work as a computer brain near a close distance, and it exists in the limitations of the work. .

Similarly, in the Internet of Things, it is often necessary to use a dedicated control device to control the control of the Internet of Things. , , but also exist in the position of the location requirements, only can be close to the distance to go into the operation. .

It can be seen that, the use of the existing input and input devices - the preparation of control and control of the power of the computer brain or the Internet of Things network equipment, all exist in a variety of species Restricted system, especially in particular, it is impossible to implement the current user control system in the remote computer network or the Internet of Things connection network at a relatively long distance. ,, the drop is lower, and the user is right.

Summary of the invention

In view of the above problem that the existing input device cannot realize the control of the computer or the Internet of Things device at a long distance, and reduces the operating experience of the household on the electric rot or the Internet of Things device, the present invention has been proposed to provide an overcoming of the above. A device control method that solves the above problems or at least partially solves the problem with a mobile communication terminal.

According to an aspect of the present invention, a device control method is provided, including: a mobile communication terminal The control operation instruction for controlling the operation of the specified device by the set HTML page, wherein the HTML page is used to provide an interface for the user to control the specified device through the mobile communication terminal, and receive the control operation generated by the user on the interface operation. And transmitting a control operation instruction; the mobile communication terminal sends the control operation instruction to the designated device through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

According to another aspect of the present invention, a mobile communication terminal is provided, including: a receiving module configured to receive a control operation instruction for a user to perform a control operation on a specified device through a set HTML page, where the HTML page is used to provide a user Receiving, by the mobile communication terminal, an operation operation command for the designated device, receiving a control operation instruction generated by the user for the interface operation, and transmitting a control operation instruction; and the control module is configured to send the control operation instruction to the designated device through the HTML page, so as to specify The device performs the control operations indicated by the control operation instructions.

According to still another aspect of the present invention, there is provided a computer readable recording medium having recorded thereon a program for executing the above device control method.

According to the device control scheme of the present invention, the mobile communication terminal receives the user's control operation on the designated device through the HTML page, and sends a control operation instruction to the designated device, so that the designated device performs the corresponding control operation according to the control operation instruction. The HTML page is widely used and supported by almost all operating systems. The HTML page is used to control the operation of the specified device by the mobile communication terminal. No special operation control software is required on the mobile communication terminal side and the designated device side, and the operation is convenient. Simple implementation, good compatibility, and low implementation cost. With the solution of the present invention, the operation of a designated device such as a PC or an Internet of Things device can be controlled by a mobile communication terminal at a long distance. The mobile communication terminal of the invention can replace the traditional input device, realizes convenient operation of the device away from the device, and monitors the device in real time, and comprehensively controls the device, thereby solving the problem that the existing input device cannot realize the user Long-distance distance control of computers or IoT devices reduces the user's experience of operating the computer or IoT devices, achieves long-distance manipulation of the device, and enhances the user's operating experience.

The above description is only an overview of the technical solutions of the present invention, and the technical means of the present invention can be more clearly understood, and can be implemented in accordance with the contents of the specification, and the above and other objects, features and advantages of the present invention can be more clearly understood. Specific embodiments of the invention are set forth below. DRAWINGS

Various other advantages and benefits will become apparent to those skilled in the art from a <RTIgt; The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts. In the drawing:

1 is a flow chart of steps of a device control method according to a first embodiment of the present invention; FIG. 2 is a flow chart of steps of a device control method according to a second embodiment of the present invention; FIG. 4 is a schematic diagram of an interface of a touch panel HTML page in the embodiment shown in FIG. 3; FIG. 5 is an interface of a shortcut operation HTML page in the embodiment shown in FIG. FIG. 6 is a schematic diagram of an interface for selecting a terminal HTML page in the embodiment shown in FIG. 3. FIG. 7 is a flow chart of steps of a device control method according to Embodiment 4 of the present invention; FIG. 8 is a flowchart according to the present invention. A structural block diagram of a mobile communication terminal of Example 5. DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS Exemplary embodiments of the present disclosure will be described more closely below with reference to the accompanying drawings. While the exemplary embodiments of the present invention are shown in the drawings, it is understood that Rather, these embodiments are provided so that this disclosure will be more fully understood, and the scope of the disclosure can be fully conveyed to those skilled in the art.

Embodiment 1

Referring to Fig. 1, there is shown a flow chart of the steps of a device control method according to a first embodiment of the present invention.

The device control method of this embodiment includes the following steps:

Step S102: The mobile communication terminal receives a control operation instruction for the user to perform a control operation on the designated device through the set HTML page.

The HTML page is used to provide an interface for the user to perform control operations on the designated device through the mobile communication terminal, receive a control operation instruction generated by the user for the interface operation, and send a control operation instruction. Step S104: The mobile communication terminal sends a control operation instruction to the designated device through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

In this embodiment, the mobile communication terminal receives the control operation of the designated device by the user through the HTML page, and sends a control operation instruction to the designated device, so that the designated device performs a corresponding control operation according to the control operation instruction. The HTML page is widely used and supported by almost all operating systems. The HTML page is used to control the operation of the specified device by the mobile communication terminal. No special operation control software is required on the mobile communication terminal side and the designated device side, and the operation is convenient. Simple implementation, good compatibility, and low implementation cost. With the solution of this embodiment, the operation of a designated device (e.g., a PC or an Internet of Things device) can be controlled by a mobile communication terminal at a long distance. The mobile communication terminal of this embodiment can replace the traditional input device, realizes convenient operation of the device away from the device, and monitors the device in real time, and comprehensively controls the device, thereby solving the existing input device. The user controls the electric rot or the Internet of Things device at a long distance, which reduces the user's operation experience on the computer or the Internet of Things device, and achieves the long-distance manipulation of the device and improves the user experience.

Embodiment 2

Referring to Figure 2, there is shown a flow chart of the steps of a device control method in accordance with a second embodiment of the present invention.

In this embodiment, a network control server is used to connect the mobile communication terminal with the designated device, and the WebSocket connection is established between the mobile communication terminal and the network control server, and between the network control server and the designated device.

The WebSocket protocol is HTML5, a new protocol that implements full-duplex communication between the browser and the server. The WebSocket protocol is relatively simple. The device and the normal browser use the port 80 or port 443 to perform a request handshake. The server identifies whether it is a WebSocket request according to the HTTP HEADER. If yes, the request is upgraded to a WebSocket connection. After the handshake is successful. It enters the data transfer phase of the two-way long connection.

When using the WebSocket protocol, the specified device side can set NPAPI. The NPAPK Netscape Plugin Application Programming Interface is a generic interface for executing external applications in a browser. Each plugin in the Netscape plugin declares itself Holding one or several content types, when the browser encounters this type of content, it looks for a plugin that can be decoded in the plugin library, and distributes the content to it, and in the corresponding position of the webpage. display. Through the NPAPI in the browser, the corresponding API (Application Programming Interface) in the system can be called to implement corresponding functions, such as shutting down, adjusting the volume, controlling the mouse operation and the like. For example, the browser provides SetCursorPos, Shutdown, Snapshot, MouseDown to the javascript script in the extension web page through the NPAPI mechanism (currently the mechanism has been applied to Chrome, browsers based on Chromium open source projects and browsers such as Firefox). , MouseUp, KeyDown, KeyUp and other interface functions, javascript communicate with the server through websocket, after receiving the control command, call these javascript interface functions extended by NPAPI, realize the control cursor position, simulate mouse click and keyboard press, computer Snapshot, computer shutdown and other functions.

Based on the above settings, the device control method of this embodiment includes the following steps:

Step S202: The mobile communication terminal acquires information of at least one designated device to be controlled from the network control server.

The network control server may pre-store information of one or more designated devices to be controlled, and send information of the designated devices to be controlled to the mobile communication terminal upon receiving the request of the mobile communication terminal; the network control server may also be Upon receiving the request from the mobile communication terminal, the information is transmitted to the designated device to which it is connected, and information of one or more designated devices to be controlled is acquired, and then returned to the mobile communication terminal.

In this embodiment, the mobile communication terminal sends a request to the network control server through the WebSocket connection, and acquires information of at least one designated device to be controlled from the network control server through the WebSocket connection. That is, the communication between the mobile communication terminal and the network control server follows the WebSocket protocol. Similarly, there is a WebSocket connection between the network control server and the specified device, and the two also follow the WebSocket protocol for communication.

Preferably, when the mobile communication terminal acquires information of the at least one designated device to be controlled from the network control server, the information of the specified device includes at least one of the following: a two-dimensional code of the specified device, an identifier of the designated device, and an IP address of the designated device; The mobile communication terminal recognizes and displays the designated device by specifying the information of the device. Or, the network control server knows according to the information of the designated device to be controlled The device does not specify a device, and the information of the specified device includes at least one of the following: a two-dimensional code of the specified device, an identifier of the designated device, and an IP address of the designated device; the mobile communication terminal acquires the specified device identified by the network control server from the network control server. Information.

Step S204: The mobile communication terminal selects a designated device to be controlled according to the user input, and sends the authentication information to the network control server.

The authentication information is used to request authorization for the control operation of the selected designated device.

Step S206: The mobile communication terminal receives the confirmation authorization information returned by the network control server. By authorizing the authentication, it is possible to ensure the security of the operation control of the device by the mobile communication terminal.

Step S208: The mobile communication terminal acquires an HTML page of the selected designated device.

The mobile communication terminal can obtain the HTML page of the selected specified device in a variety of ways. For example, the mobile communication terminal sends a message acquisition command to the network control server by using a WebSocket connection, where the information acquisition command is used to obtain a control operation supported by the specified device; the network control server uses the WebSocket connection to send the information acquisition command to the designated device, so that the specified device Obtaining, by the NPAPI function, the control operation supported by the specified device and returning to the network control server, and forwarding the information to the mobile communication terminal by the network control server according to the information acquisition command; the mobile communication terminal according to the control operation information supported by the specified device returned by the network control server, The controls corresponding to the control operations supported by the specified device are displayed in the HTML page, and finally the HTML page corresponding to the specified device is generated.

For another example, the mobile communication terminal sends a message acquisition command to the network control server by using a WebSocket connection, where the information acquisition command is used to obtain a control operation supported by the specified device; the network control server sends the information acquisition command to the designated device by using a WebSocket connection, so that the specified The device obtains the control operation supported by the specified device through the NPAPI function and returns it to the network control server according to the information acquisition command. The network control server generates a corresponding HTML page according to the information of the control operation supported by the specified device, and returns the corresponding HTML page to the mobile communication terminal.

For another example, the mobile communication terminal is pre-built with set HTML pages, and the pre-built HTML pages include at least one of the following: a touchpad page, a shortcut operation page, and a selection terminal page, wherein the touchpad page is used to receive the user's touch The information of the touch operation of the board page generates a control operation instruction according to the information of the touch operation to control the cursor operation of the designated device; The page includes at least one of the following controls: a volume adjustment control, a device desktop display control, a device screen capture control, and a device shutdown control, and the shortcut operation page is configured to receive information about a user's operation on the control in the shortcut operation page, according to the operation of the control The information generation control operation instruction controls the designated device to perform the control operation indicated by the control operation instruction; the selection terminal page is used to display the information of the device to be controlled and the controlled state of the device to be controlled, so that the user selects the device to be controlled The Select Terminal page also includes an authorization password input control for the user to input an authorization password for control operation authentication. After the mobile communication terminal obtains the authorization of the designated device, the mobile communication terminal displays all or part of the above-prepared HTML pages in advance for the user to operate. It should be noted that, in practical applications, a person skilled in the art may construct one or more of the above HTML pages according to actual needs, or integrate one or more of the above HTML pages into one HTML page, and use an HTML page. It can realize the operation function of the specified device, and it is flexible and easy to expand.

Step S210: The mobile communication terminal receives a control operation instruction for the user to perform a control operation on the designated device through the set HTML page.

In this embodiment, the HTML page in the mobile communication terminal listens to the control operation performed by the user on the specified device through the HTML page, and obtains the information of the control operation through the DOM (Document Object Model) event of the HTML page, according to the control operation. The information generates control operations instructions.

The DOM is the standard programming interface for W3C organizations that handles markup languages. It is a platform- and language-independent application interface that dynamically accesses programs and scripts, updating their content, structure, and style of www documents. Event processing is the most important part of the DOM. The sophisticated event mechanism allows programs such as JavaScript programs to trigger different execution methods based on specific events, making the program more interactive and intelligent. For example, the mouse movement data on the HTML page can be obtained through the touchstart event, the touchmove event, and the touchend event.

Step S212: The mobile communication terminal uses a WebSocket connection to send a control operation instruction to the network control server through the HTML page.

Among them, the network control server can be built using Node.js. Node.js is a set of JavaScript toolkits for writing high-performance web servers with single-threaded, non-blocking 10, event-driven, and many more. Of course, not limited to this, the network control server can also use other suitable methods. Build, such as building an Apache (Apache) server. The network control server establishes a connection between the mobile communication terminal and the designated device controlled by the mobile communication terminal through the WebSocket protocol.

Step S214: The network control server forwards the control operation instruction to the designated device by using a WebSocket connection.

Step S216: The designated device receives the control operation instruction, and according to the control operation instruction, invokes the system API of the specified device through the NPAPI to implement the control operation indicated by the control operation instruction.

Preferably, a browser and a browser extension having an NPAPI mechanism are installed in the designated device of the embodiment. Among them, the browser extension includes an NPAPI plug-in (ie, a DLL file), the browser extension is used to implement WebSocket communication between the specified device and the network control server, and the system API of the specified device is invoked through the NPAPI plug-in.

The specified device listens to and receives the control operation instruction sent by the network control server through the HTML page of the browser extension; further, the HTML page of the browser extension calls the NPAPI function corresponding to the control operation indicated by the control operation instruction, and passes the NPAPI function. The corresponding system API function is called to implement the control operation indicated by the control operation instruction. For example, when the control operation indicated by the control operation instruction is a cursor movement operation, the NPAPI function called by the HTML page of the browser extension of the specified device is SetCursorPos, and the NPAPI function calls the mouse-event function of the system to realize the movement of the cursor.

Through the embodiment, the function of the mobile communication terminal as a remote controller is realized, such as a mobile phone remote controller, and the mobile communication terminal remote controller completes communication between the mobile communication terminal, such as a mobile phone and a designated device such as a computer, through a two-way communication mechanism of WebSocket, through NPAPI. The plugin mechanism invokes the system's API to execute user commands, thereby enabling full control of the specified device in the event that the user is completely detached from the traditional input device.

Embodiment 3

Referring to Fig. 3, there is shown a flow chart of the steps of a device control method according to a third embodiment of the present invention.

In this embodiment, the mobile communication terminal is a mobile phone, and the designated device is a personal computer as an example. The user controls the device control method of the present invention by performing a touch movement operation on the touch panel page of the mobile communication terminal to control the cursor operation of the designated device. Description. The device control method of this embodiment includes the following steps:

Step S302: Set up a network control server.

In this embodiment, a network control server is built by using Node.js. As an emerging back-end language, Node.js has many advantages, such as: RESTfol APL single-threaded, non-blocking 10, Google V8, event-driven, and more.

The construction of the network control server includes:

Step 1: Install Cygwin; then, download and install Node.js.

Step 2: Implement WebSocket communication with socket.io.js.

Socket.io.js is a js library file that uses the WebSocket specification in the HTML5 draft to implement two-way communication between the browser and the server. It provides functions such as sending a message (socket.emit ( )) and receiving information (socket.on ( ) ), broadcast information (socket.broadcast.emit ( )), etc. APL

The third step: realize the authorization and authentication of the mobile communication terminal.

Implementation mode: The designated device sends the controlled password to the network control server, and the network control server backs up the controlled password. When the control terminal, that is, the mobile communication terminal wants to control a specified device, it needs to input the controlled password and send it to the network control server. Verify that if the password is entered correctly, control is obtained.

Step 4: The network control server acquires the command of the designated device, the control terminal, that is, the mobile communication terminal, and implements the transfer.

The acquisition and transit of this step are completed through WebSocket communication.

Step S304: Construct an HTML page in the mobile communication terminal, configured to perform a control operation on the designated device by the user, and send a control operation instruction generated according to the control operation.

In this embodiment, the constructed HTML page includes a touch panel page, a shortcut operation page, and a selection terminal page.

The interface of the touchpad page is as shown in FIG. 4. The touchpad page is used to receive information of a touch operation of the user on the touchpad page, and generates a control operation instruction according to the information of the touch operation to control the cursor operation of the device. Control the movement of the specified device cursor by dragging the touch point of the touchpad page (such as the white point in Figure 4) and calculating the distance it can be dragged (the unit can be a pixel). The distance conversion method is Isometric conversion, that is, how much distance the touch point drags on the touch panel page of the mobile communication terminal such as the mobile phone, and how much distance the cursor on the computer moves. When implemented, the control terminal, that is, the mobile communication terminal acquires the movement information or the click event of the touch point through the DOM event of the HTML page (including touchstart, touchmove, touchend, etc.), and then sends the corresponding data to the network control by using the socket.emit ( ) method. The server, the network control server listens to the control terminal, that is, the data sent by the mobile communication terminal through the socket.on ( ) and sends the data to the specified device by using the socket.broadcast.emit ( ) method, and the browser extension of the specified device passes the socket.on ( Listening to the data sent by the network control server, and calling the relevant system APIs (such as: mouse_ event , keybd_ event , OpenProcessToken , LookupPrivilege Value , AdjustTokenPrivileges , ExitWindowsEx ) through the NPAPI plugin (a DLL of the browser extension) Operational actions. Furthermore, the distance conversion method can also be other suitable methods, such as proportional conversion.

The shortcut operation page is shown in FIG. 5, and is configured to receive information about the operation of the user on the shortcut operation page, and generate a control operation instruction according to the operation information of the control to control the device to perform the control operation indicated by the control operation instruction. In this embodiment, the shortcuts in the shortcut operation page are: a volume adjustment control, a device desktop display control, a device screen capture control, and a device shutdown control, that is, the shortcut operations performed on the device through the page include: adjusting the volume , display desktop, screen capture and shutdown. When implemented, the control terminal, that is, the shortcut operation of the mobile communication terminal, listens to the click event of each button, and then sends the corresponding information to the network control server, and the network control server sends the data to the designated device, and the specified device obtains the data according to the The type of data is invoked by the NPAPI plugin mechanism to call the corresponding system API.

In addition, the shortcut button of the embodiment is further provided with a selection terminal button control, and when the button is clicked, the selection terminal page is displayed, as shown in FIG. 6.

The interface of the terminal page is displayed as shown in FIG. 6 , and is used for displaying the information of the designated device to be controlled and the controlled state of the specified device to be controlled, so that the user selects the designated device to be controlled; The authorization password input control is used for the user to input an authorization password for control operation authentication. The Select Terminal page displays the specified device list obtained from the network control server and the controlled state of the specified device. After inputting the authorization password and verifying by the network control server, the user can obtain the control right of the selected designated device. When implemented, the specified device will have local information (for example, calculations) The machine name, IP address, controlled password, etc. are sent to the network control server, and the network control server can send related information (in addition to the controlled password) to the control terminal, that is, the mobile communication terminal, and the control terminal, that is, the mobile communication terminal selects and specifies The device (such as according to the IP address or computer name) and input the controlled password and send it to the network control server for verification. After the password is correct, the network control server returns the successful information, and the control terminal, that is, the mobile communication terminal obtains the control authority, and then the control terminal That is, the mobile communication terminal can control the designated device.

Step S306: Setting a specified device.

Specifies that the device has a browser and browser extension with NPAPI mechanism, and the browser extension program specifies that the device needs to rely on a browser extension (.crx, .xpi file). The background page of the extension program is an HTML page for communication with the network control server. The extension's NPAPI plugin is a DLL that completes the system API call.

When the control terminal, that is, the HTML page of the mobile communication terminal sends the control operation instruction to the network control server through the WebSocket, the network control server sends the control operation instruction to the designated device by using the WebSocket, so that the designated device accepts the instruction of the control terminal, that is, the mobile communication terminal. .

After specifying the control operation command sent by the network control server, the background page of the specified device browser extension parses the instruction and calls the related function of NPAPI, and then calls the system API (such as mouse_event, keybd_event, OpenProcessToken) by the NPAPI function. , LookupPrivilege Value, AdjustTokenPrivileges, ExitWindowsEx, etc.) Complete mouse clicks, move cursors, turn off events, and more. This is because the browser extension cannot directly call the system API, but can call the export function of the NPAPI plugin. The NPAPI plugin is a DLL that can directly call the system API, so that the specified device can receive the API through the NPAPI call system. The command.

It should be noted that the execution between the foregoing steps S302, S304, and S306 may be performed in no particular order.

After the mobile communication terminal, the network control server, and the designated device complete the foregoing corresponding settings, the mobile communication terminal can start the operation control of the designated device, including:

Step S308: The mobile communication terminal acquires a list of designated devices (PCs).

The specified device sends the information of the machine (such as computer name, IP address, controlled password, etc.) to the network. The network control server sends the related information of the specified device (in addition to the controlled password) to the control terminal, that is, the mobile communication terminal.

Step S310: The mobile communication terminal selects a designated device (PC) and performs authorization authentication.

The designated device sends the controlled password to the network control server, and the control terminal is the mobile communication terminal.

The authorization password input control of the HTML page, such as the input control, enters a password and sends it to the network control server for verification. If the entered password is correct, it is authorized.

Step S312: The mobile communication terminal moves the cursor through the touch area of the touch panel page to generate a control operation instruction.

When the user drags the touch point in the touch area of the touch panel page to move the cursor, the DOM event of the touch panel page acquires the movement information and the click event of the touch point, thereby generating a control operation instruction.

Similarly, when the user initiates an operation command (such as adjusting the volume, displaying the desktop, taking a screenshot, and shutting down) through the button of the shortcut operation page, the DOM event of the shortcut operation page acquires the button click information, thereby generating a control operation instruction.

Step S314: The mobile communication terminal communicates with the network control server by using a WebSocket connection, and sends a control operation instruction to the network control server through the touchpad page.

Step S316: The network control server communicates with the designated device by using a WebSocket connection, and sends a control operation instruction to the designated device.

Step S318: The specified device obtains a control operation instruction sent by the network control server through a background page of the browser extension program, and invokes a related NPAPI function.

For example, when a control operation instruction indicates a cursor operation, the device calls the NPAPI function SetCursorPos.

Step S320: The designated device invokes the API function of the system through the NPAPI function, and performs a control operation indicated by the control operation instruction.

For example, the API function mouse_ event corresponding to the NPAPI function in step S318 is called to implement the cursor operation of the specified device.

In this embodiment, the device control method of the present invention is described by taking the cursor control operation of the designated device as an example. However, it should be understood by those skilled in the art that other control operations such as volume control, shutdown operation, and the like can be performed with reference to the present embodiment. With the embodiment, the mobile communication terminal, such as a mobile phone, is implemented. A computer input device such as a traditional keyboard, touchpad, and mouse that conveniently operates the computer away from the computer and monitors the computer's functions in real time. In this embodiment, the new web standard and the characteristics of the mobile communication terminal, such as the mobile phone, are adopted, and the mobile communication terminal such as the mobile phone is used as the input device of the computer, which greatly improves the user's access to the computer while maximizing the convenience of the computer input. Operating experience.

Embodiment 4

Referring to Figure 7, there is shown a flow chart of the steps of a device control method according to a fourth embodiment of the present invention.

The device control method of this embodiment can be applied to the control of the IoT terminal device by the mobile communication terminal.

The device control method of this embodiment includes the following steps:

Step S402: The mobile communication terminal acquires information of at least one specified device within a certain range and displays it for the user to select the designated device.

For example, the mobile communication terminal transmits a corresponding request to a designated device within a certain distance through a specific connection such as Bluetooth to obtain information of the surrounding designated device. For example, the mobile communication terminal can search for multiple IoT devices (such as smart TVs, smart refrigerators, etc.) in the same room through Bluetooth, obtain information of these devices, and then display them in the mobile communication terminal for the user to select and control. Specify the device.

Step S404: The mobile communication terminal receives the user's selection, determines the designated device, and sends a pairing request to the designated device, requesting to establish a pairing connection with the designated device.

The pairing request includes information of the mobile communication terminal. Preferably, the information of the mobile communication terminal comprises at least one of the following: a two-dimensional code of the mobile communication terminal, an identification of the mobile communication terminal, and an IP address of the mobile communication terminal.

Step S406: The mobile communication terminal receives the pairing response returned by the designated device and carries the information of the designated device, and establishes a pairing connection with the designated device according to the pairing response.

Preferably, the information of the designated device comprises at least one of the following: a two-dimensional code of the specified device, an identifier of the designated device, and an IP address of the designated device.

Step S408: The mobile communication terminal acquires an HTML page corresponding to the designated device. For example, after establishing a pairing connection with the designated device, the mobile communication terminal sends an information acquisition command to the designated device through the pairing connection to request to acquire the control operation supported by the designated device; and further, according to the information of the supported control operations returned by the specified device. Generate an HTML page.

For another example, the mobile communication terminal stores the matching with one or more types of designated devices.

The HTML page; after obtaining the information of the specified device, the mobile communication terminal determines the type of the specified device according to the information, and then selects the matching HTML page.

For another example, in a case where some servers can also participate, the mobile communication terminal sends the obtained information of the specified device to the server; the server determines the type of the specified device, and generates

The HTML page is sent to the mobile communication terminal, or the server returns an operation supported by the designated device of the type to the mobile communication terminal according to the type of the designated device, and the corresponding HTML page is generated by the mobile communication terminal.

Taking a smart TV as an example, the mobile communication terminal generates a corresponding information according to the information returned by the smart TV.

The HTML page, which can include a switch button control, a volume adjustment control, a channel adjustment control, a menu operation control, etc., through which the remote control of the smart TV can be realized.

Step S410: The mobile communication terminal receives a control operation instruction that the user performs a control operation on the designated device through the HTML page.

Step S412: The mobile communication terminal sends the control operation instruction to the designated device successfully paired by using the paired connection through the HTML page.

Step S414: The designated device receives a control operation instruction sent by the mobile communication terminal through the pairing connection, and performs a control operation indicated by the control operation instruction.

With this embodiment, point-to-point device control is achieved. Only one mobile communication terminal can realize the control of a variety of designated devices, without special remote control or control software, which realizes compatible operation of multiple devices in the case of far away from the device, and improves the user experience.

Embodiment 5

Referring to Figure 8, there is shown a block diagram of a structure of a mobile communication terminal in accordance with a fifth embodiment of the present invention. The apparatus for performing device control is provided in the mobile communication terminal of the embodiment, and includes: a receiving module 502 configured to receive a control operation of the specified device by the user through the set HTML page. Controlling an operation instruction, wherein the HTML page is configured to provide an interface for the user to perform a control operation on the designated device through the mobile communication terminal, receive a control operation instruction generated by the user for the interface operation, and send a control operation instruction; the control module 504 is configured to pass The HTML page sends a control operation instruction to the designated device, causing the designated device to perform the control operation indicated by the control operation instruction.

Preferably, the control module 504 is configured to use a WebSocket connection to send a control operation instruction to the network control server through the HTML page, and the network control server uses the WebSocket connection to forward the control operation instruction to the designated device, so that the designated device according to the control operation instruction. The control function indicated by the control operation instruction is implemented by calling the system API of the specified device through the NPAPI.

Preferably, the designated device is installed with a browser and a browser extension with NPAPI, wherein the browser extension includes an NPAPI plug-in, the browser extension is used to implement WebSocket communication between the specified device and the network control server, and is invoked through the NPAPI plug-in. Specifies the system API of the device.

Preferably, the control module 504 causes the designated device to monitor and receive the network control through the HTML page of the browser extension program when the specified device performs the control operation indicated by the control operation instruction by using the system API of the specified device according to the control operation instruction. The control operation instruction sent by the server; the HTML page of the browser extension program invokes an NPAPI function corresponding to the control operation indicated by the control operation instruction, and invokes the corresponding system API function through the NPAPI function to implement the control operation indicated by the control operation instruction .

Preferably, the mobile communication terminal of this embodiment further includes: a selection module 506 configured to acquire at least one from the network control server before the receiving module 504 receives a control operation instruction that the user controls the specified device through the set HTML page. Information of the designated device to be controlled; selecting a designated device to be controlled, and transmitting authentication information to the network control server, the authentication information is used to request authorization for the control operation of the selected designated device; and receiving the confirmation authorization information returned by the network control server.

Preferably, the selecting module 506 is further configured to: after receiving the confirmation authorization information returned by the network control server, send a message acquisition command to the network control server by using a WebSocket connection, where the information acquisition command is used to obtain a control operation supported by the designated device; The control server sends a message acquisition command to the specified device by using a WebSocket connection, so that the specified device obtains the information according to the information. The command obtains the control operation supported by the specified device through the NPAPI function and returns it to the network control server, which is forwarded to the selection module by the network control server; the selection module displays the information in the HTML page according to the information of the control operation supported by the specified device returned by the network control server. A control that corresponds to the control operations supported by the specified device.

Preferably, the selecting module 506 acquires information of at least one designated device to be controlled from the network control server when acquiring information of the at least one specified device to be controlled from the network control server, where the information of the specified device includes at least one of the following: Specifying the QR code of the device, the identifier of the specified device, and the IP address of the specified device; the mobile communication terminal identifies and displays the designated device by specifying the information of the device; or the network control server identifies the designated device according to the information of the designated device to be controlled; The mobile communication terminal acquires information of the designated device identified by the network control server from the network control server.

Preferably, the network control server is built using Node.js.

Preferably, the control module 504 is configured to send a control operation instruction to the designated device successfully paired with the mobile communication terminal through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

Preferably, the mobile communication terminal of this embodiment further includes: a pairing module 508, configured to send a pairing request to the designated device before the receiving module 502 receives a control operation instruction that the user controls the specified device through the set HTML page, And requesting to establish a pairing connection with the specified device, where the pairing request includes information of the mobile communication terminal; receiving a pairing response returned by the specified device and carrying the information of the designated device, and establishing a pairing connection with the designated device according to the pairing response.

Preferably, the information of the mobile communication terminal comprises at least one of: a two-dimensional code of the mobile communication terminal, an identifier of the mobile communication terminal, an IP address of the mobile communication terminal; the information of the designated device includes at least one of the following: Code, the identifier of the specified device, and the IP address of the specified device.

Preferably, the mobile communication terminal of this embodiment further includes: a building module 510 configured to construct a set HTML page before the receiving module 502 receives a control operation instruction that the user controls the specified device through the set HTML page. The HTML page includes at least one of the following: a touchpad page, a shortcut operation page, and a selection terminal page, wherein the touchpad page is configured to receive information of a touch operation of the user on the touchpad page, and generate a control operation finger according to the information of the touch operation. To control the cursor operation of the specified device; the shortcut operation page includes at least one of the following controls: a volume adjustment control, a device desktop display control, a device screen capture control, and a device shutdown control, and the shortcut operation page is used to receive the user in the shortcut operation page. The information of the operation of the control generates a control operation instruction according to the information of the operation of the control to control the control operation indicated by the specified operation instruction by the designated device; the terminal page is configured to display the information of the device to be controlled and the device to be controlled Control state, for the user to select the device to be controlled, the selection terminal page also includes an authorization password input control for the user to input an authorization password for control operation authentication.

Preferably, the receiving module 502 is configured to monitor, by the set HTML page, a control operation performed by the user on the specified device through the HTML page, obtain information of the control operation through the DOM event of the HTML page, and generate a control operation instruction according to the information of the control operation.

The mobile communication terminal in this embodiment is used to implement the corresponding device control method in the foregoing method embodiments, and has the beneficial effects of the corresponding method embodiments, and details are not described herein again. The present invention also provides a computer readable recording medium having recorded thereon a program for executing the control method of any of the foregoing embodiments 1 to 4. The computer readable recording medium includes any mechanism for storing or transmitting information in a computer readable form. For example, a machine-readable medium includes a read only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash storage medium, an electrical, optical, acoustic, or other form of propagated signal (eg, a carrier wave) , infrared signals, digital signals, etc.). The device control schemes provided herein are not inherently related to any particular computer, virtual system, or other device. Various general purpose systems can also be used with the teaching based on the teachings herein. From the above description, it is apparent that the structure required to construct the system having the solution of the present invention is apparent. Moreover, the invention is not directed to any particular programming language. It will be appreciated that the teachings of the invention described herein may be implemented in a variety of programming languages, and that the description of the specific language above is for the purpose of illustrating the preferred embodiments of the invention.

Numerous specific details are set forth in the description provided herein. However, it is understood that the embodiments of the invention may be practiced without these specific details. In some instances, it is not Well-known methods, structures, and techniques are shown in detail so as not to obscure the description.

Similarly, the various features of the present invention are sometimes grouped together into a single embodiment, in the above description of the exemplary embodiments of the invention, Figure, or a description of it. However, the method disclosed is not to be interpreted as reflecting the intention that the claimed invention requires more features than those recited in the appended claims. Rather, as the following claims reflect, inventive aspects reside in less than all features of the single embodiments disclosed. Therefore, the claims following the specific embodiments are hereby explicitly incorporated into the specific embodiments, and each of the claims as a separate embodiment of the invention.

Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively changed and placed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and further, they may be divided into a plurality of sub-modules or sub-units or sub-assemblies. In addition to the fact that at least some of such features and/or processes or units are mutually exclusive, any combination of the features disclosed in the specification, including the accompanying claims, the abstract and the drawings, and any methods so disclosed may be employed. Or combine all the processes or units of the device. Each feature disclosed in the specification (including the accompanying claims, the abstract, and the drawings) may be replaced by alternative features that provide the same, equivalent, or similar purpose, unless stated otherwise.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features that are included in other embodiments and not in other features, combinations of features of different embodiments are intended to be within the scope of the present invention. Different embodiments are formed and formed. For example, in the claims, any one of the claimed embodiments can be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components of the device control scheme in accordance with embodiments of the present invention. The invention can also be implemented as a device or device program (e.g., a computer program and a computer program product) for performing some or all of the methods described herein. Such an implementation of the invention The program can be stored on a computer readable medium or can be in the form of one or more signals. Such signals may be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

It is to be noted that the above-described embodiments are illustrative of the present invention and are not intended to limit the invention, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as a limitation. The word "comprising" does not exclude the presence of the elements or steps that are not recited in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means can be embodied by the same item. The use of the words first, second, and third does not indicate any order. These words can be interpreted as names.

Claims

claims

1. A device control method, including:

The mobile communication terminal receives control operation instructions for the user to control the designated device through the set HTML page, wherein the HTML page is used to provide an interface for the user to control the designated device through the mobile communication terminal. , receiving the control operation instructions generated by the user's operation on the interface, and sending the control operation instructions;

The mobile communication terminal sends the control operation instruction to the designated device through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

2. The method according to claim 1, wherein the mobile communication terminal sends the control operation instruction to the designated device through the HTML page, so that the designated device executes the instruction indicated by the control operation instruction. The steps to control operations include:

The mobile communication terminal uses a WebSocket connection to send the control operation instructions to the network control server through the HTML page, and the network control server forwards the control operation instructions to the designated device using the WebSocket connection, The designated device is caused to call the system API of the designated device through NPAPI according to the control operation instruction to implement the control operation indicated by the control operation instruction.

3. The method according to claim 2, wherein the designated device is installed with a browser and a browser extension program having NPAPI, wherein the browser extension program includes an NPAPI plug-in, and the browser extension program is used to Implement WebSocket communication between the designated device and the network control server, and call the system API of the designated device through the NPAPI plug-in.

4. The method according to claim 3, wherein the step of the designated device calling the system API of the designated device through NPAPI to implement the control operation indicated by the control operation instruction according to the control operation instruction includes:

The designated device monitors and receives the control operation instructions sent by the network control server through the HTML page of the browser extension;

The HTML page of the browser extension calls the NPAPI function corresponding to the control operation indicated by the control operation instruction, and calls the corresponding system API function through the NPAPI function to realize the control indicated by the control operation instruction. operate.

5. The method according to claim 1, wherein: receiving user information at the mobile communication terminal Before the steps of control operation instructions for controlling the specified device through the set HTML page, it also includes:

The mobile communication terminal obtains information about at least one designated device to be controlled from the network control server;

Select one of the designated devices to be controlled, and send authentication information to the network control server, where the authentication information is used to request control operation authorization for the selected designated device;

Receive the confirmation authorization information returned by the network control server.

6. The method according to claim 5, wherein after receiving the confirmation authorization information returned by the network control server, further comprising:

The mobile communication terminal uses a WebSocket connection to send an information acquisition command to the network control server, where the information acquisition command is used to obtain the control operations supported by the designated device;

The network control server uses the WebSocket connection to send the information acquisition command to the designated device, so that the designated device obtains the command according to the information, obtains the control operations supported by the designated device through the NPAPI function and returns it to The network control server forwards it to the mobile communication terminal by the network control server;

The mobile communication terminal displays controls corresponding to the control operations supported by the specified device in the HTML page based on the information about the control operations supported by the specified device returned by the network control server.

7. The method according to claim 5 or 6, wherein the step of the mobile communication terminal obtaining the information of at least one designated device to be controlled from the network control server includes:

The mobile communication terminal obtains information about at least one designated device to be controlled from the network control server, wherein the information about the designated device includes at least one of the following: the QR code of the designated device, the designated The identification of the device and the IP address of the designated device; The mobile communication terminal identifies and displays the designated device through the information of the designated device;

or,

The network control server identifies the designated device according to the information of the designated device to be controlled; the mobile communication terminal obtains the network control server from the network control server Information about the identified specified device.

8. The method according to claim 2, wherein the network control server is built using Node.js.

9. The method according to claim 1, wherein the mobile communication terminal sends the control operation instruction to the designated device through the HTML page, so that the designated device executes the control operation instruction. The steps to control operations include:

The mobile communication terminal sends the control operation instruction to the designated device that is successfully paired with the mobile communication terminal through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

10. The method according to claim 9, wherein before the step of the mobile communication terminal receiving the control operation instruction from the user to control the designated device through the set HTML page, it further includes:

The mobile communication terminal sends a pairing request to the designated device, requesting to establish a pairing connection with the designated device, wherein the pairing request includes information about the mobile communication terminal; the mobile communication terminal receives the designation The device returns a pairing response that carries information about the specified device, and establishes a pairing connection with the specified device based on the pairing response.

11. The method according to claim 10, wherein the information of the mobile communication terminal includes at least one of the following: a QR code of the mobile communication terminal, an identification of the mobile communication terminal, an identification number of the mobile communication terminal IP address; The information of the designated device includes at least one of the following: the QR code of the designated device, the identification of the designated device, and the IP address of the designated device.

12. The method according to claim 1, wherein before the step of the mobile communication terminal receiving the control operation instruction from the user to control the designated device through the set HTML page, it further includes:

Construct the HTML page of the setting, and the HTML page includes at least one of the following: a touch panel page, a shortcut operation page and a terminal selection page, wherein the touch panel page is used to receive the user's input on the touch panel page. The touch operation information is used to generate the control operation instructions according to the touch operation information to control the cursor operation of the specified device; the shortcut operation page includes at least one of the following controls: volume adjustment control, device desktop display control , device screenshot control and a device shutdown control, the shortcut operation page is used to receive the information about the user's operation of the control in the shortcut operation page, and generate the control operation instruction according to the information about the operation of the control to control the specified The device performs the control operation indicated by the control operation instruction; the selection terminal page is used to display the information of the device to be controlled and the controlled status of the device to be controlled, so that the user can select the device to be controlled. Equipment, the terminal selection page also includes an authorization password input control for the user to input the authorization password for control operation authentication.

13. The method according to claim 1 or 2, wherein the step of the mobile communication terminal receiving a control operation instruction from a user to control a designated device through a set HTML page includes:

The set HTML page monitors the control operation performed by the user on the designated device through the HTML page, obtains the information of the control operation through the DOM event of the HTML page, and generates information based on the information of the control operation. The control operation instructions.

14. A mobile communication terminal, including:

The receiving module is configured to receive control operation instructions for the user to control the designated device through the set HTML page, wherein the HTML page is used to provide the user to control the designated device through the mobile communication terminal. The interface, receives the control operation instructions generated by the user's operation on the interface, and sends the control operation instructions;

The control module is configured to send the control operation instruction to the designated device through the HTML page, so that the designated device performs the control operation indicated by the control operation instruction.

15. The mobile communication terminal according to claim 14, wherein the control module is configured to use a WebSocket connection to send the control operation instructions to the network control server through the HTML page, and is used by the network control server The WebSocket connection forwards the control operation instruction to the designated device, so that the designated device calls the system API of the designated device through NPAPI to implement the control operation indicated by the control operation instruction according to the control operation instruction. .

16. The mobile communication terminal according to claim 15, wherein the designated device is installed with a browser and a browser extension program having NPAPI, wherein the browser extension program includes an NPAPI plug-in, and the browser extension program Used to implement the specified equipment and the network control service WebSocket communication of the server, and calling the system API of the specified device through the NPAPI plug-in.

17. The mobile communication terminal according to claim 16, wherein the control module causes the designated device to call the system API of the designated device through NPAPI according to the control operation instruction to implement the instructions indicated by the control operation instruction. During the control operation, the designated device is caused to monitor and receive the control operation instruction sent by the network control server through the HTML page of the browser extension program; the HTML page call of the browser extension program is related to the control operation. The control operation indicated by the operation instruction corresponds to the NPAPI function, and the corresponding system API function is called through the NPAPI function to implement the control operation indicated by the control operation instruction.

18. The mobile communication terminal according to claim 14, wherein the mobile communication terminal further includes:

The selection module is configured to obtain the information of at least one designated device to be controlled from the network control server before the receiving module receives the control operation instruction of the user to control the designated device through the set HTML page; select one of the designated devices. Describe the designated device to be controlled, and send authentication information to the network control server, where the authentication information is used to request control operation authorization for the selected designated device; receive confirmation authorization information returned by the network control server.

19. The mobile communication terminal according to claim 18, wherein the selection module is further configured to use a WebSocket connection to send information to the network control server after receiving the confirmation authorization information returned by the network control server. Obtain a command, wherein the information acquisition command is used to obtain the control operations supported by the specified device; the network control server uses the WebSocket connection to send the information acquisition command to the specified device, so that the specified device According to the information acquisition command, the control operations supported by the specified device are obtained through the NPAPI function and returned to the network control server, which is forwarded to the selection module by the network control server; the selection module is based on the network control The information about the control operations supported by the specified device returned by the server displays controls corresponding to the control operations supported by the specified device in the HTML page.

20. The mobile communication terminal according to claim 18 or 19, wherein when the selection module obtains the information of at least one designated device to be controlled from the network control server, The network control server obtains information about at least one designated device to be controlled, where the information about the designated device includes at least one of the following: the QR code of the designated device, the identification of the designated device, the designated device the IP address; the mobile communication terminal identifies and displays the designated device through the information of the designated device; or, the network control server identifies the designated device based on the information of the designated device to be controlled; The mobile communication terminal obtains the information of the designated device identified by the network control server from the network control server.

21. The mobile communication terminal according to claim 15, wherein the network control server is built using Node.js.

22. The mobile communication terminal according to claim 14, wherein the control module is configured to send the control operation instruction to the designated device that is successfully paired with the mobile communication terminal through the HTML page, causing the specified device to perform the control operation indicated by the control operation instruction.

23. The mobile communication terminal according to claim 22, wherein the mobile communication terminal further includes:

The pairing module is configured to send a pairing request to the designated device before the receiving module receives the control operation instruction of the user to control the designated device through the set HTML page, requesting to establish a pairing connection with the designated device, wherein , the pairing request includes the information of the mobile communication terminal; receiving a pairing response returned by the designated device that carries the information of the designated device, and establishing a pairing connection with the designated device according to the pairing response. .

24. The mobile communication terminal according to claim 23, wherein the information of the mobile communication terminal includes at least one of the following: a QR code of the mobile communication terminal, an identification of the mobile communication terminal, the mobile communication terminal The IP address of the terminal; the information of the designated device includes at least one of the following: the QR code of the designated device, the identification of the designated device, and the IP address of the designated device.

25. The mobile communication terminal according to claim 14, wherein the mobile communication terminal further includes:

a building module configured to build the set HTML page before the receiving module receives the control operation instruction of the user to control the specified device through the set HTML page, The HTML page includes at least one of the following: a touch panel page, a shortcut operation page and a terminal selection page, wherein the touch panel page is used to receive information about the user's touch operations on the touch panel page, according to The control operation instructions are generated from the touch operation information to control the cursor operation of the specified device; the shortcut operation page includes at least one of the following controls: volume adjustment control, device desktop display control, device screenshot control and device shutdown control, The shortcut operation page is used to receive information about the user's operation of the controls in the shortcut operation page, and generate the control operation instructions according to the information about the operation of the controls to control the designated device to perform the control. The control operation indicated by the operation instruction; the selection terminal page is used to display the information of the device to be controlled and the controlled status of the device to be controlled, so that the user can select the device to be controlled, the selection The terminal page also includes an authorization password input control for the user to input the authorization password for control operation authentication.

26. The mobile communication terminal according to claim 14 or 15, wherein the receiving module is configured to monitor the user's control operation on the designated device through the set HTML page. , obtain the information of the control operation through the DOM event of the HTML page, and generate the control operation instruction according to the information of the control operation.

27. A computer-readable recording medium having recorded thereon a program for executing the method of claim 1.