TW202340925A - Remote operating method of human-machine interface, and remote operating system for using the same - Google Patents

Abstract

A remote operating method of human-machine interface is disclosed and includes following steps: obtaining a component information of at least one control component of a human-machine interface when an inquiring command is received; transmitting the component information to a remote electronic device by the human-machine interface; generating and providing an operation related information in accordance with the component information by a translating software of the remote electronic device; receiving an operating activity aimed at the control component; translating the operating activity into a component operating instruction by the translating software and transmitting the component operating instruction to the human-machine interface; and, triggering the control component correspondingly by the human-machine interface based on the component operating instruction being received.

Description

Remote operation method and remote operating system of human-machine interface

The present invention relates to a human-machine interface, and in particular to a remote operation method and a remote operating system of the human-machine interface.

Many industrial equipment today are equipped with a Human Machine Interface (HMI), allowing users to operate industrial equipment through the HMI.

Generally speaking, the human-machine interface will provide a touch screen or be equipped with physical function buttons. When there is an operation requirement, the user can refer to the instructions displayed on the human-machine interface to touch or press the corresponding control to cause the industrial equipment to perform the corresponding function.

With the development of the Internet, some users have the need for remote operations. When the user wants to operate the human-machine interface from a remote location, he or she can mainly access the screen of the human-machine interface through a web page and operate the human-machine interface on the web page. Alternatively, the user can copy and share the screen of the human-machine interface to a remote device through Virtual Network Computing (VNC) software, so that the user can operate the human-machine interface through the remote device.

However, the above-mentioned method of converting the screen of the human-machine interface into a responsive web page requires special web design, which requires additional web design and interface verification costs; in addition, the screen of the human-machine interface is copied and shared through VNC. The method of moving to a remote device is not suitable for human-machine interface use with fewer hardware resources and weaker processing capabilities.

The present invention provides a remote operation method and a remote operating system for a human-machine interface, which instructs the user's operating behavior on the human-machine interface, so that the user can remotely operate the human-machine interface through other electronic devices.

In an embodiment of the present invention, the remote operation method of the human-machine interface is applied to a human-machine interface connected to a remote electronic device, the remote electronic device runs a translation software, and the remote operation method mainly includes the following Steps:

a) Receive a query command;

b) Obtain a control information of at least one control in the human-machine interface based on the query command, where the control information includes a control type and a control description of the control;

c) Send the control information to the remote electronic device;

d) The translation software performs a translation process on the control information to generate corresponding operation-related information, and provides the operation-related information through the remote electronic device;

e) Accept an operation performed on the control;

f) The translation software translates the operation behavior into a corresponding control operation instruction and transmits it to the human-machine interface; and

g) The human-machine interface triggers the control according to the control operation instructions.

In the embodiment of the present invention, the remote operating system of the human-machine interface includes:

a translation software running in a remote electronic device;

A human-machine interface is connected to the remote electronic device through a transmission unit, and has an integration unit configured to obtain a control information of at least one control in the human-machine interface when receiving a query command, and use the A communication protocol corresponding to the transmission unit transmits the control information to the remote electronic device, where the control information includes a control type and a control description of the control;

Wherein, the translation software is configured to perform a translation process on the control information to generate and provide corresponding operation-related information, and when accepting an operation behavior performed on the control, translate the operation behavior into a corresponding operation behavior. Control operation instructions are sent to the human-machine interface;

Wherein, the integration unit is configured to trigger the control when receiving the control operation instruction.

Compared with related technologies, the present invention only captures and transmits the control information of each control in the human-machine interface to the remote electronic device. It does not require substantial changes to the existing human-machine interface and does not require the design of remote connection devices. Web pages do not require the use of VNC software, so remote operations can be achieved with the lowest cost and minimum hardware requirements.

A preferred embodiment of the present invention is described in detail below with reference to the drawings.

First, please refer to FIG. 1 , which is a block diagram of a remote operating system according to a first specific embodiment of the present invention. The present invention mainly discloses a remote operating system of a human-machine interface (hereinafter referred to as an operating system in the specification), which can combine the descriptions, attributes and corresponding controls of controls (ie, control elements) in the human-machine interface 1 The content and other information are integrated and sent to the remote electronic device 2 . The remote electronic device 2 generates and provides an optimized command interface based on this information, so that the user can issue commands to specific controls on the human-machine interface 1 through the remote electronic device 2, thereby simulating the user's direct control of the human-machine interface. The controls on interface 1 perform touch operation behavior.

As shown in Figure 1, the operating system of the present invention mainly includes a human-machine interface 1, and a translation software 21 developed relative to the human-machine interface 1. The translation software 21 runs in the remote electronic device 2, and the human-machine interface 1 is connected to the remote electronic device 2 through the transmission unit 12 thereon.

In one embodiment, the remote electronic device 2 can be various electronic devices with different hardware capabilities, such as a remote control without a screen, a personal computer with a normal screen and a normal input interface (such as a keyboard/mouse), Notebooks or smart mobile devices with touch screens, tablets, etc., but are not limited to this.

The remote electronic device 2 has a processor (not shown in the figure). After the processor executes the translation software 21, the remote electronic device 2 can establish a connection with the human-machine interface 1, and generate a corresponding command interface based on the information of the human-machine interface 1 and provide it to the user. Thereby, the user can issue instructions to the human-machine interface 1 through the remote electronic device 2 to simulate the user's direct touch operation behavior on the human-machine interface 1 .

The transmission unit 12 may be a physical serial port, serial port, wireless transmission unit or network line interface, etc., without limitation. In one embodiment, the transmission unit 12 uses RS-232, RS-485, Bluetooth, Ethernet, Wi-Fi or Controller Area Network Bus (CAN bus). ) and other communication protocols, but are not limited to this.

As shown in FIG. 1 , the human-machine interface 1 has an integration unit 11 . Specifically, the firmware of the human-machine interface 1 of the present invention is modified to generate the integration unit 11 internally. In other words, the integration unit 11 of the present invention is a firmware module in the human-machine interface 1 .

The human-machine interface 1 establishes a connection with the remote electronic device 2 through the transmission unit 12 . When the human-machine interface 1 receives the query command sent by the remote electronic device 2 through the transmission unit 12, it will obtain at least one control in the human-machine interface (for example, as shown in FIG. 2) through firmware (ie, through the integration unit 11). Control information of control 4). Furthermore, the human-machine interface 1 transmits the control information to the remote electronic device 2 through the communication protocol used by the transmission unit 12 to be processed by the translation software 21 in the remote electronic device 2 .

In one embodiment, the integration unit 11 may build an instruction set, and the instruction set includes at least a set of remote query commands. After the remote electronic device 2 establishes a connection with the human-machine interface 1, it can send a query command consistent with the instruction set to the human-machine interface 1. If the integration unit 11 determines that the query command is consistent with the instruction set, it can accept the query request from the remote electronic device 2 .

Generally speaking, the screen of the human-machine interface 1 includes components related to operations (ie, controls 4) and components unrelated to operations (such as logos, patterns, colors, contact information, explanatory text, etc.). The technical difference between the present invention and related technologies is that the operating system of the present invention does not copy and transmit the entire screen of the human-machine interface 1 to the remote electronic device 2 for display, but extracts the operation-related information through the integration unit 11 information of the component (ie, the control 4 ), and only transmits these operation-related information to the remote electronic device 2 . In other words, the data that will be sent to the remote electronic device 2 through the hand-shaking process is only the data related to the components on the human-machine interface 1 that can be operated and can generate operating behaviors.

After receiving the above information, the remote electronic device 2 will process and optimize it through the translation software 21 to generate a corresponding command interface and provide it to the user for reference. After referring to the command interface, the user can perform the corresponding operation behavior. Through the above technical solutions of the present invention, it is possible to overcome the problem of additional design costs when providing operation interfaces through web pages, and to overcome the problem that low-cost devices with weak hardware capabilities are not suitable for VNC software for screen sharing.

Please also refer to FIG. 2 , which is a schematic diagram of a remote operating system according to a first specific embodiment of the present invention. The human-machine interface 1 is equipped with many controls 4 based on the original design project. In one embodiment, the integration unit 11 transmits the control information of one or more controls 4 displayed on the screen 10 of the human-machine interface 1 to the remote end. Electronic device 2. After the remote electronic device 2 receives the control information, the translation software 21 performs a translation process on the control information to generate corresponding operation-related information 5 and provide it to the user. In the embodiment of FIG. 2 , the translation software 21 displays the generated operation-related information 5 on the screen 20 of the remote electronic device 2 , but it is not limited to this.

In the present invention, the translation software 21 can determine the presentation mode of the control information on the remote electronic device 2 according to the hardware capabilities of the remote electronic device 2, and then execute the translation program according to this presentation mode. For example, if the remote electronic device 2 is a remote control without a screen, the translation software 21 can translate the control information into operation-related information 5 in voice form, and play the operation-related information 5 on the remote controller through voice. If the remote electronic device 2 is an electronic device with a general screen or a touch screen, the translation software 21 can translate the control information into operation-related information 5 in the form of text, graphics, voice or a combination thereof, and use the screen or voice to Provide the operation-related information 5 on the electronic device.

However, the above are only some specific implementation examples of the present invention, but the form of the operation related information 5 is not limited to the above.

Specifically, when using the operating system of the present invention, the user can still view the screen 10 of the human-machine interface 1. Therefore, the operation-related information 5 provided by the remote electronic device 2 can only enable the user to understand the operation on the human-machine interface 1. The content of each control 4 and the operable behaviors it can perform are sufficient, and there is no need to limit the presentation method of the operation-related information 5 .

It is worth mentioning that the control information mainly includes the control type of control 4 (ie, the properties of control 4) and the control description (including the name, function prompts or menu content of control 4, etc.). In the present invention, the translation software 21 mainly executes the translation program based on the control type and control description of a control 4, and generates the operation-related information 5 corresponding to the control 4. In the present invention, the translation software 21 uses the received operation related information 5 of all controls 4 to form a command interface, and then provides this command interface to the user for operation through the remote electronic device 2 .

After the operation-related information 5 is generated, the translation software 21 continuously monitors the remote electronic device 2 and determines whether the remote electronic device 2 receives the operation behavior for any control 4 . When the remote electronic device 2 accepts an operation behavior for a control 4 , the translation software 21 translates the operation behavior into a corresponding control operation instruction and sends it back to the human-machine interface 1 .

In the embodiment of FIG. 2 , the remote electronic device 2 is a tablet computer with a touch screen as an example, and a plurality of operation-related information 5 is displayed on the touch screen in the form of a combination of text and graphics (i.e., boxes). on the control screen to describe the corresponding control 4 on the human-computer interface 1. However, what is shown in FIG. 2 is only one embodiment of the present invention, and the display method of the operation-related information 5 in the present invention is not limited to what is shown in FIG. 2 .

In the embodiment of FIG. 2 , the operation behavior performed on a specific control 4 refers to the behavior of directly touching the operation-related information 5 used to describe this specific control 4 on the touch screen. In other embodiments, the operation behavior may also be the behavior of selecting the operation related information 5 used to describe the specific control 4 through the keyboard or mouse, or the behavior of selecting the specific control 4 through voice instructions, etc., but not This is the limit.

Specifically, the operating system of the present invention converts and expands the user's operating behavior on the human-machine interface 1 through the remote electronic device 2 and the translation software 21 . As mentioned above, based on the hardware capabilities of the remote electronic device 2, the user can perform operations by direct touch control, using a keyboard, mouse, etc., after referring to the operation related information 5 provided by the remote electronic device 2. The operation is performed through an input interface or through voice, so that the remote electronic device 2 generates corresponding control operation instructions.

After the human-machine interface 1 receives a control operation instruction from the remote electronic device 2, the integration unit 11 can trigger the corresponding control 4 on the human-machine interface 1 based on the received control operation instruction. In this way, the operation behavior of the user directly triggering the control 4 on the human-machine interface 1 can be simulated.

As shown in Figure 1, the translation software 21 of the present invention may mainly include a remote command processing module 211 and an interface operation translation module 212. Specifically, the translation software 21 records computer executable program code. When the processor of the remote electronic device 2 executes the computer executable program code, the translation software 21 can be virtually divided into remote command processing based on the executed functions. Module 211 and interface operation translation module 212. In other words, the remote command processing module 211 and the interface operation translation module 212 are software modules.

In one embodiment, the remote command processing module 211 is used to provide protocol communication capabilities between the human-machine interface 1 and the remote electronic device 2 after the remote electronic device 2 establishes a connection with the human-machine interface 1 . Thereby, the remote electronic device 2 can transmit control information and control operation instructions to the human-machine interface 1 through the remote command processing module 211 .

In one embodiment, the interface operation translation module 212 is used to monitor the status of the remote electronic device 2 to translate the control information provided by the human-machine interface 1 into operation-related information 5 that can be presented by the remote electronic device 2 . Furthermore, the interface operation translation module 212 monitors the user's operation behavior on the remote electronic device 2 and translates the operation behavior into control operation instructions that the human-machine interface 1 can understand.

It is worth mentioning that some types of human-machine interface 1 will be configured as multiple pages (such as the first page, the second page, the third page, etc.) based on operational requirements, and each page has one or more corresponding pages. Controls with different functions 4. Moreover, corresponding physical switching buttons 3 can also be configured on the human-machine interface 1 according to the number of pages. In the embodiment of FIG. 2 , when the switch button 3 corresponding to the first page is pressed, the human-machine interface 1 switches to display one or more controls 4 on the first page; when the switch button 3 corresponding to the second page is pressed. When pressed, the human-machine interface 1 switches to display one or more controls 4 on the second page, and so on.

In one embodiment, when the human-machine interface 1 receives a query command from the remote electronic device 2, the integration unit 11 only retrieves the control information of one or more controls 4 in the current page of the human-machine interface 1, and These control information are sent to the remote electronic device 2 . In this way, consistency in information provision between the human-machine interface 1 and the remote electronic device 2 can be achieved.

Please continue to refer to FIGS. 1 to 3 , wherein FIG. 3 is a flow chart of the remote operation method according to the first specific embodiment of the present invention. The present invention further provides a remote operation method of the human-machine interface (hereinafter referred to as the operation method in the description). The operation method is mainly applied to the operating system as shown in Figures 1 and 2.

As shown in Figure 3, first, the machine interface 1 is connected to the remote electronic device 2 used by the user. After the user operates the remote electronic device 2 (for example, triggers the remote electronic device 2 to execute the translation software 21), the human-machine interface 1 can receive the query command sent by the remote electronic device 2 through the transmission unit 12 (step S10).

When the human-machine interface 1 receives the query command, it can be known that the remote electronic device 2 wants to provide a corresponding command interface for the human-machine interface 1 for the user to operate. At this time, the integration unit 11 of the human-machine interface 1 obtains control information of at least one control 4 in the human-machine interface 1 based on the query command (step S12). In one embodiment, the control information at least includes the control type and control description of control 4.

As mentioned above, the human-machine interface 1 may be configured to have a plurality of pages, wherein each page has one or more controls 4 respectively. In step S12 , the integration unit 11 mainly obtains the control information of all controls 4 in the page currently displayed on the screen 10 of the human-machine interface 1 when receiving the query command, but is not limited to this.

Specifically, in addition to the aforementioned control type and control description, the control information may also selectively include information existing in the human-machine interface 1 such as page identification code, page name, page description, control identification code, and control supplementary information. . By carrying this information into the control information and providing it to the remote electronic device 2, the command interface generated after translation by the translation software 21 can be enriched. This ensures that the user can perform correct operations through the command interface (ie, the control operation information 5) provided by the remote electronic device 2.

After step S12, the integration unit 11 transmits the obtained control information to the remote electronic device 2 (step S14). Specifically, the integration unit 11 mainly processes the control information based on the communication protocol supported by the transmission unit 12 of the human-machine interface 1 , and then transmits the processed control information to the remote electronic device 2 through the transmission unit 12 .

In one embodiment, the transmission unit 12 can be a serial port, a serial port, a wireless transmission unit or a network line interface, etc., and the communication protocol can be RS-232, RS-485, Bluetooth, Ethernet, Wi-Fi or CAN. bus, etc., but not limited to this.

After step S14, the remote electronic device 2 receives control information from the human-machine interface 1 (step S16). Next, the remote electronic device 2 performs a translation process on the control information through the translation software 21 to generate corresponding operation-related information 5, and further provides the operation-related information 5 (step S18).

Specifically, after using the remote electronic device 2 to connect to the human-machine interface 1, the user can continue to confirm the updates of the remote electronic device 2, that is, confirm whether the remote electronic device 2 is based on the content of the human-machine interface 1 And provide operation related information 5. Thereby, the user can use the remote electronic device 2 to further operate the human-machine interface 1 .

In one embodiment, the remote electronic device 2 receives control information from the human-machine interface 1 through the remote command processing module 211 in the translation software 21, and operates the translation module 212 through the interface in the translation software 21 to process the control. Information translation process, but not limited to this.

In one embodiment, the translation software 21 executes the translation process based on a fixed strategy. That is to say, regardless of the hardware capabilities of the remote electronic device 2, the translation software 21 uses a fixed method to translate the control information to generate a fixed form of operation-related information 5, and provides the generated operation in a fixed presentation method. Related information 5. For example, regardless of whether the remote electronic device 2 has a touch screen, the translation software 21 translates the control information into operation-related information 5 in voice form, and plays the generated operation-related information 5 in voice mode. Furthermore, the translation software 21 accepts the user's voice operation.

In another embodiment, in step S18, the translation software 21 determines the presentation mode of the control information on the remote electronic device 2 based on the hardware capabilities of the remote electronic device 2 itself, and then displays the control information based on the determined presentation mode. The control information executes the corresponding translation program.

For example, if the remote electronic device 2 does not have a screen, the translation software 21 may decide to play the control information in voice mode, and therefore translate the control information into operation-related information 5 in voice form. For another example, if the remote electronic device 2 has a touch screen, the translation software 21 may decide to display the control information in the form of touchable graphics, and therefore translate the control information into operation-related information 5 in the form of a combination of text and graphics. However, the above are only some specific implementation examples of the present invention, but are not limited thereto.

As mentioned above, based on different hardware capabilities, the translation software 21 can present the operation-related information 5 in text, graphics, voice or a combination thereof after translation, but is not limited to this.

After the remote electronic device 2 provides the operation-related information 5, the translation software 21 continuously monitors the status of the remote electronic device 2 and determines whether the remote electronic device 2 has received an external operation behavior for any control 4 ( Step S20). In the present invention, the operation behavior may be, for example, direct touch control, selection through an input interface, or selection through voice commands, but is not limited to this.

If it is determined in step S20 that the remote electronic device 2 has indeed received an external operation behavior for any control 4, the translation software 21 will translate the operation behavior into a control operation instruction corresponding to this control 4 and return it. Go to human-machine interface 1 (step S22).

Specifically, after the human-machine interface 1 transmits the control information to the remote electronic device 2 in step S14, it can wait and continue to determine whether to receive the control operation command sent by the remote electronic device 2.

Please refer to FIG. 4A , FIG. 4B and FIG. 4C at the same time. FIG. 4A to FIG. 4C are respectively the first to third specific embodiments of the control schematic diagram of the present invention.

In the present invention, the control type of the control 4 of the human-machine interface 1 can be a button control (button control 41 as shown in Figure 4A), a menu control (a menu control 42 as shown in Figure 4B) or an input control (as shown in Figure 4C The input control 43) is shown, and the control description is the name, function prompt or menu content of this control 4 in the human-machine interface 1.

The embodiment of FIG. 4A discloses two button controls 41, and the control descriptions of the two button controls 41 are "OK" and "Cancel" respectively. In this embodiment, when the translation software 21 performs the translation process on the control information, operation-related information 5 that can represent the selection function of confirmation and cancellation will be generated. After referring to the operation-related information 5, the user can learn that the current screen of the human-machine interface 1 includes the two key controls 41 of "OK" and "Cancel".

On the human-machine interface 1, the user's operations on the button control 41 include pressing (Press), releasing (Release) and clicking (Click). Therefore, when the remote electronic device 2 receives an operation behavior performed on the button control 41, the translation software 21 translates the operation behavior into a press operation command, a release operation command or a click operation command corresponding to the button control 41.

The embodiment of FIG. 4B discloses a menu control 42, and the controls of the menu control 42 are described as "neutron type". In this embodiment, when the translation software 21 performs a translation process on the control information, operation-related information 5 that can represent the menu function of the neutron type will be generated. After referring to the operation-related information 5, the user can learn that the current screen of the human-machine interface 1 includes a "neutron type" menu control 42.

On the human-machine interface 1, the user's operation on the menu control 42 includes selection. Therefore, when the remote electronic device 2 receives an operation behavior performed on the menu control 42 , the translation software 21 translates the operation behavior into a selection operation instruction corresponding to the menu control 42 .

In one embodiment, the control information received by the remote electronic device 2 also includes content information of the menu control 42 (for example, menu content such as "Option 1" and "Option 2" in Figure 4B). When the user operates on a menu control 42, the translation software 21 can directly provide the menu content of the menu control 42 (for example, displayed on the screen 20, or played through voice) for the user to perform further selection operations.

In another embodiment, when the user performs an operation on a menu control 42, the translation software 21 converts this operation behavior into a corresponding control operation instruction and sends it back to the human-machine interface 1, and uses this control operation instruction Request the menu content of this menu control 42 from the human-machine interface 1 (detailed later). In this embodiment, the operating system can effectively limit the data size of the control information to ensure smooth communication between the human-machine interface 1 and the remote electronic device 2 .

The embodiment of FIG. 4C discloses an input control 43, and the control of the input control 43 is described as "setting value". In this embodiment, when the translation software 21 performs a translation process on the control information, operation-related information 5 that can represent the input function of the set value will be generated. After referring to the operation-related information 5, the user can know that the current screen of the human-machine interface 1 includes the input control 43 of "setting value".

On the human-machine interface 1, the user's operation on the input control 43 includes input. Therefore, when the remote electronic device 2 receives an operation behavior for the input control 43, the translation software 21 can further provide a corresponding input interface 431. When the user inputs corresponding values through the input interface 431 (for example, through keyboard input or voice input), the translation software 21 translates the user's input behavior and input content into input operation instructions corresponding to the input control 43 .

However, the above are only some specific implementation examples of the present invention. The operating system and operating method of the present invention can be applied to various control types supported by the existing human-machine interface 1, and are not limited to the key control 41, menu control 42 and input control 43 shown in Figures 4A to 4C.

Return to Figure 3. After step S22, the human-machine interface 1 can receive the control operation command returned by the remote electronic device 2 through the transmission unit 12. At this time, the integration unit 11 triggers the corresponding control 4 in the human-machine interface 1 according to the control operation instruction (step S24). Thereby, the user's operation behavior on the remote electronic device 2 can be simulated as the user's direct touch operation behavior on the control 4 on the human-machine interface 1 .

Specifically, after the user performs the operation behavior on the remote electronic device 2 in step S20, the user can continue to confirm whether the screen of the human-machine interface 1 is updated. Thereby, the user can determine whether the human-machine interface 1 triggers the control 4 corresponding to the user's operation behavior.

In one embodiment, after completing the operation behavior, the user can confirm the update of the screen of the human-machine interface 1 after the operation behavior is performed, thereby determining whether to perform the next operation behavior through the remote electronic device 2 . In another embodiment, after completing the operation behavior, the user can continue to monitor the human-machine interface 1 to refresh the screen regularly, thereby obtaining the relevant information continuously provided by the human-machine interface 1 after triggering the corresponding control 4 .

In the first embodiment, the translation software 21 translates the first button control on the human-machine interface 1 and generates the first operation-related information, and after referring to the first operation-related information, the user performs operations on the first button control. Click action. In this embodiment, the integration unit 11 will click the first button control according to the received control operation instruction in step S24.

In the second embodiment, the translation software 21 translates the first menu control on the human-machine interface 1 and generates second operation-related information, and after referring to the second operation-related information, the user performs operations on the first menu control. Select an action. In this embodiment, the integration unit 11 will select the first menu control according to the received control operation instruction in step S24.

In the third embodiment, the translation software 21 translates the first input control on the human-machine interface 1 and generates third operation-related information, and after referring to the third operation-related information, the user performs operations on the first input control. Enter the action. In this embodiment, the integration unit 11 will add corresponding data to the first input control according to the received control operation instruction in step S24.

However, the above are only some specific implementation examples of the present invention, but are not limited thereto.

In the present invention, the translation software 21 will continuously determine whether the user's operation is completed. For example, if the user clicks on a menu control 42, the translation software 21 returns to step S12 to obtain the menu content of the menu control 42 from the human-machine interface 1, and waits for further selection operations by the user. For another example, if the user clicks on an input control 43, the translation software 21 returns to step S12 to obtain the input content of the input control 43 from the human-machine interface 1, and waits for further input operations by the user.

If it is determined that the user's operation has been completed, the translation software 21 can end the operation method of the present invention. For example, the translation software 21 can control the remote electronic device 2 to end the connection with the human-machine interface 1 .

Please refer to FIGS. 1 to 5 simultaneously. FIG. 5 is a schematic diagram of remote operation according to the first specific embodiment of the present invention. FIG. 5 is used to illustrate the information flow when the user operates the key control 41 on the human-machine interface 1 through the remote electronic device 2 .

In the embodiment of FIG. 5 , the screen 10 of the human-machine interface 1 has two button controls 41 . When the human-machine interface 1 receives the query command from the remote electronic device 2 , the integration unit 11 obtains the control information of the key control 41 and transmits it to the remote electronic device 2 . The remote electronic device 2 performs a translation process on the control information through the translation software 21 to generate corresponding operation-related information 5 and display it on the screen 20 .

After viewing the operation-related information 5 displayed on the remote electronic device 2 , the user can perform operations on the specific key control 41 based on the operation-related information 5 . For example, in the embodiment of FIG. 5 , the user can issue the operation input 6 through the voice command of "press YES". After receiving the user's operation input 6, the translation software 21 processes the operation input 6 based on the operation related information 5 to translate the operation input 6 into a control operation instruction corresponding to the key control 41 whose control description is "YES". In the embodiment of FIG. 5 , the control operation instruction takes "EVT_KEY_PRESS_Key 1" as an example, where "Key 1" can be, for example, the control number of the key control 41 corresponding to this control operation instruction, but is not limited thereto.

Specifically, the control operation instruction is the original instruction of the key control 41 in the human-machine interface 1 . After the human-machine interface 1 receives the control operation command sent by the remote electronic device 2, the integration unit 11 can directly trigger the corresponding button control 41 according to the control operation command, so that the human-machine interface 1 executes the corresponding function of the button control 41. In this way, the operation behavior of the user directly triggering the key control 41 on the human-machine interface 1 can be simulated.

As mentioned above, the present invention converts the user's touch behavior on the human-machine interface 1 into voice input behavior on the remote electronic device 2, thereby expanding the operational flexibility of the human-machine interface 1.

Please continue to refer to FIGS. 1 to 6 at the same time. FIG. 6 is a schematic diagram of a remote operation according to a second specific embodiment of the present invention. FIG. 6 is used to illustrate the information flow when the user operates the menu control 42 on the human-machine interface 1 through the remote electronic device 2 .

In the embodiment of FIG. 6 , the screen 10 of the human-machine interface 1 has two menu controls 42 . When the human-machine interface 1 receives the query command from the remote electronic device 2 , the integration unit 11 obtains the control information of the menu control 42 and transmits it to the remote electronic device 2 . The remote electronic device 2 performs a translation process on the control information through the translation software 21 to generate corresponding operation-related information 5 and display it on the screen 20 .

After viewing the operation-related information 5 in the remote electronic device 2 , the user can click a specific menu control 42 based on the operation-related information 5 . For example, in the embodiment of FIG. 6 , the user can touch the screen 20 of the remote electronic device 2 to realize the operation input 6 of “clicking the drop-down menu”. After receiving the user's operation input 6, the translation software 21 processes the operation input 6 based on the operation related information 5, so as to translate the operation input 6 into a controller corresponding to the control described as "neutron/twisting teeth [1]" Control operation instructions of the menu control 42. In the embodiment of FIG. 6 , the control operation instruction takes "EVT_KEY_PRESS_Menu 1" as an example, where "Menu 1" can be, for example, the control number of the menu control 42 corresponding to this control operation instruction, but is not limited thereto.

The control operation command is the original command of the menu control 42 in the human-machine interface 1 . When the human-machine interface 1 receives the control operation command sent by the remote electronic device 2, the integration unit 11 can click the corresponding menu control 42 according to the control operation command. Thereby, the operation behavior of the user directly touching the menu control 42 on the human-machine interface 1 can be simulated.

When the menu control 42 is clicked, the human-machine interface 1 opens the menu control 42 according to the instruction and displays the menu content of the menu control 42 . Then, the integration unit 11 uses the menu content as part of the control information of the menu control 42 and transmits the control information including the menu content to the remote electronic device 2 .

After receiving the updated control information, the remote electronic device 2 performs a translation process on the control information again through the translation software 21 to update the originally displayed operation-related information 5 . As shown in Figure 6, the updated operation-related information 5 displays all the menu contents of this menu control 42, including "pre-ejector", "neutron five", "neutron six", etc.

After viewing the operation-related information 5 in the remote electronic device 2 , the user can select specific menu content in the menu control 42 based on the operation-related information 5 . For example, the user can touch the screen 20 of the remote electronic device 2 to realize the operation input 6 of "click on box 6". After receiving the user's operation input 6 , the translation software 21 processes the operation input 6 based on the operation related information 5 to translate the operation input 6 into corresponding control operation instructions and sends them back to the human-machine interface 1 . In the embodiment of FIG. 6 , the control operation instruction is “EVT_KEY_SELECT_neutron six” as an example.

After the human-machine interface 1 receives the control operation command sent by the remote electronic device 2, the integration unit 11 can select the corresponding menu content (ie, neutron six) in the menu control 42 according to the control operation command. Thereby, the operation behavior of the user directly selecting the option "neutron six" in the menu control 42 on the human-machine interface 1 is simulated.

As mentioned above, the operating system and operating method of the present invention first establish the connection between the human-machine interface 1 and the remote electronic device 2, and then transmit the control information and control operation instructions. In order to ensure data security, the operating system and operating method of the present invention may further include a connection protection mechanism before starting to transmit data.

Please refer to FIGS. 1 to 7 , where FIG. 7 is a first specific embodiment of the connection verification flow chart of the present invention.

As shown in FIG. 7 , when the user needs to operate the human-machine interface 1 , the remote electronic device 2 can be used to issue a connection request to the human-machine interface 1 (step S30 ). After receiving the connection request from the remote electronic device 2, the human-machine interface 1 can immediately generate the corresponding passcode and provide it to the remote electronic device 2 (step S32).

In one embodiment, the human-machine interface 1 can generate a two-dimensional barcode recording the passcode in step S32, and display the two-dimensional barcode on the screen 10 of the human-machine interface 1. In another embodiment, after the human-machine interface 1 generates the passcode (for example, numbers, letters, symbols and combinations thereof) in step S32, it can be directly displayed on the screen 10, or sent to the remote end in the form of a message. Electronic device 2.

After step S32, the remote electronic device 2 can input the passcode (step S34). In one embodiment, the remote electronic device 2 can scan the two-dimensional barcode displayed on the screen 10 of the human-machine interface 1 to input the passcode. In another embodiment, the remote electronic device 2 can automatically or manually input the passcode provided by the human-machine interface 1 into the corresponding verification interface, but is not limited thereto.

After the passcode is input, the remote electronic device 2 notifies the human-machine interface 1 of the fact that the passcode input is completed (step S36). At this time, the human-machine interface 1 can authenticate the remote electronic device 2, and establish a connection with the remote electronic device 2 after passing the authentication (step S38).

Through the use of the passcode, it is possible to avoid the risk of malicious persons using unverified equipment to remotely operate the human-machine interface 1, causing damage to the user.

As mentioned above, the present invention can achieve the lowest cost and minimum hardware requirements through the technical solution of transmitting the control information of the human-machine interface to the remote electronic device and allowing the user to issue operating instructions on the remote electronic device. Next, the purpose of transforming and expanding the user's operating behavior for the human-machine interface is achieved.

The above descriptions are only preferred specific examples of the present invention, which do not limit the patent scope of the present invention. Therefore, all equivalent changes made by applying the content of the present invention are equally included in the scope of the present invention, and are hereby stated. bright.

1: Human-computer interface 10:Screen 11: Integrated unit 12:Transmission unit 2: Remote electronic device 20:Screen 21: Translation software 211:Remote command processing module 212:Interface operation translation module 3: Switch button 4:Control 41:Button control 42:Menu control 43:Input controls 431:Input interface 5: Operation related information 6: Operation input S10~S24: Operation steps S30~S38: Verification steps

Figure 1 is a block diagram of a remote operating system according to a first specific embodiment of the present invention.

Figure 2 is a schematic diagram of a remote operating system according to the first specific embodiment of the present invention.

Figure 3 is a first specific embodiment of the flow chart of the remote operation method of the present invention.

FIG. 4A is a schematic diagram of a control according to the first specific embodiment of the present invention.

FIG. 4B is a second specific embodiment of the control schematic diagram of the present invention.

Figure 4C is a third specific embodiment of the control schematic diagram of the present invention.

Figure 5 is a first specific embodiment of the remote operation schematic diagram of the present invention.

Figure 6 is a second specific embodiment of the remote operation schematic diagram of the present invention.

Figure 7 is a first specific embodiment of the connection verification flow chart of the present invention.

S10~S24: Operation steps

Claims (15)

A remote operation method of a human-machine interface, applied to a human-machine interface connected to a remote electronic device, the remote electronic device runs a translation software, and the remote operation method includes: a) Receive a query command; b) Obtain a control information of at least one control in the human-machine interface based on the query command, where the control information includes a control type and a control description of the control; c) Send the control information to the remote electronic device; d) The translation software performs a translation process on the control information to generate corresponding operation-related information, and provides the operation-related information through the remote electronic device; e) Accept an operation performed on the control; f) The translation software translates the operation behavior into a corresponding control operation instruction and transmits it to the human-machine interface; and g) The human-machine interface triggers the control according to the control operation instructions. The remote operation method as described in claim 1, wherein the human-machine interface is configured with a plurality of pages, each of which has at least one control. The step b) is to obtain a parameter when the human-machine interface receives the query command. Information about the control in the current page. The remote operation method as described in request item 1, wherein the control type is a button control, a menu control or an input control, and the control is described as the name, function prompt or menu content of the control in the human-machine interface. The remote operation method as described in request item 3, wherein step f) is to translate the operation behavior into a press operation command (Press), a release operation command (Release) or a click operation corresponding to the button control command (Click), a selection operation command (Select) corresponding to the menu control, or an input operation command (Input) corresponding to the input control. The remote operation method as described in claim 1, wherein the step d) is to determine a presentation method of the control information on the remote electronic device based on a hardware capability of the remote electronic device, and based on the presentation method Perform the translation process on the control information. The remote operation method as described in claim 1, wherein the operation-related information is presented in text, graphics, voice, or a combination thereof. The remote operation method as described in claim 1, wherein step a) further includes the following steps: a01) The remote electronic device requests a connection from the human-machine interface; a02) After step a01), a passcode is generated and provided by the human-machine interface; a03) The remote electronic device inputs the passcode; a04) After step a03), the remote electronic device notifies the human-machine interface; and a05) After the human-machine interface verification is successful, a connection between the human-machine interface and the remote electronic device is established. A human-machine interface remote operating system, including: a translation software running in a remote electronic device; A human-machine interface is connected to the remote electronic device through a transmission unit, and has an integration unit configured to obtain a control information of at least one control in the human-machine interface when receiving a query command, and use the A communication protocol corresponding to the transmission unit transmits the control information to the remote electronic device, where the control information includes a control type and a control description of the control; Wherein, the translation software is configured to perform a translation process on the control information to generate and provide corresponding operation-related information, and when accepting an operation behavior performed on the control, translate the operation behavior into a corresponding operation behavior. Control operation instructions are sent to the human-machine interface; Wherein, the integration unit is configured to trigger the control when receiving the control operation instruction. The remote operating system as described in claim 8, wherein the translation software is executed by the remote electronic device to generate a remote command processing module and an interface operation translation module, and the remote command processing module provides the user with A protocol communication capability between the machine interface and the remote electronic device is used to transmit the control information and the control operation instructions. The interface operation translation module translates the control information into the operation-related information and translates the operation behavior into the control. Operating instructions. The remote operating system as described in claim 9, wherein the control type is a button control, a menu control or an input control, and the control is described as the name, function prompt or menu content of the control in the human-machine interface. The remote operating system as described in claim 10, wherein the interface operation translation module translates the operation behavior into a press operation command (Press), a release operation command (Release) or a click operation corresponding to the button control command (Click), a selection operation command (Select) corresponding to the menu control, or an input operation command (Input) corresponding to the input control. The remote operating system as described in claim 9, wherein the interface operation translation module determines a presentation method of the control information on the remote electronic device based on a hardware capability of the remote electronic device, and based on the presentation Method to perform the translation process on the control information. The remote operating system of claim 12, wherein the operation-related information is presented in text, graphics, voice, or a combination thereof. The remote operating system as described in claim 8, wherein the human-machine interface is configured with a plurality of pages, each of which has at least one control. When receiving the query command, the integration unit obtains a parameter of the human-machine interface. Information about the control in the current page. The remote operating system as described in request item 8, wherein the communication protocol is RS-232, RS-485, Bluetooth, Ethernet, Wi-Fi or Controller Area Network Bus (CAN) bus).

Images