JP2018142926A - Remote operation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote operation system capable of easily and reliably operating an operation object apparatus by a remote operation apparatus even if the operation object apparatus is not provided with a communication function or the display content of a monitor screen of the operation object apparatus is not recognized on a remote operation apparatus side.SOLUTION: A remote operation system 100 includes a screen acquisition unit 121, an image determination unit 122, a state providing unit 123, an operation reception unit 131, an operation signal output unit 132, an operation correctness determination unit 133, and a state database DB. Furthermore, the remote operation system 100 includes: a capture device 141 to capture an image of a monitor X1 of an object apparatus X; and a control interface 142 connected to a keyboard X2 of the control object X in a superimposed manner.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a remote control system for remotely operating a target device from a remote control device based on the state of a monitor screen of the target device.

  Conventionally, as this type of remote operation system, as shown in Patent Document 1 below, the operation target device is configured to be able to communicate with the remote operation device, and the operation screen of the operation target device is displayed on the remote operation device side. Systems that accept remote operations are known.

JP 2013-62778 A

  However, in such a conventional remote operation system, the operation target device does not have a communication function, or the monitor screen of the operation target device is merely a display (for example, an analog screen display) and is not recognized on the remote operation device side. In this case, there is a problem that it is difficult to operate the operation target device with the remote operation device.

  In view of the above circumstances, the present invention provides simple and reliable operation even when the operation target device does not have a communication function or when the display content of the monitor screen of the operation target device is not recognized on the remote operation device side. An object of the present invention is to provide a remote operation system capable of operating a target device with a remote operation device.

The remote operation system of the first invention is a remote operation system for remotely operating the target device from the remote operation device based on the state of the monitor screen of the target device.
A screen acquisition unit that captures and acquires the monitor screen of the target device as an image;
An image determination unit that determines the state of the image from the image acquired from the screen acquisition unit;
A state providing unit that provides the remote control device with the state determined by the image determining unit;
An operation receiving unit that receives an operation from the remote control device;
An operation signal output unit that converts the operation received by the operation receiving unit into an operation signal of the target device and outputs the operation signal to the target device.

  According to the remote operation system of the first invention, since the screen acquisition unit captures the monitor screen of the target device and acquires it as an image, even when the monitor screen of the target device is merely an analog screen display, Information on the monitor screen can be provided to the remote control device.

  Here, when various images are displayed on the monitor screen, the state of the captured image acquired by the screen acquisition unit is determined in advance by the image determination unit, and the state of the image is provided to the remote control device. The screen information suitable for remote operation can be provided to the remote operation device.

  Further, according to the remote operation system of the first invention, the operation from the remote operation device received by the operation reception unit is converted into an operation signal of the target device and output to the target device. Therefore, even if the operation target device does not have a communication function, the target device can be directly operated by the operation signal.

  Thus, according to the remote operation system of the first invention, even when the operation target device does not have a communication function or when the display content of the monitor screen of the operation target device is not recognized on the remote operation device side, it is simple and easy. The operation target device can be surely operated by the remote operation device.

  In the remote operation system of the first invention, the capture of the monitor screen includes not only hardware capture using a capture device but also any concept of capture using software such as remote desktop.

The remote control system of the second invention is the first invention,
The image determination unit determines whether the target device is in a screen state that can be operated as the state of the image from the image acquired from the screen acquisition unit,
The state providing unit generates an operation input screen corresponding to the operable image state and provides it to the remote operation device when the state of the image is an image state operable by the target device. Features.

  According to the remote operation system of the second invention, the image determination unit determines in advance whether the image acquired from the screen acquisition unit is in a screen state that can be operated by the target device, and the image state is operable by the target device. In the case of the image state, the state providing unit generates an operation input screen corresponding to the operable image state and provides it to the remote operation device. Therefore, an operation input screen suitable for remote operation can be provided to the remote operation device.

  Thus, according to the remote operation system of the second invention, even if the operation target device does not have a communication function or the display content of the monitor screen of the operation target device is not recognized on the remote operation device side, the remote operation is performed. The operation target device can be easily and surely operated by the remote operation device through the operation input screen suitable for the operation.

The remote control system of the third invention is the first or second invention,
The operation signal output unit is connected in a superimposed manner to a human interface (for example, a keyboard) of the target device, and converts the operation received by the operation reception unit into an operation signal of the target device as an emulation signal of the human interface. And outputting to the target device.

  According to the remote operation system of the third invention, the operation received by the operation receiving unit is converted into an operation signal of the target device and output to the target device as an emulation signal of the human interface of the target device.

  Therefore, according to the remote operation system of the third invention, even when the operation target device does not have a communication function or an external connection input, the operation target device can be easily and reliably operated by the remote operation device.

A remote control system according to a fourth aspect of the present invention is the remote control system according to any one of the first to third aspects of the invention
After the operation signal is output to the target device by the operation signal output unit, an operation correctness determination unit that determines operation correctness from the state of the image acquired via the screen acquisition unit is provided.

  According to the remote operation system of the fourth aspect of the invention, the operation correctness determination unit outputs the operation signal to the target device by the operation signal output unit, and then determines whether the operation is correct from the state of the image acquired through the screen acquisition unit. Do. For this reason, it is possible to confirm whether the remote operation has been reliably executed by the change in the state of the target device.

  Thus, according to the remote operation system of the fourth invention, even when the operation target device does not have a communication function or when the display content of the monitor screen of the operation target device is not recognized on the remote operation device side, The operation target device can be reliably operated by the remote operation device, and the execution of the operation can be confirmed.

The system block diagram which shows the outline | summary of the remote control system of this embodiment. The flowchart which shows the processing content in the remote control system of FIG. The flowchart which shows the processing content in the remote control system of FIG.

  As illustrated in FIG. 1, the remote operation system 100 according to the present embodiment further includes a case where the target device X that is an operation target device does not have a communication function or does not have an external connection interface. Even if the screen of the monitor X1 is an analog screen display, the remote operation system remotely operates the target device X from the remote control device Y based on the state of the monitor X1 screen of the target device X.

  The remote operation system 100 includes a screen acquisition unit 121, an image determination unit 122, a state provision unit 123, an operation reception unit 131, an operation signal output unit 132, an operation correctness determination unit 133, and a state database DB. .

  Furthermore, the remote operation system 100 includes a capture device 141 that captures an image of the monitor X1 of the target device X, and a control interface 142 that is connected in a superimposed manner to a keyboard X2 that is a human interface of the control target X.

  The screen of the monitor X1 of the target device X is, for example, a screen of a front computer in an accommodation facility that is the target device X. For example, on the screen, the “in-room”, “Vacancy”, “Cleaning”, “Reservation”, etc. are displayed in different colors.

  The screen acquisition unit 121 captures and acquires the screen on the monitor X1 via the capture device 141. The capture device 141 outputs an analog image signal of a monitor cable (VGA cable) connecting the front computer X and the monitor X1 to the remote operation system 100 as image data sampled at a constant period.

  For example, the capture device 141 may have a function as a VGA distribution cable in addition to the VGA video capture board. The capture device 141 is not limited to video capture from a VGA cable, but a video capture board that captures from a video (S-Video or composite) input / output cable, a DVI capture board for digital display, It may be an SDI capture board for video, a security camera, an HDMI (registered trademark) capture board for general video equipment, or the like.

  The image determination unit 122 determines the state of the image from the image acquired from the screen acquisition unit 121. Note that the image determination unit 122 performs filter processing on the image acquired from the screen acquisition unit 121 as necessary, and determines the state of the image. The filter process is, for example, RGBA conversion, and converts the captured image into an image having a specific number of colors. Note that the filtering process may be PNGA conversion with a small number of colors in addition to RGBA conversion.

  Specifically, the image determination unit 122 determines whether the target device X is in a screen state that can be operated as the state of the image acquired from the screen acquisition unit 121.

  The state providing unit 123 provides the remote operation device Y with the screen state of the target device X determined by the image determination unit.

  Specifically, the state providing unit 123 performs an operation corresponding to an operable image state when the screen state of the target device X is an image state in which the target device X can be operated as a result of the determination by the image determination unit 122. An input screen is generated and provided to the remote operation device Y.

  The above is the configurations 121 to 123 for grasping the state of the image of the monitor X1 of the target device X in the remote operation system 100 and providing it to the external remote operation device Y.

  The operation reception unit 131 receives an operation from the remote operation device Y in the remote operation system 100. The remote operation device Y may be a personal computer Y1 or a smartphone Y2 as long as it is an external operation device connected via a network NW such as the Internet.

  The operation signal output unit 132 converts the operation from the remote operation device Y received by the operation reception unit 131 into an operation signal of the target device X and outputs the operation signal to the target device X.

  More specifically, the operation signal output unit 132 emulates the operation of the keyboard X2 of the target device X by the operation received by the operation reception unit 131 via the control interface 142 connected to the keyboard X2 of the target device X in a superimposed manner. The signal is output to the control interface 142 so as to be a signal.

  The operation correctness determination unit 133 determines whether the operation is correct from the state of the image acquired via the screen acquisition unit 121 after the operation signal output unit 132 outputs the operation signal to the target device X.

  The above is the configurations 131 to 133 for operating the target device X by the operation from the remote operation device Y in the remote operation system 100.

  In addition, the remote operation system 100 stores and holds the state of the image determined by the image determination unit 122 and the correctness of the operation determined by the operation correctness determination unit 133 by the state database DB.

  The above is the configuration of the remote operation system 100. The remote operation system 100 includes, for example, hardware such as a central processing unit (CPU), a read only memory (ROM), and a random access memory (RAM). Processing means 121 to 123, 131 to 133, a program for executing processing by DB is stored and held in a memory (not shown), and by executing the program, an arithmetic device (sequencer) for executing the above control processing Function.

  Next, with reference to FIG. 2, a process of grasping the state of the image on the monitor X1 of the target device X by the remote operation system 100 configured as described above and providing it to the external remote operation device Y will be described. .

  First, the remote operation system 100 acquires the screen on the monitor X1 as a captured image (in practice, an image captured at a certain period) via the capture device 141 by the screen acquisition unit 121 (STEP 121 / FIG. 2). ).

  Next, in the remote operation system 100, the image determination unit 122 determines whether the screen of the target device X is operable as the state of the captured image acquired in STEP 121 (STEP 122 / FIG. 2).

  Specifically, various methods can be adopted. For example, the color information of a specific point on the image is acquired, and the monitor X1 of the target device X is obtained from the combination of the colors of the acquired color information (RGB data). The image state of the screen (the image state in which the target device X is operable or in the case of the operable image state is determined) is determined.

  As another determination method, for example, the image state of the screen of the monitor X1 of the target device X is determined by recognizing characters (including character strings) displayed on the image by OCR (optical character recognition). May be.

  Furthermore, in order to improve the determination accuracy, even if the image state of the screen of the monitor X1 of the target device X is determined by a combination of color information on the image and characters recognized by OCR (optical character recognition). Good.

  Thus, for example, when the target device X is a front computer in an accommodation facility, when the screen of the monitor X1 is captured, “in-room”, “vacant”, “cleaning”, It is determined that an image in which “reservation” or the like is displayed in different colors is in an operable state and an image state in which a reservation regarding a vacant room can be accepted.

  Then, the remote operation system 100 stores the determination result by the image determination unit 122 in the state database DB (STEP 123 / FIG. 2).

  Next, the state providing unit 123 extracts an operable function from the image state (determination result read from the state database DB) of the screen of the monitor X1 of the target device X determined by the image determination unit 122 (STEP 124). / Figure 2).

  Specifically, the state providing unit 123 extracts an operable function corresponding to the determination result by the image determination unit 122 by referring to a state table in which the determination result and the operable function are arranged in a matrix.

  Next, the state providing unit 123 generates an operation input screen (STEP 125 / FIG. 2).

  Specifically, the state providing unit 123 generates an operation input screen corresponding to the operable function extracted in STEP 124.

  Here, the operation input screen is a screen that can be displayed by HTML output to a browser, a dedicated application, or the like. For example, when the target device X is a front computer in an accommodation facility, it is displayed in the ROOM number portion of the vacant room. Only the operation input screen displaying the reservation button is generated.

  Here, the operation input screen is not simply generated as a screen listing the operable functions, but the operation input screen is generated so as to provide only necessary functions according to the additional conditions.

  As an additional condition, for example, when the number of vacancies is 3 or less, the reservation button that is a reservation reception function is not displayed so that the reservation reception is stopped, or a specific time (2:00 am (˜5 o'clock) is a condition that the reservation button as a reservation reception function is not displayed so as to stop the reservation reception.

  Next, the state providing unit 123 provides the operation input screen generated in STEP 125 to the remote operation device Y via the network NW (STEP 126 / FIG. 2).

  The above is the content of the process of grasping the state of the image of the monitor X1 of the target device X by the remote operation system 100 and providing it to the external remote control device Y. With this process, the screen of the monitor X1 of the target device X Even when the display is only an analog screen display, the information on the screen of the monitor X1 can be provided to the remote control device.

  Furthermore, when various images are displayed on the screen of the monitor X1, the state of the captured image acquired by the screen acquisition unit 121 is determined in advance by the image determination unit 122, and an operation input screen corresponding to the state of the image is displayed. Is generated by the state providing unit 123 and provided to the remote operation device Y, information on a screen suitable for remote operation can be provided to the remote operation device Y.

  The state providing unit 123 generates the operation input screen at a fixed timing, in addition to the case where the operation input screen is generated. In addition, the change in the image state is monitored at a fixed processing cycle. The operation input screen may be generated at the timing.

  The timing at which the state providing unit 123 generates the operation input screen is the timing at which the remote operation device Y accesses the remote operation system 100 or the timing at which the remote operation device Y requests the operation of the operation target device X. It may be. In this case, it is preferable that the state providing unit 123 operates the generated operation input screen only for the remote operation device Y that has accessed the remote operation system 100 or the remote operation device Y that has received an operation request.

  Next, a process for operating the target device X by an operation from the remote operation device Y in the remote operation system 100 will be described with reference to FIG.

  As a premise, as described above, when the state of the image of the monitor X1 of the target device X is grasped and an operation input screen corresponding to the content is provided to the external remote operation device Y, the user of the remote operation device Y Determines the operation content of the operation target device X while viewing the operation input screen.

  At this time, the remote operation system 100 monitors the operation reception unit 131 for an operation request from the remote operation device Y at a constant processing cycle (STEP 131 / FIG. 3).

  When there is an operation request from the remote operation device Y (YES in STEP 131 / FIG. 3), the operation content is specified (STEP 132 / FIG. 3). On the other hand, when there is no operation request from the remote operation device Y (NO in STEP 131 / FIG. 3), this monitoring is repeatedly executed until an operation request is received.

  The operation content is specified by specifying the operation content on the operation input screen. More specifically, the operation content is specified by the URL parameter transmitted from the remote operation device Y to the remote operation system 100. Is done.

  For example, when the target device X is a front computer in an accommodation facility and the reservation operation for the vacant room 101 is performed from the remote control device Y, [http: // server address / yoyaku (operation command)? Room (target vacant room) ) = 101] The operation of the reservation button in the 101st room is specified by a URL parameter such as

  Next, the operation signal output unit 132 converts the operation content specified in STEP 132 into an operation signal of the target device X (STEP 133 / FIG. 3).

  Specifically, the operation signal is converted by the operation received by the operation receiving unit 131 via the control interface 142 connected to the keyboard X2 of the target device X in an overlapping manner with the emulation signal of the keyboard X2 of the target device X. As described above, this is executed as a conversion to a control signal (execution command) of the control interface 142 according to the operation content.

  Here, in the control signal (execution command) to the control interface 142, a control signal (execution command) that has a delay (delay time) that would occur when a person operates the keyboard X2 is appropriately inserted.

  Next, the operation signal output unit transmits the operation signal of the target device X converted in STEP 133 to the control interface (STEP 134 / FIG. 3).

  As a result, the control interface 142 generates a control signal (operation code) that allows the target device X to recognize the control signal (execution command) and outputs the control signal (operation code) to the target device X. Also in this control signal (operation code), a control signal (execution command) for giving a delay (delay time) that would occur when a person operates the keyboard X2 is appropriately inserted.

  For example, when the target device X is a front computer in an accommodation facility, when the reservation operation for the vacant room 101 is performed from the remote operation device Y, the control signal corresponding to the key operation of the room number 101 of the keyboard X2 and the reservation key operation ( An operation code) is generated with a control signal (execution command) having a delay (delay time) interposed therebetween, and is output to the target device X.

  In the present embodiment, the case where a control signal (operation code) that can be recognized by the target device X is generated via the control interface has been described. However, such control is performed by the operation signal output unit 132 of the remote operation system 100. The signal (operation code) may be directly converted and output.

  The above is the processing content for operating the target device X by the operation from the remote operation device Y in the remote operation system 100. In addition to such processing, the operation correctness determination unit 133 may perform the operation correctness determination processing. Preferred (STEP 135-136 / FIG. 3).

  Specifically, the operation correct / incorrect determination unit 133 obtains an operation screen after the operation signal output unit 132 outputs an operation signal to the target device X (actually, after waiting for the counter timer or the like to expire for a certain period of time). It is determined from the state of the image acquired via the unit 121 whether the operation has been successful (STEP 135 / FIG. 3).

  Here, the analysis of the state of the image acquired through the operation screen acquisition unit 121 can be operated by the target device X corresponding to the operation content by the same processing as the image determination unit 122 (see STEP 122 / FIG. 2). If there is a significant change in the screen state, it is determined that the operation is successful (YES in STEP 135 / FIG. 3), and the series of processes is terminated.

  On the other hand, when there is no change in the screen state that can be operated by the target device X corresponding to the operation content, it is determined that the operation is not successful (NO in STEP 135 / FIG. 3), and error notification is performed on the remote operation device Y or the like. (STEP 136 / FIG. 3), a series of processing is terminated.

  Instead of the error notification, the operation signal output unit 132 may transmit the operation signal again (see STEP 134) (for example, return to STEP 134).

  The determination result (image state) by the operation correctness determination unit 133 is preferably stored (overwritten) in the state database DB.

  The above is the processing for operating the target device X by the operation from the remote operation device Y in the remote operation system 100 and the processing for determining the correctness of the operation.

  In this process, the operation from the remote operation device Y received by the operation receiving unit 131 is converted into an operation signal of the target device X and output to the target device X. Therefore, the operation target device X does not have a communication function. Even in this case, the target device X can be directly operated by the operation signal.

  Furthermore, since the operation correct / incorrect determination unit 133 determines the operation correct / incorrect according to the change in the state of the target device X, it is possible to confirm whether the remote operation has been executed reliably.

  Thus, according to the remote operation system of this embodiment, when the operation target device X does not have a communication function, the display content of the screen of the monitor X1 of the operation target device X is not recognized on the remote operation device Y side. Even in this case, it is possible to easily and reliably operate the operation target device X with the remote operation device Y and confirm the execution of the operation.

  In the present embodiment, the case of hard capture using the capture device 141 has been described as the capture of the monitor screen. However, the capture is not limited to this, and is capture by software such as remote desktop. May be.

  Further, in the present embodiment, the case where the keyboard X2 is emulated as the human interface of the control target X has been described. However, the present invention is not limited to this, and human input such as a mouse / trackball or a mechanical push button is possible. Any human interface that can be emulated can be used.

  Furthermore, although the case where a personal computer is operated as the target device X has been described in the present embodiment, the present invention is not limited to this, and a printer device X3 connected to the personal computer may be an operation target.

DESCRIPTION OF SYMBOLS 100 ... Remote operation system, 121 ... Screen acquisition part, 122 ... Image determination part, 123 ... State provision part, 131 ... Operation reception part, 132 ... Operation signal output part, 133 ... Operation correctness determination part, 141 ... Capture apparatus, 142 ... control interface, X ... target device, X1 ... monitor, X2 ... keyboard (human interface), X3 ... printer, Y ... remote control device, DB ... status database, NW ... network.

The remote operation system of the first invention is a remote operation system for remotely operating the target device from the remote operation device based on the state of the monitor screen of the target device.
A screen acquisition unit that captures a monitor screen of the target device via a capture device and acquires it as an image;
An image determination unit that determines the state of the image from the image acquired from the screen acquisition unit;
A state providing unit that provides the remote control device with the state determined by the image determining unit;
An operation receiving unit that receives an operation from the remote control device;
An operation signal output unit that converts the operation received by the operation reception unit into an operation signal of the target device and outputs the operation signal to the target device via a control interface ;
The image determination unit determines whether the target device is in a screen state that can be operated as the state of the image from the image acquired from the screen acquisition unit,
The state providing unit generates an operation input screen corresponding to the operable image state and provides it to the remote operation device when the state of the image is an image state operable by the target device. Features.

Further, according to the remote operation system of the first invention, in advance, the image determining unit, an image obtained from the screen acquisition unit determines whether the target device is capable of screen state operation, an image of the state target device is operated When the image state is possible, the state providing unit generates an operation input screen corresponding to the operable image state and provides the operation input screen to the remote operation device. Therefore, an operation input screen suitable for remote operation can be provided to the remote operation device.

Thus, according to the remote operation system of the first invention, even if the operation target device does not have a communication function or the display content of the monitor screen of the operation target device is not recognized on the remote operation device side, the remote operation is performed. The operation target device can be easily and surely operated by the remote operation device through the operation input screen suitable for the operation.

Remote control system of the second invention, in the first shot Akira,
The operation signal output unit is connected in a superimposed manner to a human interface (for example, a keyboard) of the target device, and converts the operation received by the operation reception unit into an operation signal of the target device as an emulation signal of the human interface. And outputting to the target device.

According to the remote operation system of the second invention, the operation received by the operation receiving unit is converted into the operation signal of the target device and output to the target device as an emulation signal of the human interface of the target device.

Therefore, according to the remote operation system of the second invention, even when the operation target device does not have a communication function or an external connection input, the operation target device can be easily and reliably operated by the remote operation device.

The remote control system of the third invention is the first or second invention,
After the operation signal is output to the target device by the operation signal output unit, an operation correctness determination unit that determines operation correctness from the state of the image acquired via the screen acquisition unit is provided.

According to the remote operation system of the third invention, after the operation correctness determination unit outputs the operation signal to the target device by the operation signal output unit, the operation correctness determination is performed from the state of the image acquired through the screen acquisition unit. Do. For this reason, it is possible to confirm whether the remote operation has been reliably executed by the change in the state of the target device.

As described above, according to the remote operation system of the third invention, even when the operation target device does not have a communication function or when the display content of the monitor screen of the operation target device is not recognized on the remote operation device side, The operation target device can be reliably operated by the remote operation device, and the execution of the operation can be confirmed.

Claims (4)

In a remote operation system for remotely operating the target device from a remote control device based on the state of the monitor screen of the target device,
A screen acquisition unit that captures and acquires the monitor screen of the target device as an image;
An image determination unit that determines the state of the image from the image acquired from the screen acquisition unit;
A state providing unit that provides the remote control device with the state determined by the image determining unit;
An operation receiving unit that receives an operation from the remote control device;
A remote operation system comprising: an operation signal output unit that converts the operation received by the operation reception unit into an operation signal of the target device and outputs the operation signal to the target device. The remote control system according to claim 1, wherein
The image determination unit determines whether the target device is in a screen state that can be operated as the state of the image from the image acquired from the screen acquisition unit,
The state providing unit generates an operation input screen corresponding to the operable image state and provides it to the remote operation device when the state of the image is an image state operable by the target device. Features remote control system. The remote control system according to claim 1 or 2,
The operation signal output unit is connected in a superimposed manner to the human interface of the target device, and converts the operation received by the operation reception unit as an emulation signal of the human interface into an operation signal of the target device. Remote operation system characterized by outputting to equipment. The remote control system according to any one of claims 1 to 3,
A remote operation comprising an operation correct / incorrect determination unit for determining operation correct / incorrect from the state of an image acquired via the screen acquisition unit after an operation signal is output to the target device by the operation signal output unit system.