WO2024096085A1 - Facility control system, facility control device, and facility control method - Google Patents

Abstract

The present invention is a facility control system for controlling operation of a facility with a serial signal obtained on the basis of an RPA scenario. An image recognition unit (512) recognizes an image of the operation state of the facility, such state being in response to the control. An input display signal utilization and manipulation device (611) transmits, to a facility control device (101), the serial signal which is generated on the basis of a recognition result of the image by the image recognition unit (512) and on the basis of the RPA scenario. A display panel emulation function (614) included in the input display signal utilization and manipulation device (611) generates a panel display signal for executing a next sequence specified by the RPA scenario, and passes this signal to an emulator display panel device (617), the panel display signal being furthermore encoded by an operation position encoder (618).

Description

Facility control system, facility control device, and facility control method

Embodiments of the present invention relate to an equipment control system, an equipment control device, and an equipment control method.

There are various types of equipment in factories. The possible operating periods for this equipment include the operating period when the equipment starts up (warms up), the operating period when the equipment is manufacturing something, the operating period when the equipment stops (cools down), and the period when the equipment is stopped.

Furthermore, in each operating period, there are multiple processes in which the operating state and operating functions of the equipment change over time. Here, a manual operating device is used to give the equipment's drive system an operating signal to move the equipment from the current process to the next process. Here, a "process" is primarily equipment used for manufacturing, for example, and refers to each stage in which parts are placed, processed, and moved. Therefore, a process refers to a so-called manufacturing process in which multiple operations are performed on equipment in a step-by-step procedure.

These manual operations are performed by an operator. The operator visually checks the display contents of the buttons and lamps of the manual operation device, determines whether the next operation is possible, and manually performs the operation for the process transition.

However, this operation system requires a full-time operator to be present at each facility. Also, to ensure proper operation, a period of time is required to educate the operators and develop them into skilled operators. As a result, in the past, a lot of manpower and training time was required to operate the facilities.

Therefore, to reduce manpower and training time, there is a technology that focuses on the operating procedures (or operation patterns) for transitioning between each process of equipment. With this technology, a robot automatically outputs the operation signals required for each process according to the operating procedures. This type of operation by a robot is called RPA (Robotic Process Automation).

JP 2022-14215 A

Incidentally, touch panel type operation devices are becoming more common as manual operation devices for controlling equipment. However, these touch panel type operation devices also require manual operation, and it cannot be said that they have solved the labor shortage or reduced the training and education period for operators. Furthermore, the layout of the operation screen of touch panel type operation devices may differ depending on the equipment and manufacturer.

In addition, while the above-mentioned touch panel type operation device is one in which the user touches operation buttons displayed on the screen, there are also click panel type operation devices. This click panel type operation device is a type in which the user operates a mouse, aligns the cursor with the position of a specific operation button, and performs a click operation via the mouse. For this reason, hereafter, the term "display panel device" will be used to refer to both touch panel type operation devices and click panel type operation devices.

The problem that this invention aims to solve is to provide an equipment control system, equipment control device, and equipment control method that reduces the burden on users (operators) who use equipment, devices, systems, etc., and that effectively utilizes the display signal of a display panel device when controlling equipment equipped with a display panel device based on an RPA scenario.

According to one embodiment of the present application, there is provided an equipment control system that controls the operation of an equipment to be operated by a serial signal obtained based on a predefined robotic process automation scenario (RPA scenario), the equipment control system having an image recognition unit that recognizes an image from an on-site display panel device that displays, via an equipment control device, the operation status of the equipment to be operated that responds to the control, and an input display signal utilizing manipulation device that transmits the recognition result by the image recognition unit and the serial signal generated based on the RPA scenario to the equipment control device, the input display signal utilizing manipulation device having a display panel emulation function, the display panel emulation function generating a signal for executing the next sequence specified by the RPA scenario as a panel display signal, providing this panel display signal to an emulator display panel device, and further providing the panel display signal to an operation position encoder, thereby encoding the panel display signal into the serial signal.

FIG. 1 is a system configuration diagram showing an overview of an example of a facility control system to which an embodiment is applied. FIG. 2 also shows the equipment control system of FIG. 1, but is a configuration explanatory diagram showing an overview of an example of the internal configuration of a secondary control system. FIG. 3A is an explanatory diagram showing an example of the operation of the equipment control system shown in FIGS. 1 and 2 in the form of a time sequence. FIG. 3B is a diagram illustrating the operation following FIG. 3A. FIG. 4 is a diagram for explaining the relationship between the display panel device and the click/touch conversion unit in FIGS. 3A and 3B. FIG. 5A is an explanatory diagram showing an example of a main operation screen of the emulator display panel device of FIG. FIG. 5B is an explanatory diagram showing an example of the main operation screen when the emulator display panel device of FIG. 2 is in the viewing mode off. FIG. 5C is an explanatory diagram showing an example of a screen when the screen of the emulator display panel device of FIG. 2 is switched to a setting operation screen. FIG. 5D is an explanatory diagram showing an example of a screen when the screen of the emulator display panel device of FIG. 2 is switched to a version information confirmation screen. FIG. 6 is an explanatory diagram showing an example of a screen transition when moving from the screen of FIG. 5A or FIG. 5B to the setting operation screen of FIG. 5C. FIG. 7 is an explanatory diagram showing an example of a screen transition when moving from the screen of FIG. 5A or FIG. 5B to the setting operation screen of FIG. 5D. FIG. 8 is an explanatory diagram of an image displayed on the screen of the emulator display panel device when the secondary control system of FIGS. 3A and 3B is connected to the primary control system and started. FIG. 9 is an explanatory diagram showing an example of a screen when the screen of the emulator display panel device of FIG. 8 is switched to a setting operation screen. FIG. 10 is an explanatory diagram showing an example of a screen when the screen of the emulator display panel device of FIG. 8 is switched to a version information confirmation screen. FIG. 11 is a diagram showing another example of the embodiment of the present invention. FIG. 12 is a diagram showing yet another example of the embodiment of the present invention and explaining the operation of audio signal processing.

Hereinafter, an embodiment will be described with reference to the drawings.
<Explain the overall overview>
1 is a schematic diagram of an equipment control system according to an embodiment. Reference numeral 11 denotes a factory, and reference numeral 13 denotes an office. The factory 11 and the office 13 are connected by an in-house LAN 12. A primary control system 100, a secondary control system 200, and manufacturing-related equipment 15 are installed in the factory 11. The manufacturing-related equipment 15 includes equipment for directly carrying out manufacturing, as well as equipment for adjusting the environment of the equipment (temperature control equipment, air conditioning control equipment, equipment for supplying and transporting materials, parts, fuel, etc.).

The manufacturing-related equipment 15 is directly controlled by a primary control system 100 that is located at the site where the manufacturing-related equipment 15 is installed.

The primary control system 100 is basically controlled based on a robotic process automation scenario (hereinafter referred to as an RPA scenario) 615 (shown in FIG. 2) provided in the secondary control system 200. Therefore, it can be said that the secondary control system 200 indirectly controls the manufacturing-related equipment 15.

This secondary control system 200 is controlled by a tertiary control system 300 installed in an office 13 via an in-house LAN 12. Therefore, it can be said that the tertiary control system 300 indirectly controls the manufacturing-related equipment 15 via the second control system 200 and the primary control system 100.

Each of the control systems 100, 200, 300 will be further described hereinafter.
The primary control system 100 has an equipment control personal computer (hereinafter referred to as an equipment control device or equipment control PC) 101, an on-site operation display panel device (which may also be referred to as an on-site display panel device) 102, and branch switches 103, 104 provided between the equipment control PC 101.

The branching device 103 is disposed on the notification route of the response signal (which may also be called a status notification signal, display signal, or display pattern signal) from the equipment control PC 101. The response signal (display signal) is input to the on-site display panel device 102 via the branching device 103, and is also input to the display signal capture unit 105. On the other hand, the branching device 104 is disposed on the transmission route of an operation signal (command signal) generated by a touch (or click) operation to the equipment control PC 101. In addition, since the equipment control PC 101 has the function of OCR (Optical Character Recognition) processing, it is also possible to send an OCR signal to the outside.

The above operation signal (command signal) is given as a serial signal from the secondary control system 200, which will be described later. In the secondary control system 200, when a user touches an operation button displayed on the display panel, the coordinate data of the touch position becomes a command signal (serial signal) and is sent to the equipment control PC 101 of the primary control system 100. Note that instead of the above "touch," a cursor on the screen may be moved to the position of the operation button with a mouse and then clicked. In the following explanation, the signal generated by a touch operation or click operation is referred to as an operation signal.

The above coordinate data (operation signal) controls the equipment control PC 101 via the branching device 104, which then controls the manufacturing-related equipment 15. Furthermore, the operating status of the manufacturing-related equipment 15 is also given to the on-site display panel device 102 via the branching device 103. Then, the operating status on the display panel in the secondary control system 200 is reproduced on the on-site display panel device 102. Therefore, the automatic operating status based on the RPA scenario in the secondary control system 200 can be confirmed on the on-site display panel device 102. In other words, the display signal displayed on the on-site display panel device 102 indicates the control status of the manufacturing-related equipment 15 and the equipment response status based on the control. Therefore, the on-site user can confirm the control status and response status of the manufacturing-related equipment 15 by visually checking the lighting position of the display button, the status of the displayed graph, or the displayed numerical values, etc.

The secondary control system 200 will be explained in detail later with reference to Figure 2, but will be explained briefly here.

The secondary control system 200 receives the display signal via the branching device 103 and display signal capture unit 105, and displays the display signal on the screen of the emulator display panel device 617 (shown in FIG. 2). This allows the user to recognize the control state and response state of the manufacturing-related equipment 15 on the screen of the emulator display panel device 617. The secondary control system 200 is equipped with a robotic process automation scenario (RPA scenario) 615 (shown in FIG. 2), and can generate each operation signal required for multiple processes for the manufacturing-related equipment 15 in accordance with the scenario. Multiple operation signals may be generated simultaneously depending on the operation content.

This operation signal is input to the equipment control PC 101 via the branching device 104. The equipment control PC 101 controls the manufacturing-related equipment 15 based on the operation signal. The control state and response state of the manufacturing-related equipment 15 are grasped by the equipment control PC 101, and a display signal (which may be called a display signal or equipment state signal) corresponding to (representing) the control state and response state is input to the secondary control system 200 via the branching device 103 and the display signal capture unit 105.

When the operation starts, the initial full-screen data of the display signal is sent to the secondary control system 200. If automatic operation of the manufacturing-related equipment 15 is started thereafter, a partial display signal, that is, a part of the display signal (including the response signal of the equipment itself) in response to the operation (touch position or click position) is sent to the secondary control system 200.

This secondary control system 200 can further transmit a display signal indicating the status of the manufacturing-related equipment 15 to the tertiary control system 300 (inside the office 13).

Furthermore, in this system, as shown in FIG. 1, either a microphone 702 or a speaker 703, or both, may be placed near the manufacturing-related equipment 15. The microphone 702 and the speaker 703 are connected to an audio signal input/output terminal 701 of the secondary control system 200. The audio signal is used, for example, to check for abnormal or normal operating sounds of the manufacturing-related equipment 15. The speaker 703 is also used when instructions are given to on-site workers from a remote office 13.

FIG. 2 shows each functional block in the secondary control system 200 .
The display signal for the on-site display panel device output from the primary control system 100 is captured by the display signal capture unit 105 and input to an image recognition section 512 in the secondary control system 200 via a USB cable and a USB terminal 511. This image recognition section 512 may be incorporated in a manipulator using input display signal 611 or a display panel emulation function 614, which will be described later.
The secondary control system 200 further includes, as external connection functions, a communication device 513 for connecting to the tertiary control system 300 and an output terminal 514 for sending operating signals to the primary control system 100 .

The secondary control system 200 includes an input display signal utilizing manipulation device (hereinafter may be simply referred to as a manipulation device, etc.) 611 connected to the image recognition unit 512 inside. The manipulation device 611 includes a display panel emulation function 614 and an RPA scenario 615 (the data of this scenario may be stored in a buffer circuit or a storage device 613). The manipulation device 611 further includes a system control unit (which may be referred to as a system CPU) 621 that controls the overall operation sequence of the system, security functions, etc., and a storage device 613. The system control unit (which may be referred to as a system CPU) 621 can cause the emulation function 614 to execute various operations based on software stored in the storage device 613.

The display panel emulation function 614 includes a display panel control unit 616, and controls an emulator display panel device 617 and a click/touch conversion unit (which may also be referred to as an operation position encoder) 618 via the display panel control unit 616. Furthermore, the display panel emulation function 614 controls the manufacturing-related equipment 15 by operating the primary control system 100 based on the RPA scenario.

Furthermore, the display panel emulation function 614 outputs the operating status of the display panel control unit 616 as log data. For example, the operation signal output via the output terminal 514 to control the manufacturing-related equipment 15 can be output from the output unit 622 as an execution log (which may also be called an operation log). In this case, the execution log includes not only the operation signal but also sequence information of the RPA scenario (e.g., process number) to identify the operation process. Furthermore, the execution log includes not only the execution log of the operation signal, but also the display signal (display signal of characters, graphs, etc.) recognized by the image recognition unit 512 and the corresponding sequence information.

If a crash occurs during an operation process, it can be output as a crash log from the output unit 623. In this case, the crash log also includes sequence information of the RPA scenario (e.g., the process number) to identify the process of the scenario.

The above execution logs and crash logs are stored in the storage device 613. As described above, there are at least the following types of logs: an execution log (reception log) when the display panel emulation function 614 recognizes a display signal input from the outside, an execution log (output log) for controlling the on-site display panel device 102, and a crash log. These log data contain the various reference information mentioned above, such as the time of occurrence. For this reason, past log data can be used for various analyses and verifications.

Furthermore, the display panel emulation function 614 is connected to an operation mode setting unit 624. This operation mode setting unit 624 is used when setting the operation mode for the display panel emulation function 614 from the outside. For example, it is possible for the tertiary control system 300 to give a pause command to the display panel emulation function 614, to write new software to the storage device 613, or to modify, rewrite, or add a part of the RPA scenario. Furthermore, in addition to being able to modify or rewrite a part of the RPA scenario 615, the RPA scenario 615 may be replaceable with a memory card storing another RPA scenario.

The display panel emulation function 614, which will be explained in detail later, can set various functions via the operation screen. For example, it is possible to obtain consistency between the operation of the emulator display panel device 617 and the operation of the on-site display panel device 102 of the primary control system 100. Items that can be set include the signal method, serial port settings, display panel coordinate offset settings, display signal capture device settings, frame rate, etc.

Furthermore, the audio signal input/output terminal 701 is connected to an audio processing unit 704, and this audio processing unit 704 can be linked to the business PC 301 in the office via the communication device 513. The audio processing unit 704 may be configured integrally with the display panel emulation function 614, or may be provided independently. Furthermore, the input/output data of the audio processing unit 704 may be stored in the storage device 613.

3A and 3B show the mutually related operations of the above-mentioned first control system 100, second control system 200, and third control system 300 over time. An overview of the system operations will be described with reference to FIG. 2, FIG. 3A, and FIG. 3B.
<Explanation of the first step operation>
Now, assume that a user performs a certain operation on the manufacturing-related equipment 15 using the display panel of the business PC 301 of the tertiary control system 300. In the example shown in the figure, the operation may be any operation such as starting up the manufacturing-related equipment 15 or starting production.

The business PC 301 determines the operation content of the operation instruction from the operator, such as a startup operation or an operation to start manufacturing (S11), and sends a work start signal to the secondary control system 200. Then, the display panel emulation function 614 of the secondary control system 200 recognizes the content of the work start signal (S21). Next, the display panel emulation function 614 identifies a scenario corresponding to the work from among multiple RPA scenarios 615 (S31). For example, the display panel emulation function 614 identifies the work code included in the work start signal and the RPA scenario to which this work code is added and managed, and notifies the display panel control unit 616 of the identification information.

The display panel control unit 616 recognizes the scenario, reads out a display signal from the storage device 613 to display the initial operation screen corresponding to that scenario, and provides this to the emulator display panel device 617.

Furthermore, the display panel emulation function 614 generates an operation signal (coordinate data) for executing the first operation step set in the scenario via the display panel control unit 616. This operation signal is given to the emulator display panel device 617, and the screen of the emulator display panel device 617 changes to a state that reflects the operation signal. At the same time, the operation signal (coordinate data) is converted by the click/touch conversion unit (operation position encoder) 618 into a serial signal of a specified format, and is output to the output terminal 514 (S22).

The primary control system 100 receives the serial signal. Note that the display state of the on-site display panel device 102 is the initial screen when the power of the primary control system 100 is turned on.

When the on-site display panel device 102 receives the serial signal, the display state becomes the same as that of the emulator display panel device 617 (S51).

The above serial signal is also input to the equipment control PC 101. The equipment control PC 10 determines the operation content based on the serial signal and provides an operation (control) signal to the manufacturing-related equipment 15 (S41).

The manufacturing-related equipment 15 operates based on the control signal from the equipment control PC 101 and executes its response action (S61). This response action is confirmed by the equipment control PC 101 (S42). There are various ways to check this, for example, it is checked based on the output state of sensors, encoders, etc.

When the equipment control PC 101 confirms the response operation, it returns a confirmation signal (specifically, for example, coordinate data for displaying an operation completion button) to the on-site display panel device 102 and the display panel emulation function 614 (S43). This causes the on-site display panel device 102 to display an indication that the operation of the first step has been performed (S52). Note that in this embodiment, the coordinate data also indicates (includes) a VGA signal. The display panel emulation function 614 also recognizes that the operation of the first step has been performed (S23) and controls the emulator display panel device 617. This causes the emulator display panel device 617 to also display a status indicating that the operation of the first step has been performed (S23).

Furthermore, the secondary control system 200 also transmits display information to the tertiary control system 300 indicating that the operation of the first step has been performed. The display information at this time may be in the same format as the previous coordinate data (serial signal), or in another format.

As a result, the same information is displayed on the display device connected to the business PC 301 of the tertiary control system 300, the emulator display panel device 617 installed in the secondary control system 200, and the on-site display panel device 102 installed in the primary control system 100 (S12).

Furthermore, the secondary control system 200 (specifically, the manipulation device 611 or the display panel emulation function 614) generates an execution log indicating that it has at least outputted the serial signal instructing the operation of the first process to the output terminal 514 and recognized that the operation of the first process has been executed. This execution log is stored, for example, in the storage device 613. The data in the storage device 613 is periodically controlled by the system control unit 621 and transmitted to the tertiary control system 300, or transmitted in response to a request from the tertiary control system 300.
<Explanation of the second step operation>
Furthermore, the display panel emulation function 614 checks the scenario for the next process operation (second process operation) of the RPA scenario (S23).

Then, based on confirmation of this scenario, the display panel emulation function 614 generates an operation signal (coordinate data) for executing the second operation step set in the scenario via the display panel control unit 616. This operation signal is given to the emulator display panel device 617, and the screen of the emulator display panel device 617 changes to a state that reflects the operation signal. At the same time, the operation signal is converted by the click/touch conversion unit (operation position encoder) 618 into a serial signal of a specified format, and is output to the output terminal 514 (S24).

In this case, in the primary control system 100, the display state of the on-site display panel device 102 becomes the same as that of the emulator display panel device 617 based on the serial signal (S53). The serial signal is also input to the equipment control PC 101. The equipment control PC 101 determines the operation content based on the serial signal, and provides a control signal for equipment operation to the manufacturing-related equipment 15 (S44).

The manufacturing-related equipment 15 operates based on the control signal from the equipment control PC 101 and executes its response action (S62). This response action is confirmed by the equipment control PC 101 (S45).

When the equipment control PC 101 confirms the response operation, it sends a confirmation signal (specifically, coordinate data for displaying the operation confirmation button) back to the on-site display panel device 102 and the display panel emulation function 614 (S46). This causes the on-site display panel device 102 to display an indication that the operation of the second process has been performed (S54). The display panel emulation function 614 also recognizes that the operation of the second process has been performed (S25) and controls the emulator display panel device 617. This causes the emulator display panel device 617 to also display an indication that the operation of the second process has been performed.

Furthermore, the second control system 200 also transmits display information to the third control system 300 indicating that the operation of the second process has been executed. Furthermore, the display panel emulation function 614 checks the scenario for the next process operation (second process operation) of the RPA scenario (S25).

If the next scenario exists, operations similar to the first and second process operations described above are repeated. Steps S26, S55, S47, S63, S48, S45, S56, and S27 represent the sequence when the nth process operation is executed. When the display panel emulation function 614 confirms the final process of the scenario in step S22, it causes the emulator display panel device 617 to display that all scenarios (operation processes) have been completed. The display panel emulation function 614 also notifies the tertiary control system 300 that the final process has been completed. Each process of the scenario is identified by sequence information.

At this stage, the tertiary control system 300 may request the log data stored in the storage device 613 from the display panel emulation function 614 (S15).

If the above first to nth process operations are to be further repeated, the final scenario of the RPA scenario describes instructions for returning to the first process operation. Furthermore, the final scenario may include time information, period information, and information regarding the maximum number of repetitions to limit the repeated operations.

FIG. 4 is a diagram for explaining the relationship between the emulator display panel device 617 shown in FIGS. 2, 3A, and 3B and the click/touch conversion unit (operation position encoder) 618. Here, the functional block of the click/touch conversion unit (operation position encoder) 618 is particularly shown. When the emulator display panel device 617 is operated (touched or clicked), position data of the touch or click position is acquired (detected) (68a).

In the case of a touch panel, for example, pixel electrodes are arranged two-dimensionally within a display panel to display an image. Also, touch detection electrodes are arranged two-dimensionally within the display panel to detect the touch position. The touch position detection function detects that a change in capacitance occurs between the detection electrode and the common electrode when a touch operation is performed. In other words, the detection electrode with the changed capacitance is detected via a readout line, and the detection signal at this time is used as touch position detection data.

On the other hand, the click position detection function displays a cursor (pointer) of a specific pattern and level within the image. This function has a cursor movement control circuit that can move the cursor to any position on the image in response to the user's mouse operation. This function also has a cursor tracking circuit that constantly tracks changes in the cursor's coordinate data on the screen and grasps the cursor's current position data. This function also has a latch circuit that latches the current position data (coordinate data) when a click operation is performed from the mouse. The output of this latch circuit is used as click position detection data (coordinate data).

The above click or touch position detection data (coordinate data) is then temporarily stored in the coordinate acquisition unit 68a. The coordinate conversion function 68b then converts the data into coordinate data that is consistent with the on-site display panel device 102 based on a preset offset value. This coordinate data is then sent as an operation signal to the branching device 104 of the primary control system 100 by the operation signal sending function 68c. At this time, the operation signal sending function 68c outputs the operation signal at this time to the output unit 622 as an execution log.

As described above, in this system, the display panel emulation function 614 has a click/touch conversion unit (operation position encoder) 618, and therefore can transmit coordinate data (operation signal) that matches the on-site display panel device 102.

FIGS. 5A, 5B, 5C, and 5D show examples of various screens 617a, 617b, 617c, and 617d that can be displayed by the emulator display panel device 617. These screen examples show that the emulator display panel device 617 is designed to have high compatibility with various types of on-site display panel devices 102.

FIG. 5A is an explanatory diagram showing an example of a main operation screen 617a of the emulator display panel device 617 of FIGS. 3A and 3B. In the main operation screen 617a, a "Start button" for the operation is displayed in the display area 67a, and a "Stop button" for the operation is displayed in the display area 67b.

When the main operation screen 617a is displayed, "Touch Panel Emulator" is displayed as the title bar. A menu bar is also displayed, with the words "File (F)" 67g and "Help (H)" 67h written in the menu bar area. These are operation buttons for moving to the settings screen, which will be described later. These utilize Windows (registered trademark) functions.

Within the screen,
The individual identification number (here, "0123") that the manufacturing equipment is to process is displayed in the display area 67d.
The display area 67e displays the product identification name (here, "abcd") that the manufacturing facility is going to process.
The display area 67f displays a sensor setting value such as pressure for processing by the manufacturing equipment or a sensor value during processing (here, "9876").
The display area 67c displays in graph form the change in the sensor value while the manufacturing equipment is processing.
The above is an example of a screen at the stage of adjustment after the secondary control system 200 is connected to the primary control system 100 and the equipment 15 is test-run. This screen is an example, and can be changed arbitrarily according to the user's request for control or monitoring purposes, and may be switched to a state in which multiple items are displayed simultaneously, or a state in which individual items are displayed individually. For example, when a special operation is performed in an emergency, the display form is automatically switched together with the switching of the RPA scenario. In this case, the emergency RPA scenario is read, and the corresponding video data is read from the storage device 613. In this case, the on-site display panel 102 on the primary control system side is assumed to have image data to be displayed in an emergency from the beginning.

FIG. 5B is a diagram showing an example of a screen 617b when the emulator display panel device 617 in FIGS. 3A and 3B displays the browsing mode off. When the browsing mode off screen 617b is displayed, "Touch panel emulator browsing mode off" is displayed in the title bar. This screen 617b also has operation buttons "File (F)" 67g and "Help (H)" 67h in the menu bar area for moving to the settings screen.

FIG. 5C shows the settings screen 617c that is displayed when "File (F)" 67g on the menu bar is touched or clicked.

This screen 617c has a setting section for setting the operation signal to be output as a serial signal from the output terminal 514.
Box 67j is a box for setting a serial port, and a port to be connected to the corresponding primary control system 100 is selected from among a plurality of serial ports.
Box 67k is a box for selecting a parity bit to be added to the serial signal.
Box 67m is a box for selecting the number of transmission bits per second of the serial signal.
Box 67n is a box for specifying control data for executing flow control. There are various types of primary control systems that constitute this system, and therefore the processing speed differs depending on the primary control system, and this setting unit exists to accommodate that diversity.
When flow control is used, for example, the following preparations are made in advance: The designer or user sends a serial signal based on the system's RPA scenario A1 to the primary control system. Next, the response time of the display signal returning from the primary control system is measured in advance. Then, when sending the next serial signal based on the next RPA scenario A2, the response time is set according to the response time. That is, the delay time at which the next serial signal is output from the buffer circuit is set to an appropriate timing so that the primary control system can reliably receive the serial signal. Note that the above setting program is assumed to be installed in the input display signal utilization manipulation device 611.

Box 67p is a box for setting a stop bit at the end of the serial signal to accommodate asynchronous operation between systems. Note that the box does not necessarily have to be a box, but can be any mark that allows an operation to be performed.

Also, boxes 67q and 67r are provided to accommodate touch/click panel coordinate offsets. That is, the display coordinate data of the various buttons on the on-site display panel device 102 of the primary control system 100 and the display coordinate data of the various buttons on the emulator display panel device 617 do not necessarily correspond one-to-one. There is an offset in the coordinates at the operation positions of the corresponding operation buttons on the on-site display panel device 102 and the emulator display panel device 617. Therefore, in this system, it is possible to input offset values, for example, of the X and Y coordinates of the upper left and the X and Y coordinates of the lower right on the screen. The user may make the adjustments at this time by visually checking both the on-site display panel device 102 and the emulator display panel device 617.

As described above, at least the display panel emulation function 614 can display a setting screen on the emulator display panel device 617, and can display data input boxes 67q, 67r on this setting screen for aligning the X and Y coordinate offsets at the top right and bottom left of the screen between the display image of the emulator display panel device 617 and the display image of another on-site display panel device 102.

The emulator display panel device 617 of this system can have an operation screen mode and a viewing screen mode. Therefore, a button for setting the viewing mode on and off is also provided. Furthermore, the secondary control system 200 receives display signals from the primary control system 100 via the display signal capture unit 105. Therefore, it is also possible to specify the device ID of the display signal capture unit 105. A box 67t is provided for this purpose. The display signal capture unit 105 is sometimes called a video capture device.

There is also a box 67u for setting the frame rate at which the display signal is processed (received). This is because, depending on the equipment, the time intervals for each of the operation steps described in Figures 3A and 3B may be long. In such cases, there is no need to monitor (recognize) the same screen contents (operation status) multiple times in one operation step. Therefore, in such cases, the frame rate is made adjustable.

The following buttons are provided for each of the above settings so that the user can confirm them after checking the manual or specifications, for example. If the settings are correct, an "OK" button is provided to confirm the settings, and if the settings are incorrect, a "Cancel" button is provided to cancel the settings.

FIG. 5D is an explanatory diagram showing an example of yet another operation screen 617d of the emulator display panel device 617 of FIGS. 3A and 3B. This screen 617d is displayed when the "Help (H)" 67h area of the menu bar is selected on screen 617a or 617b.

This screen 617d displays version information for the display panel emulation function 614.

FIG. 6 shows an example of the screen transitions leading up to the screen of FIG. 5C. The same parts as in FIG. 5A and FIG. 5C are given the same reference numerals. When "File (F)" 67g on screen 617a is clicked or touched, a menu screen P11 is displayed in a picture-in on screen 617a. This menu screen P11 displays the items "Settings U," "Snapshot S," and "Exit X."

When the item "Settings U" is clicked or touched, the previously described screen 617c for various settings is displayed. The user uses this screen 617c to configure various settings for the display panel emulation function 614. After completing the settings, pressing the "OK" button or the "Cancel" button returns the screen to the original screen 617a.

When the "Snapshot S" item is operated, the display signal (or equipment status signal) at the time of operation is stored, for example, in the storage device 613, together with the time information and equipment ID at that time. When the "Exit X" item is operated, the window is closed and the application is terminated.

Fig. 7 shows an example of screen transitions leading to the display of screen 617d in Fig. 5D. The same parts as in Fig. 5A and Fig. 5D are given the same reference numerals. When screen 617d in Fig. 5D is displayed,
When "Help (H)" 67h on the screen 617a is clicked or touched, a menu screen P12 is displayed in a picture-in on the screen 617a. This menu screen P12 describes "Version Information A". When the user clicks or touches this "Version Information A", a screen 617d is displayed. On this screen 617d, for example, the product name, its version information, and the manufacturer name of the input display signal utilization manipulation device 611 are displayed. When the "OK" button is operated on the screen 617d, the screen returns to the original screen 617a.

The explanations in Figures 4 to 7 show examples of screens at the stage when the secondary control system 200 is connected to the primary control system 100 and adjustments are being made after, for example, a test run of the equipment 15 has been performed.

Next, referring to Figures 8 to 10, changes in the screen of the emulator display panel device 617 when the secondary control system 200 is connected to the primary control system for the first time, or when the secondary control system 200 is replaced, will be described. In particular, in this system, the emulator display panel device 617 and the on-site display panel device 102 must be consistent in terms of the operation functions for operating the equipment. For this reason, various items must be set for the emulator display panel device 617.

Figure 8 shows an example of an image displayed on the screen of the emulator display panel device 617 when the secondary control system 200 of Figures 3A and 3B is connected to the primary control system 100 and started. Also, Figure 9 shows an example of a screen when the screen of the emulator display panel device 617 of Figure 8 is switched to a setting operation screen. Furthermore, Figure 10 shows an example of a screen when the screen of the emulator display panel device of Figure 8 is switched to a version information confirmation screen. In the following explanation, the same parts as those of Figures 4 to 7 are given the same reference numerals.

As shown in FIG. 8, when the secondary control system 200 is connected to the primary control system 100 and powered on, the emulator display panel device 617 displays a screen that is the same as the operation screen of the on-site display panel device 102 of the primary control system 100. This display data may be sent from the equipment control PC 101 of the primary control system 100, or may be data that has been previously recorded in the storage device 613.

In this way, by displaying the same screen as the operation screen of the on-site display panel device 102 on the manufacturing-related equipment 15 to be controlled, the user is prevented from connecting the secondary control system 200 to the wrong primary control system.

In addition, even if the on-site display panel device 102 is configured to display not only the operation screen but also an external image of the manufacturing-related equipment 15, or an image of its characteristic external appearance, or an image of a mark representing its characteristics, the user can be prevented from connecting the secondary control system 200 to the wrong primary control system.

Next, the user performs settings to match the emulator display panel device 617 with the on-site display panel device 102. For example, when the user touches or clicks on "File (F)" 67g in the menu bar area, a menu screen P11 is displayed as a picture-in on the screen 617a. This menu screen P11 displays the items "Settings U," "Snapshot S," and "Exit X."

When the user touches or clicks "Settings U," settings screen 617c is displayed. This screen displays boxes (marks) for various settings, similar to the settings screen described in FIG. 6. The same parts as in FIG. 6 are given the same reference numerals and will not be described again, but the positions of the boxes on the settings screen in FIG. 6 may differ from those on the settings screen in FIG. 9.

Also, box 67w is displayed, which is a box for setting the data length of one unit of data (D). In the example shown in the figure, 8 bits are set. Note that this display example is just one example, and byte length or the number of bytes that constitute the unit may also be used, following the instruction manual for the secondary control system.

The above setting screen 617c may be replaced with the setting screen described in FIG. 5C and FIG. 6. From the top, the screen in FIG. 9 is divided by frames into a setting area for turning the viewing mode on and off, a setting area for the touch panel touch signal method and touch panel coordinate settings, a setting area for the serial port, and a setting area for the image signal. This makes it easy for the user to understand the types of work involved in each setting.

FIG. 10 shows a state in which “Help (H)” 67h on the screen 617a is clicked or touched, causing a picture-in menu screen P12 to be displayed on the screen 617a, and a state in which “Version Information A” on this menu screen P12 is clicked or touched.
When "Version Information A" on screen P12 is operated, a screen 617d for confirming the "Version Information" is displayed. On this screen 617d, for example, the product name, its version information, and the manufacturer name of the input display signal utilization manipulation device 611 are displayed. When the "OK" button is operated on screen 617d, the screen returns to the original screen 617a.

FIG. 11 is a diagram showing another example of an embodiment of the present invention. The embodiment shown in FIGS. 1 and 2 above shows a configuration example in which the primary control system 100 and the secondary control system 200 have a one-to-one relationship. However, the primary control system 100 and the secondary control system 200 may have a multiple-to-one relationship. When such a configuration is realized, the various setting functions described in FIGS. 5A to 5D, 6, and 7 are effectively utilized in the secondary control system 200.

The example in FIG. 11 is a diagram assuming that multiple (two) primary control systems 100A, 100B control equipment 15A, 15B, respectively. In this case, a selection switch 1100 that allows arbitrary selection of the primary control system is disposed between the secondary control system 200 and the primary control systems 100A, 100B.

This selection switch 1100 can selectively obtain a state in which the secondary control system 200 is connected to the primary control system 100A, and a state in which it is connected to the primary control system 100B. The example in FIG. 11 shows a state in which the secondary control system 200 is connected to the primary control system 100B. In other words, it shows the state in which the selection switch 1100 selects the primary control system 100B.

In this embodiment, the manipulation device 611 of the secondary control system 200 has an RPA scenario 615A for the primary control system 100A and an RPA scenario 615B for the primary control system 100B prepared therein. In addition, a switch control signal for switching the system control state of the selection switch 1100 is output from the system control unit 621 of the secondary control system 200. The switching timing of the selection switch 1100 is set in relation to the repetition frequency of the period required to control and monitor the equipment 15A of the primary control system 100A and the repetition frequency of the period required to control and monitor the equipment 15A of the primary control system 100B. For example, it is possible to set the setting so that "10 seconds" out of 30 seconds are assigned to the control and monitoring of the equipment 15A and the remaining 30 seconds are assigned to the control and monitoring of the equipment 15B. In this case, the color of the background screen of the emulator display panel may change at the timing of the control and monitoring of the equipment 15A and the equipment 15B.

The present invention is not limited to the above embodiment. The input display signal utilizing manipulation device 611 may further have an operation function for controlling the brightness of the on-site display panel device 102. For example, the period during which the brightness is bright can be set to only when a user is present on-site. This setting may be performed by an operator of the tertiary control system 300. Having this function makes it possible to save power and conceal operation buttons, etc.

In this system, the display of operation status and equipment operation status uses coordinate data of display signals such as buttons displayed on the panel. It is possible to change the display position of buttons etc. periodically or randomly, and also to easily change the coordinate data for displaying operation signals and equipment operation status signals. This allows confidential processing regarding equipment operation. At the same time, the brightness of the display panel device can be reduced to prevent images from being seen from outside, further improving confidential processing.

In order to take the above measures, software for matching (synchronizing) coordinate data is added to the transmitting side and the receiving side. For example, the emulator display panel device 617 and the on-site display panel device 102 are provided with a software switching button (which may not be displayed normally). For example, a special scenario is set that occurs when the RPA scenario is repeated many times (a preset number of times). This special scenario is not coordinate data indicating a touch position or click position for equipment operation, but a scenario for generating coordinate data (coordinate data for switching operation modes) for specifying the software switching button. When the emulator display panel device 617, the on-site display panel device 102, and the on-site equipment control PC 101 recognize the above coordinate data, the coordinate positions of the respective operation buttons are shifted. In addition, the on-site equipment control PC 101 is provided with an adapter that switches the recognition function for the coordinate data for the RPA scenario according to the coordinate data for switching operation modes. This makes it possible to realize a function for concealing the operation state of the RPA scenario.
This is just one example, and the present system uses coordinate data and an RPA scenario as described above. Therefore, various measures for system control can provide a function (adapter) for changing coordinate data and responding to the change, thereby ensuring confidentiality of the operation status.

Although Figures 5A and 5B show an example in which the equipment operation status is displayed on the emulator display panel device 617, any image (such as a landscape image) may be displayed instead. Furthermore, the "equipment operation status display" and "landscape image display" may be switched between as desired.

Providing this function is particularly useful when the secondary control system and the tertiary control system are remote, or when the network between them uses an external network (including the Internet). Furthermore, such an operation (or control) system can be applied in other fields as well.

According to the above-mentioned equipment control system, equipment control device, and equipment control method, the on-site display panel device of the primary control system and the input display signal utilizing manipulation device of the secondary control system can grasp the operation status and the equipment operation status of each other by using the display signal. The on-site display panel device and the emulator display panel device can share the equipment operation signal. Unlike what is generally called a command (machine language), the display signal can be visually recognized by the user on the display device. The RPA scenario can also determine the operation status and the equipment operation status from the display signal and determine the scene of the next process. For this reason, even if the equipment is from various countries with different languages and the language displayed on the on-site panel display device of the primary control system is different from the language used in the secondary control system, the operation status and the equipment operation status can be grasped by the coordinate data. That is,
In the above embodiment, the coordinate data of the display panel can be signals for various commands and instructions. Also, the coordinate data can be signals for notifications, responses, etc., which is unique. By using a display device, this system can be easily applied by creating an RPA scenario for each control system even if the software, language, or command format is different.

FIG. 12 shows yet another example of an embodiment of the present invention and is a diagram explaining the operation of audio signal processing. The tertiary control system 300 has a function that enables selective or simultaneous monitoring of the operating status of multiple factories (first factory 11A, second factory 11B, third factory 11C) via the in-house LAN 12. The in-house LAN 12 may be replaced with the external Internet.

The business PC 301 of the tertiary control system 300 is connected to a display device 311, and images 321, 322, and 323 showing the same operation status as the enumerator display panel device 617 of multiple factories (11A, 11B, and 11C) can be displayed on the display device 311. Below the images 321, 322, and 323 showing the operation status, the factory names, for example, of the first factory, the second factory, and the third factory, are displayed.

The illustrated example shows an example in which images 321, 322, and 323 showing the operating status of the first factory, second factory, and third factory are displayed simultaneously. This display is obtained by operating button 324 labeled "simultaneous." Here, touching or clicking the left or right arrow of scroll button 325 slides the screen to the left or right. Furthermore, when the screen has been slid, touching or clicking "Enlarge" button 326 enlarges the screen located in the center. In the illustrated example, operating "Enlarge" button 326 enlarges the screen of the second factory.

The display device 311 displays a screen with the same contents as the emulator display panel device 617. It is also possible to perform various settings such as those described in Figures 6, 7, 9, and 10 from the tertiary control system 300 installed in the office.

Next, the audio processing system will be explained. Audio signals collected near the manufacturing-related equipment 15 are output from a speaker 337 installed on the display device 311 in the office. The collected audio signals can also be analyzed by an analysis device 351 built into the business PC 301.

The analysis device 351 analyzes the frequency and level of the audio signal, for example, and pairs the waveform data when the equipment is operating stably with the scene identification data of the RPA scenario when that waveform is obtained, and stores them in the storage device. If the waveform data of the audio signal becomes a pattern or value different from when the equipment is stable, the analysis device 351 displays a "warning" on the display device 311. The sound at this time may also be output from the speaker 327. The example in the figure shows an example in which a "warning" 321w is displayed on the image 321 for the first factory. In such a case, the manager in the office can turn on the microphone switch MSW1 and issue voice instructions on the site or instructions on how to deal with the situation via the site speaker 703.

The tertiary control system 300 described above may be provided with the functions of the secondary control system 200 described in FIGS. 1, 2, 3A, and 3B.
Also, the above system has been described on the assumption that a new secondary control system 200 is connected to an existing primary control system 100. However, the present invention is not only applicable to such a case, but also to a case where a new primary control system 100 is connected to an existing secondary control system 200.

As explained above in Figures 4 to 10, the manipulation device 611 that uses the input display signal of the secondary control system 200 has extremely flexible adaptability to the primary control system. This is because it is equipped with a setting screen that allows the input of various pieces of matching information for matching both systems 100 and 200. Also, the RPA scenario can be changed by downloading, by installing from the outside, or even by a memory card. Note that the setting information on the setting screen is not limited to the example given, and it may be designed so that various types of restriction information can be added. For example, the selection of the next scenario may be switched depending on the range of fluctuation of various measurement data from the equipment. In other words, this means that the operation of the equipment may change.

In this sense, the system can of course be applied not only to manufacturing equipment, but also to various moving objects (including flying objects) that are equipped with operation panels.

The representative aspects of the present invention described above will be summarized.
A1: The operation of the equipment to be operated is controlled by a serial signal for equipment control obtained based on a predefined robotic process automation scenario (RPA scenario). Based on this control, the on-site equipment control device 101 of the primary control system 100 controls the equipment 15, grasps the equipment operation status from the response of the equipment 15, displays the equipment operation status on the on-site display panel device 102, and outputs a signal indicating the equipment operation status as a display signal. This is an equipment control system.
The secondary control system 200 has an image recognition unit 512 that recognizes an image of the display signal, and an input display signal utilization manipulation device 611 that transmits the serial signal generated based on the recognition result by the image recognition unit 512 and the RPA scenario 615 to the equipment control device 101. The input display signal utilization manipulation device 611 has a display panel emulation function 614. The display panel emulation function 614 further includes:
Generate a control signal for executing the next sequence specified by the RPA scenario 615 as a panel display signal, and provide the panel display signal to an emulator display panel device 617;
Further, the panel display signal is provided to an operation position encoder 618 so as to encode the panel display signal into the serial signal.

A2: In A1, the serial signal corresponds to an execution log, and this execution log is output to the execution log output unit 622 together with sequence information of the RPA scenario and stored in a storage device.

A3: In A1, the image recognized by the image recognition unit 512 includes OCR processed data, and this data is also treated as an execution log.

B1: In any of A1-A3 above, the input display signal manipulation device 611 has a function of transmitting an audio signal collected by a microphone 702 placed near the facility to a business PC 301 placed remotely via a network 12, and the business PC includes a speaker that outputs the audio signal or an analysis device 351 that analyzes the audio signal.

B2: In B1, the business PC includes a display device 311, and a microphone 328 can be connected to the display device 301, and audio output can be obtained from the microphone 328 via the network to a speaker 723 near the manufacturing-related equipment.

B3: In B1, the business PC includes a display device 311, which is capable of monitoring images displayed by each emulator display panel device in multiple factories and is configured to allow multiple images or a single image to be viewed simultaneously.

In A1 or B1, the facility is a mobile object.

In A1 or B1, the on-site display panel device is either installed after the emulator display panel device is installed, or the on-site display panel device is installed before the emulator display panel device is installed.

　This system also has the following characteristic aspects. That is, the secondary control system is provided with a means for receiving the first operation status image data from an on-site display panel device that issues commands to the on-site equipment control PC starting from the coordinate data of the first operation image. Next, an RPA is provided that identifies a next scenario based on an RPA scenario based on the first operation status image data. Furthermore, an emulator display panel device is provided that displays the first operation image and changes the first operation image to a second operation image based on the next scenario identified by the RPA. And, an encoder is provided that extracts coordinate data of at least the changed portion of the second operation image and transmits it to the on-site equipment control PC.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their variations are within the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. Furthermore, the scope of the present invention includes cases in which each component of the claims is expressed separately, or multiple components are expressed together, or these are expressed in combination. Multiple embodiments may also be combined, and examples consisting of such combinations are also within the scope of the invention.

In addition, in order to make the explanation clearer, the drawings may show the width, thickness, shape, etc. of each part in a schematic manner compared to the actual embodiment. Furthermore, the device of the present invention is applied when the claims are expressed as control logic, when expressed as a program including instructions for causing a computer to execute, and when expressed as a computer-readable recording medium containing the instructions. Furthermore, there is no limitation on the names and terms used, and other expressions that have substantially the same content and intent are also included in the present invention.

100: Primary control system, 101: Equipment control PC, 102: On-site display panel device, 103, 104: Branching device, 105: Display signal capture unit,
200: Secondary control system, 300: Tertiary control system, 301: Business PC, 302: Communication device,
511: USB terminal, 512: image recognition unit, 513: communication device, 514: output terminal, 611: input display signal utilization manipulation device, 621: system control unit, 613: memory device, 614: display panel emulation function, 615: RPA scenario, 616: display panel control unit, 617: emulator display panel device, 618: operation position encoder, 622: execution log output unit, 623: crash log output unit, 624: operation mode setting unit.

Claims (10)

1. The operation of the equipment to be operated is controlled using a serial signal for equipment control obtained based on a predefined robotic process automation scenario (RPA scenario),
   In response to the control, an on-site equipment control device and an on-site display panel device respond, and the equipment control device grasps an equipment operation status from the response of the equipment while controlling the equipment, and outputs a signal indicating the equipment operation status as a display signal.
   an image recognition unit that recognizes an image of the display signal;
   and an input display signal utilizing manipulation device that transmits the serial signal generated based on the recognition result by the image recognition unit and the RPA scenario to the equipment control device,
   The input display signal utilizing manipulation device has a display panel emulation function,
   The display panel emulation function is
   generating a control signal for executing a next sequence specified by the RPA scenario as a panel display signal, and providing the panel display signal to an emulator display panel device;
   The equipment control system is further configured to provide the panel indication signal to an operation position encoder to encode the panel indication signal into the serial signal.
2. The equipment control system of claim 1, wherein the serial signal corresponds to an execution log, and the execution log is output to an execution log output unit together with sequence information of the RPA scenario and stored in a storage device.
3. The equipment control system of claim 1, wherein the image recognized by the image recognition unit includes OCR processed data, and this data is also treated as an execution log.
4. The equipment control system of claim 1, wherein the display panel emulation function displays a setting screen on the emulator display panel device and displays a data input box on the setting screen for adjusting the coordinate offset between the display image of the emulator display panel device and the display image of the other on-site display panel device.
5. The equipment control system according to claim 1, wherein the display panel emulation function displays a setting screen on the emulator display panel device and displays a box on the setting screen for specifying a serial port that is an output terminal for the serial signal.
6. The display panel emulation function displays a setting screen on the emulator display panel device, and provides a box for setting a serial port which is an output terminal of the serial signal on the setting screen;
   2. The equipment control system of claim 1, further comprising at least two of a box for setting a parity bit, a box for setting the speed of the serial signal, a box for setting flow control, a box for setting stop bits, a box for setting the data speed, a box for setting a video capture device, and a box for setting a frame rate.
7. The equipment control system of claim 2 further comprises a communication device that transmits the execution log output by the display panel emulation function to a tertiary control system via a network.
8. an image recognition unit that recognizes an image of a display signal;
   and an input display signal utilizing manipulation device that generates a serial signal for equipment control based on the recognition result by the image recognition unit and a robotic process automation scenario (RPA scenario),
   The input display signal utilizing manipulation device includes a display panel emulation function;
   The display panel emulation function is
   A display panel control unit;
   an emulator display panel device;
   An operating position encoder is provided.
   The display panel control unit
   generating a control signal for executing a next sequence specified by the RPA scenario as a panel display signal, and providing the panel display signal to an emulator display panel device;
   Further, the panel display signal is provided to an operation position encoder to encode the panel display signal into the serial signal;
   The serial signal is transmitted to a control device that controls manufacturing-related equipment.
   Equipment control device.
9. In a facility control method using an image recognition unit and an input display signal utilizing manipulation device,
   the image recognition unit recognizes an image of a display signal;
   The input display signal utilizing manipulation device generates a serial signal for equipment control based on the result of the recognition and a robotic process automation scenario (RPA scenario);
   a display panel emulation function included in the input display signal utilization manipulation device generates a control signal for executing a next sequence specified by the RPA scenario as a panel display signal, and provides the panel display signal to an emulator display panel device;
   Further, the panel display signal is provided to an operation position encoder, and the panel display signal is encoded into the serial signal.
   Transmitting the serial signal to a control device that controls manufacturing-related equipment;
   Equipment control methods.
10. a means for receiving first operation status image data obtained from an on-site equipment control PC when a command based on image coordinate data is given to the on-site equipment control PC from an on-site display panel device or a communication means;
    A robotic process automation (RPA) scenario is identified based on the first operation situation image data.
    an emulator display panel device that displays a first operation status image and changes the first operation status image to a second operation status image based on the next scenario identified by the RPA;
    an encoder that extracts coordinate data of at least the changed portion of the second operation status image and transmits the coordinate data to an equipment control PC at the site.

Images