JP5959278B2 - Supervisory control system - Google Patents

Description

  The present invention relates to a monitoring control system that allows a supervisor to check an operation screen of an operator using a display input integrated device such as a touch panel, and more particularly to a user interface for performing a pointing operation.

  With the sales and development of OSs and displays that support touch operations, in recent monitoring and control systems, plant operation is possible with simple operations by touch. On the other hand, this simple operation has a problem that an erroneous operation is easily caused by an operator's misunderstanding or misrecognition.

  In addition, the supervisor selects one of a plurality of operator screens, and displays the same screen as the operator on the supervisor screen to supervise the operation. However, in the operation by touch, there is a problem that the operation position cannot be specified unless the operator touches the screen, and the supervisor cannot grasp the part that the operator is trying to operate on the monitoring screen.

  On the other hand, as a technique for safely performing a touch operation in the supervisory control system, for example, there are those disclosed in Patent Document 1 and Patent Document 2. In the device disclosed in Patent Document 1, the proximity state of an operator is detected to predict a contact point, and the operability is improved and easily displayed by enlarging or displaying the screen in the vicinity of the screen. In addition, the monitoring and control can be performed safely. Further, in the apparatus disclosed in Patent Document 2, when an operator selects an operation target and utters using a voice input device and a touch panel, it is automatically determined whether or not the voice corresponds to the operation target touched by a finger. -By responding to non-adaptation, supervisory control with improved operational reliability can be performed.

Japanese Patent Laid-Open No. 2001-14024 JP-A-9-114579

  In the apparatus disclosed in Patent Document 1 described above, since the position of a finger that is in proximity before the operator touches is known, erroneous operation can be prevented. However, when such a configuration is applied to a supervisory control system in which the supervisor monitors the operation of the operator, the position of the finger can be recognized. However, if this is a proximity for pointing confirmation, a button to be pressed is searched. There is a problem that the supervisor does not know the operator's intention to operate, such as whether it is close to the user or whether the finger is moved incorrectly instead of trying to operate, and the burden of supervision work increases.

  Further, in the device described in Patent Document 2, automation of pointing and calling is realized by a touch operation to reduce the burden of supervision. However, in this device, a pointing operation that does not touch the screen is not considered. Therefore, there is a problem that the supervisor cannot recognize the pointing operation of the operator.

  The present invention has been made in order to solve the above-described problems, and an object of the present invention is to obtain a monitoring control system that can reduce the supervisory burden on the supervisor and increase the safety of the system.

The supervisory control system according to the present invention is a supervisory control system that uses a display input integrated device that displays an operator screen and can confirm an input operation to the operator screen on a supervisory screen different from the operator screen. From the information on the distance and coordinates between the proximity / touch information input unit that detects the proximity of the operating tool to the operator screen for operation and the operator screen of the operating tool detected by the proximity / touch information input unit A pointing operation determination unit that determines each of an operation, a proximity operation, and a touch operation; an operator screen generation unit that generates an operator screen corresponding to the operation of the operating body based on a determination result of the pointing operation determination unit; Based on the operator screen generated by the screen generation unit, a director screen with information on the determination result in the pointing operation determination unit is generated Is that a that supervision screen generation unit.

According to the monitoring control system of the present invention, a pointing operation determination for determining any one of a pointing operation , a proximity operation , and a touch operation from information on the distance and coordinates of the operating body detected by the proximity / touch information input unit with respect to the operator screen. A supervisor screen with information on the result of determination in the pointing operation determination unit is generated based on the operator screen, thus reducing the supervisory burden on the supervisor and increasing the safety of the system Can do.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the monitoring control system by Embodiment 1 of this invention. It is explanatory drawing which shows the proximity | contact operation | movement of the operator of the monitoring control system by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the monitoring control system by Embodiment 1 of this invention. It is explanatory drawing which shows the example of a production | generation screen by the operator screen production | generation part at the time of proximity | contact and the pointing of the monitoring control system by Embodiment 1 of this invention. It is explanatory drawing which shows the example of a production | generation screen by the supervision screen production | generation part at the time of proximity | contact and the pointing of the monitoring control system by Embodiment 1 of this invention. It is explanatory drawing which shows another display example (screen display example 1) of the operator screen of the monitoring control system by Embodiment 1 of this invention, and a supervisor screen. It is explanatory drawing which shows another display example (screen display example 2) of the operator screen and supervisor screen of the monitoring control system by Embodiment 1 of this invention. It is explanatory drawing which shows another display example (screen display example 3) of the operator screen of the monitoring control system by Embodiment 1 of this invention, and a supervisor screen. It is a block diagram which shows the monitoring control system by Embodiment 2 of this invention. It is a flowchart which shows the operation | movement of the monitoring control system by Embodiment 2 of this invention. It is explanatory drawing which shows the example of the screen selected when the operator A of the monitoring control system by Embodiment 2 of this invention performs pointing operation. It is explanatory drawing which shows the example of the screen selected when the operator A and the operator B of the monitoring control system by Embodiment 2 of this invention perform pointing operation simultaneously. It is a block diagram which shows the monitoring control system by Embodiment 3 of this invention. It is explanatory drawing which shows the example of a screen before the operator of the monitoring control system by Embodiment 3 of this invention performs a pointing operation and performs voice input. It is a flowchart which shows operation | movement of the monitoring control system by Embodiment 3 of this invention. It is explanatory drawing which shows the example of the data structure of the data which the voice recognition object data holding part of the monitoring control system by Embodiment 3 of this invention hold | maintains. It is explanatory drawing which shows the example of the operator screen after the pointing and calling operation of the monitoring control system by Embodiment 3 of this invention. It is explanatory drawing which shows the example of the supervision screen after the pointing and calling operation of the monitoring control system by Embodiment 3 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a monitoring control system according to Embodiment 1 of the present invention.
The monitoring control system shown in FIG. 1 includes a control device 100, an operator screen 201, a proximity / touch information input unit 202, a supervisor screen 301, and a supervisor information input unit 302. The control device 100 is a device that performs monitoring control processing in the monitoring control system, and includes a pointing operation determination unit 101, an operator screen generation unit 102, and a supervisor screen generation unit 103.

The pointing operation determination unit 101 in the control device 100 recognizes the pointing operation from information on the distance and coordinates of the operating body with respect to the operator screen based on the input data from the proximity / touch information input unit 202, and touch / proximity A processing unit that outputs operation information such as an operation state such as pointing and an operation position. The operator screen generation unit 102 is a processing unit that generates an operation screen for the operator 1 to perform an operation input based on the determination result of the pointing operation determination unit 101. The supervision screen generation unit 103 selects a screen corresponding to the selection information of the supervision information input unit 302 from among the operator screens generated by the operator screen generation unit 102, and further, touch / proximity / It is a processing unit that generates a director screen 301 with operation information such as pointing.

The operator screen 201 is a display screen of an integrated display input device called a touch panel that displays an operation screen generated by the operator screen generation unit 102 for the operator 1 to perform an input operation. Proximity touch information input unit 202, a proximity state of the operation tool such as a finger or a pen operator 1 and test knowledge at a given sensor is the information input unit for sending the detected information to the pointing operation determination unit 101 . Note that a plurality of display input integrated devices are provided corresponding to a plurality of operators 1.

  The director screen 301 is a display screen for displaying a screen generated by the director screen generation unit 103, and is a screen displayed on a terminal for a supervisor different from the terminal for each operator 1. The supervisor information input unit 302 is an input unit for inputting supervisor information such as the supervisor 2 selecting an operator screen 201 to be displayed on the supervisor screen 301 among the plurality of operator screens 201.

FIG. 2 is an example in which an operating tool is brought close to the operator screen 201. The operation objects 21 and 22 are buttons for performing a selection operation on the operator screen 201 and indicate the finger 23 of the operator 1 as an operation body. At this time, the proximity-touch information input unit 202, you search known proximity of the operator's finger 23.

FIG. 3 is a flowchart illustrating a process in which the pointing operation determination unit 101 determines a pointing operation.
The pointing operation determination unit 101 first determines whether the sensor output value of the finger 23 detected by the proximity / touch information input unit 202 is larger than a first predetermined value (step ST101). If the sensor output value is smaller than the first predetermined value, the sensor output value of the finger 23 is stored (step ST102). The pointing operation determination unit 101 repeats step ST101 and step ST102 until the sensor output value of the finger 23 becomes larger than the first predetermined value in step ST101. That is, steps ST101 to ST102 are performed until the distance between the finger 23 and the operator screen 201 approaches a predetermined value. If the sensor output value is larger than the first predetermined value in step ST101, the process proceeds to the determination process in step ST103. The sensor output value may be any value as long as it indicates a value corresponding to the distance between the operating tool and the screen. For example, the result of analyzing finger image data or actual physical data It may be a value obtained by measuring a typical distance.

  In step ST103, it is determined whether the sensor output value of the finger 23 is smaller than a second predetermined value. Here, the second predetermined value> the first predetermined value. When the sensor output value is larger than the second predetermined value, that is, when the distance between the finger 23 and the operator screen 201 is quite close, the pointing operation determination unit 101 causes the finger 23 to contact the operator screen 201. In step ST104, the storage of the sensor output value of the finger 23 is cleared. In step ST105, the contact coordinates of the finger 23 are obtained, and the determination result of contact and the contact coordinates of the finger 23 are returned. On the other hand, if the sensor output value is smaller than the second predetermined value in step ST103, the approaching speed of the finger 23 is obtained from the stored sensor output value of the finger 23 (step ST106), and the proximity coordinates of the finger 23 are obtained (step ST106). ST107).

  Next, the pointing operation determination unit 101 determines whether the approaching speed of the finger 23 is equal to or higher than a certain level (step ST108). When the approaching speed of the finger 23 is less than a certain value, the memory of the sensor output value of the finger 23 is cleared, and the determination result of proximity and the proximity coordinates of the finger are returned (step ST109). On the other hand, when the speed at which the finger 23 approaches is greater than or equal to a certain value in step ST108, the process proceeds to the determination process in step ST110.

  In step ST110, the pointing operation determination unit 101 determines whether the proximity coordinates of the finger 23 have moved beyond a certain level. In step ST110, if the proximity coordinates of the finger 23 are moving beyond a certain level, the process proceeds to step ST109, the memory of the sensor output value of the finger 23 is cleared, and the determination result of proximity and the proximity coordinates of the finger are returned. If the proximity coordinates of the finger 23 have not moved beyond a certain level, the determination process of step ST111 is performed.

  In step ST111, the pointing operation determination unit 101 determines whether or not the proximity coordinate of the finger 23 has not moved for a certain period of time. When it does not continue for a certain period of time, the pointing operation determination unit 101 returns the determination result of proximity and the proximity coordinates of the finger. On the other hand, when the proximity coordinate of the finger 23 has not moved for a certain period of time, the finger pointing determination result and the proximity coordinate are returned.

  For the finger approach speed Vz in step ST106, for example, the stored sensor output value is divided in half, and the average value Zb of the first half output and the average value Za of the second half output are obtained, and Vz = (Zb−Za) / You may obtain | require with calculation formulas, such as the number of memory | storage samples.

  The determination as to whether or not the coordinates of the finger 23 have moved beyond a certain level in step ST110 may be based on whether the similarity with the previous sensor output value is equal to or greater than a method that uses close coordinates.

  FIG. 4 is an example of a screen generated by the operator screen generation unit 102 after the pointing operation determination unit 101 determines proximity / pointing / contact. A circle 40 is an area that is displayed semi-transparently with a predetermined color around the proximity position, and a finger 42 indicates an operator's finger that is determined to be a proximity operation. A rectangle 41 is a region that is semitransparently painted in a predetermined color on the operation target closest to the proximity position, and a finger 43 indicates an operator's finger determined to be a pointing operation.

  The operator screen generation unit 102 may be configured to display a semi-transparent circle with a predetermined color when the finger is in proximity, and to display a semi-transparent circle with a different color when pointing with the finger.

  FIG. 5 shows a director screen corresponding to the operator screen of FIG. That is, an example of a screen generated by the supervisor screen generation unit 103 when the supervisor selects the operator screen of FIG. 2 in the supervisor information input unit 302 and the pointing operation determination unit 101 determines proximity / pointing / contact. is there. The circle 50 at the time of proximity is an area that is displayed semi-transparently with a predetermined color around the proximity position, and the rectangle 51 at the time of pointing is semi-transparent with a predetermined color on the operation target closest to the proximity position. This is the area that was created.

  FIG. 6 is another display example (screen display example 1) of the screens generated by the operator screen generation unit 102 and the director screen generation unit 103, with the operator screen on the left side and the director screen on the right side. In this example, the distance between the touch panel and the finger is determined based on the magnitude of the sensor output value of the finger, and the range and color of the circle at the pointing position of the circle 40 in FIG. 4 or the circle 50 in FIG. 5 according to the finger distance. Fluctuate. A circle 61 is a circle displayed on the operator screen when the touch panel and the finger are at a distance greater than a certain distance. A circle 62 is a circle displayed on the operator screen when the touch panel and the finger are close to each other at a certain distance. A circle 63 is a circle displayed on the director screen when the touch panel and the finger are at a distance greater than a certain distance. A circle 64 is a circle displayed on the director screen when the touch panel and the finger are close to each other at a certain distance.

  FIG. 7 is another display example (screen display example 2) of the screens generated by the operator screen generation unit 102 and the director screen generation unit 103. In this example, the left side shows the operator screen and the right side shows the director screen. Yes. In this example, the distance between the touch panel and the finger is determined based on the magnitude of the sensor output value of the finger, and the circle range of the pointing position of the circle 40 in FIG. 4 or the circle 50 in FIG. 5 varies according to the finger distance. And change the transparency and color of the center and circumference of the circle. The circles 71 and 72 are circles displayed on the operator screen and the director screen when the touch panel and the finger are at a certain distance or more away, respectively.

  FIG. 8 is another display example (screen display example 3) of the screens generated by the operator screen generation unit 102 and the director screen generation unit 103. In this example, the distance between the touch panel and the finger is determined based on the magnitude of the sensor output value of the finger, and the selection target range whose color is changed according to the finger distance is changed. An operation target closer to the proximity position of the finger is highlighted. The operation target 81 is the operation target closest to the finger proximity position on the operator screen, and the operation target 82 is the operation target second closest to the finger proximity position on the operator screen. The operation target 83 is the operation target closest to the finger proximity position on the director screen, and the operation target 84 is the second operation target closest to the finger proximity position on the director screen.

As described above, in the first embodiment, as shown by the circle 40 and the rectangle 41 in FIG. 4, the operator can change the operation state and the operation position by performing different displays according to the proximity of the finger or the pointing of the finger. It can be easily recognized.
In addition, as shown by a circle 50 or a rectangle 51 in FIG. 5, the supervisor can know the operation state and the operation position of the operator by performing separate display on the director screen according to the proximity of the finger or the pointing of the finger. It is possible to easily find operation mistakes.

  Further, as shown in FIGS. 6 to 8, by changing the display range of the circle and the operation target range to be highlighted according to the distance between the finger and the touch panel, the operator's finger It is possible to prevent an event in which the insertion operation target and the detection coordinates of the proximity / touch information input unit 202 are shifted and the supervisor makes a confirmation error.

As described above, according to the monitoring control system of the first embodiment, the input operation to the operator screen can be confirmed on the supervisor screen different from the operator screen by using the display input integrated device that displays the operator screen. a monitoring control system, a proximity touch information input unit for detecting the proximity state to the operator screen of the operation member for performing an input operation, the proximity-touch information input unit with the operator screen of the operating body detected An operator screen corresponding to the operation of the operation object based on the determination result of the pointing operation determination unit and the pointing operation determination unit that determines any of the pointing operation , the proximity operation , and the touch operation from the information regarding the distance and the coordinates The operator screen generation unit that generates the information and the information on the determination result in the pointing operation determination unit based on the operator screen generated by the operator screen generation unit Since a supervisory screen generation unit for generating a given the supervision screen, reduce supervise loading supervisor, can improve the safety of the system.

  Further, according to the monitoring control system of the first embodiment, the pointing operation determination unit calculates the proximity distance between the operating body and the operator screen, and the operator screen generation unit calculates the proximity distance calculated by the pointing operation determination unit. In response to the change of the display indicating the input operation on the operator screen, the supervisor screen generator changes the display of the supervisor screen in response to the change of the display on the operator screen generator. Confirmation can be performed more reliably, and confirmation errors by the supervisor can be prevented in advance.

Embodiment 2. FIG.
The monitoring control system according to the second embodiment is configured to select an operator screen for generating a supervisor screen based on the operator's pointing designation and the importance of the operation target, and the configuration is shown in FIG. .
9, the control device 100a includes a pointing operation determination unit 101, an operator screen generation unit 102, a supervisor screen generation unit 103a, and a supervisor display screen selection unit 104. Further, as a configuration on the operator 1 side, in addition to the operator screen 201 and the proximity / touch information input unit 202, a voice information input unit 203 is provided. Furthermore, as a configuration on the supervisor 2 side, a supervisor screen 301 and a supervisor information input unit 302 are provided.

  The voice information input unit 203 detects the voice input of the operator 1 and outputs it to the supervisor display screen selection unit 104 including the presence or absence of voice input. The supervisor display screen selection unit 104 is generated by the operator screen generation unit 102 based on the presence / absence and contents of voice input from the voice information input unit 203 and operation information such as a touch state from the pointing operation determination unit 101. Of the operator screens 201 for the respective operators 1, which operator screen is to be used by the supervisor screen generation unit 103a is selected. The supervisor screen generation unit 103a generates a supervisor screen 301 based on the operator screen 201 when the operator information input unit 302 selects one of the operator screens 201, and the supervisor display screen when there is no selection operation. The director screen 301 is generated based on the operator screen 201 selected by the selection unit 104. Since other configurations are the same as those of the first embodiment, the corresponding parts are denoted by the same reference numerals and description thereof is omitted.

  FIG. 10 is a flowchart showing a process in which the supervisor display screen selection unit 104 selects a screen to be displayed to the supervisor in a monitoring control system that can be operated simultaneously by a plurality of operators.

  The supervisor display screen selection unit 104 first determines whether a plurality of operators are pointing at the same time based on the determination result of the pointing operation determination unit 101 (step ST201). When the pointing operation is not performed at the same time, the supervisor display screen selection unit 104 selects the screen of the pointing operator. When pointing at the same time, the determination process of step ST202 is performed.

  In step ST202, it is determined whether or not voice input such as pointing and calling is performed from the voice information input unit 203 to each operator. When there is a voice input from any of the operators, the supervisor display screen selection unit 104 selects the screen of the operator who has input the voice at the earliest timing. When there is no voice input from the operator, the supervisor display screen selection unit 104 extracts the importance of the operation target being pointed at by each operator in step ST203.

  When the importance of the operation target is extracted in step ST203, the supervisor display screen selection unit 104 determines whether the importance of the pointing operation target is the same (step ST204). When the importance of the pointing target is not the same, the supervisor display screen selection unit 104 selects the operator screen having the highest importance. When the importance of the pointing target is the same, the supervisor display screen selection unit 104 selects the screen of the operator who performed the pointing operation first.

FIG. 11 is an example of an operator operation screen selected by the supervisor display screen selection unit 104 when the operator A performs a pointing operation and at the same time the operator B performs a proximity operation.
111 on the operator A screen indicates the pointing operation of the operator A, and 112 on the operator B screen is the proximity operation of the operator B. Further, reference numeral 113 on the supervisor screen is an operation target at a position closest to the pointing operation position of the operator A. Since the operator A performs a pointing operation, the supervisor display screen selection unit 104 selects the screen of the operator A by the determination in step ST201. The supervisor screen generation unit 103a highlights the closest operation target 113 using the touch state and position coordinate information of the pointing operation determination unit 101. The highlighted display is a blue display that is a color assigned to the operator A.

  FIG. 12 is an example of an operator operation screen selected by the supervisor display screen selection unit 104 when the operator A and the operator B perform a pointing operation simultaneously without performing voice input. Reference numeral 121 denotes an operator A pointing operation, and reference numeral 124 denotes an operator A pointing operation target. Reference numeral 122 denotes an operator B pointing operation, and reference numeral 125 denotes an operator B pointing operation target. Further, reference numeral 123 denotes an operation target at a position closest to the operator B's pointing operation position. Since operator A and operator B are pointing at the same time without inputting voice, in step ST203, the supervisor display screen selection unit 104 extracts the importance assigned to the operation objects 124 and 125 in advance. If the importance assigned to 124 is 3 and the importance assigned to 125 is 5, as a result of the determination in step ST204, the supervisor display screen selection unit 104 points to an operation object having a high importance. Select the screen of operator B who is operating. The director screen generation unit 103a highlights the closest operation target 123 using the touch state and position coordinate information determined by the pointing operation determination unit 101. This highlighted display is a red display which is a color assigned to the operator B.

As described above, according to the monitoring control system of the second embodiment, a plurality of display input integrated devices each displaying an operator screen are used, and an input operation to each operator screen is defined as an operator screen. a possible monitoring and control systems found in different supervision screen, a proximity touch information input unit for detecting the proximity state to the operator screen of the operation member for performing an input operation, respectively input to a plurality of operators screens From the information on the distance and coordinates between the voice information input unit that inputs the voice of the operator performing the operation and the operator screen of the operating body detected by the proximity / touch information input unit, any of the pointing operation , the proximity operation , and the touch operation And each operation corresponding to the operation of the operating body based on the determination result of the pointing operation determination unit. An operator screen generation unit that generates a screen, a director screen generation unit that generates a director screen to which information on the determination result in the pointing operation determination unit is added based on a plurality of operator screens generated by the operator screen generation unit, and voice Based on the voice information input from the information input unit, the operator's pointing designation is recognized, the order of operation to the plurality of operator screens in the pointing designation and pointing operation, and the importance of the operation target of the pointing operation A supervisor display screen selection unit that selects an operator screen to be used by the supervisor screen generation unit based on at least one of them, so that the supervisor does not explicitly select one of the operator screens. Operation can be monitored automatically. As a result, the burden on the supervisor can be reduced, and the operation target requiring the pointing operation can be confirmed more reliably.

Embodiment 3 FIG.
The monitoring control system according to the third embodiment is configured to display whether or not the pointing operation matches the pointing designation, and the configuration is shown in FIG.
13, the control device 100b includes a pointing operation determination unit 101, an operator screen generation unit 102a, a supervisor screen generation unit 103b, a voice recognition target data extraction unit 105, a voice recognition unit 106, and a voice recognition target data holding unit 107. ing. The configuration on the operator 1 side and the configuration on the supervisor 2 side are the same as those in the second embodiment. However, the speech information input by the speech information input unit 203 is configured to be input to the speech recognition unit 106 of the control device 100b. Further, the director information input unit 302 is configured to be able to include information input such as selecting operation information for pointing and calling in the voice recognition unit 106 or inputting confirmation completion for the result displayed on the director screen 301. Has been.

  The voice recognition target data extraction unit 105 receives the pointing operation information determined by the pointing operation determination unit 101, and starts from the pointing position based on the position information of the operation target held in the voice recognition target data holding unit 107. It is a processing unit that selects a speech recognition target within a certain distance and extracts data for collating speech recognition results. The voice recognition unit 106 receives the operator's voice input information from the voice information input unit 203 and the voice recognition collation data extracted by the voice recognition target data extraction unit 105, and the voice recognition result of the voice input information is the voice recognition collation. It is a processing part which determines whether it corresponds with the data for use. The voice recognition target data holding unit 107 has voice input information to be operated by the operator and position information on the screen. The operator screen generation unit 102a receives operation information such as touch / proximity / pointing from the pointing operation determination unit 101 and a voice recognition determination result from the voice recognition unit 106, and generates an operator screen corresponding to the operation of each operator. It is configured as follows. Further, the director screen generation unit 103b generates information on the determination result in the pointing operation determination unit 101 and operation information on the operation of pointing and calling in the voice recognition unit 106 based on the operator screen generated by the operator screen generation unit 102a. It is configured to generate the assigned director screen. Other configurations are the same as those of the first embodiment.

  FIG. 14 shows an example of a screen before the operator performs a pointing operation and performs voice input. Reference numeral 140 denotes an operator's pointing operation, and reference numeral 141 denotes an operator's pointing operation target. Reference numeral 142 denotes a voice name input message corresponding to the operation object 141. The pointing operation can be performed by performing a pointing operation on the operation target 141 and calling the message 142 “change hydraulic pressure parameter”. If the pointing and calling is successful, the pointing and calling is successful, and the operation object 141 can be selected. If pointing and calling fails, pointing and calling fails, and the operation target 141 cannot be selected.

  FIG. 15 is a flowchart showing the pointing and calling operation. In step ST301, the pointing operation determination unit 101 determines the pointing operation by the processing of the flow illustrated in FIG. Next, the speech recognition target data extraction unit 105 determines whether the pointing operation state is set (step ST302). If not in the pointing operation state, the speech recognition target data extraction unit 105 repeats step ST301 and step ST302. In step ST302, when it is a pointing operation state, the determination process of step ST303 is performed.

  In step ST303, the speech recognition target data extraction unit 105 searches the speech recognition target data holding unit 107 for an operation target for pointing and calling within a certain range centered on the pointing position. If there is no operation target as a result of the search, the speech recognition target data extraction unit 105 returns to the process of determining the pointing operation in step ST301. In step ST303, when there is an operation target, the operator screen generation unit 102a displays an input message called an operation target for pointing and calling on the operator screen (step ST304). In addition, the director screen generation unit 103b displays an operation target of pointing and calling on the director screen (step ST305).

  Further, in step ST303, the speech recognition target data extraction unit 105 searches for an operation target of pointing and calling within a certain range, but this search range may be changed according to the distance between the finger and the touch panel. For example, when the finger is away from the touch panel by a predetermined distance or more, the search range can be expanded to increase the number of voice recognition targets.

  Next, voice recognition section 106 determines whether there is voice input information from voice information input section 203 (step ST306). When there is no voice input information, the voice recognition target data extraction unit 105 returns to the process of determining whether or not the pointing operation state is in step ST302. If there is voice input information in step ST306, the voice recognition unit 106 performs voice recognition using the word data and voice input data of the voice recognition target data holding unit 107 (step ST307).

  Next, the voice recognition unit 106 determines whether the data of the voice recognition target data holding unit 107 corresponding to the pointing position matches the voice recognition result (step ST308). When the pointing and calling fails, the voice recognition unit 106 prohibits the operation on the operation target having been pointed and called until the pointing and calling succeeds again. When the pointing and calling is successful, the voice recognition unit 106 permits the operation of the operation target having the pointing and calling.

  FIG. 16 shows an example of the data structure of data held in the speech recognition target data holding unit 107. The screen ID 160 is an ID assigned to each screen (operator screen) displayed to the operator. The center coordinate 161 is the center coordinate of the operation area on the screen. The operation area 162 is information on the operation area on the screen. The input message 163 is a message to be displayed on the operator screen when the operator tries to operate. The reading message 164 is word data referred to when the voice recognition unit 106 recognizes the voice. Further, the input message 163 and the reading message 164 are also used for collating the speech recognition result. In such a data format, in the flow of FIG. 15, the search for the operation target of pointing and calling in step ST303, the display of the input message in step ST304, and the word data in the speech recognition target data holding unit 107 in step ST307 are used. Voice recognition can be performed.

  FIG. 17 shows an example of the operator's operation screen after the pointing and calling operation. Reference numerals 170 and 171 are operator pointing operations, and reference numerals 172 and 174 are operator pointing operation targets. Reference numeral 173 denotes a message displayed when the operator succeeds in pointing and calling operation. On the other hand, 175 is an icon displayed on the operation target 174 when the operator fails to point and call, and this icon 175 disappears when the pointing operation is no longer performed. Furthermore, reference numeral 176 denotes a message displayed when the operator fails the pointing and calling operation.

  FIG. 18 shows an example of the director screen after the pointing and calling operation. Reference numerals 180 and 181 are the pointing operation targets of the operator. A message 182 is displayed on the director screen when the operator succeeds in pointing and calling operation. A message 183 is displayed on the director screen when the operator fails to point and call. An icon 184 is displayed on the operation target 181 when the operator fails to point and call.

  In FIG. 18, for example, when confirmation is input from the supervisor to the supervisor information input unit 302, the voice recognition unit 106 may be configured to delete the messages 182 and 183 and the icon 181.

  As described above, in the third embodiment, the operator 1 passes the confirmation of the supervisor 2 by displaying the display when the pointing and calling operation is successful and when the operation is unsuccessful on the operator side as shown in FIG. It is possible to recognize operation mistakes without any problems. Also, by displaying the successful and unsuccessful pointing and calling operations on the director screen 301 as shown in FIG. 18, it is only necessary to reconfirm that there are no problems with the operations, thereby reducing the burden. can do. Furthermore, since an operation that requires an operator's pointing and calling is automatically checked, it is possible to reduce human error such as an operation error or a supervisor error.

As described above, according to the monitoring control system of the third embodiment, the input operation to the operator screen can be confirmed on the supervisor screen different from the operator screen using the display input integrated device that displays the operator screen. a monitoring control system, a proximity touch information input unit for detecting the proximity state to the operator screen of the operation member for performing an input operation, voice inputting voice of the operator to perform input operation to the operator screen A pointing operation determination unit that determines one of a pointing operation , a proximity operation , and a touch operation from information on the distance and coordinates between the information input unit and the operator screen of the operating body detected by the proximity / touch information input unit; Holds audio information corresponding to the operation target that is the input operation on the operator screen and position information on the operator screen of the operation target Voice recognition target data holding unit and voice recognition for extracting operation target voice information corresponding to the pointing operation position from the voice recognition target data holding unit based on the pointing operation information determined by the pointing operation determination unit A target data extraction unit, a voice recognition unit that compares the voice information input from the voice information input unit with the voice information extracted by the voice recognition target data extraction unit, and determines whether or not they match; An operator screen that generates an operator screen corresponding to the operation of the operating tool based on the determination result of the insertion operation determination unit and generates a screen to which the determination result of the voice recognition unit is added when the operation is a pointing Based on the generation unit and the operator screen generated by the operator screen generation unit, a director screen is generated with information on the determination result in the pointing operation determination unit and the determination result in the voice recognition unit. Since a supervisory screen generation unit, it is possible to reduce the artificial erroneous operation such as an operation mistake or supervisor supervision operator error.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Operator, 2 Supervisor, 100, 100a, 100b Control apparatus, 101 Pointing operation determination part, 102, 102a Operator screen generation part, 103, 103a, 103b Supervisor screen generation part, 104 Supervisor display screen selection part, 105 Voice Recognition target data extraction unit, 106 voice recognition unit, 107 voice recognition target data holding unit, 201 operator screen, 202 proximity / touch information input unit, 203 voice information input unit, 301 director screen, 302 director information input unit.

Claims (4)

Using a display input integrated device that displays an operator screen, a monitoring control system capable of confirming an input operation to the operator screen on a supervisor screen different from the operator screen,
Proximity / touch information input unit for detecting the proximity state of the operating body for performing the input operation to the operator screen;
A pointing operation determination unit that determines each of a pointing operation, a proximity operation, and a touch operation from information on a distance and coordinates of the operation body detected by the proximity / touch information input unit;
Based on the determination result of the pointing operation determination unit, an operator screen generation unit that generates the operator screen corresponding to the operation of the operation body;
A supervisory control system comprising: a supervisory screen generation unit that generates the supervisory screen to which information on a determination result in the pointing operation determination unit is added based on the operator screen generated by the operator screen generation unit. The pointing operation determination unit calculates a proximity distance between the operation body and an operator screen,
The operator screen generation unit changes a display indicating an input operation on the operator screen according to the proximity distance calculated by the pointing operation determination unit,
The monitoring control system according to claim 1, wherein the supervision screen generation unit changes the display of the supervision screen in response to a change in display in the operator screen generation unit. A monitoring control system that uses a plurality of display input integrated devices each displaying an operator screen, and allows the input operation to each operator screen to be confirmed on a supervisor screen different from these operator screens,
Proximity / touch information input unit for detecting the proximity state of the operating body for performing the input operation to the operator screen;
A voice information input unit for inputting a voice of an operator who performs an input operation on each of the plurality of operator screens;
A pointing operation determination unit that determines each of a pointing operation, a proximity operation, and a touch operation from information on a distance and coordinates of the operation body detected by the proximity / touch information input unit;
Based on the determination result of the pointing operation determination unit, an operator screen generation unit that generates the respective operator screens corresponding to the operation of the operation body;
Based on the plurality of operator screens generated by the operator screen generation unit, a director screen generation unit that generates the director screen to which information on a determination result in the pointing operation determination unit is provided;
Based on the voice information input from the voice information input unit, the operator's pointing designation is recognized, the order of operation on the plurality of operator screens in the pointing designation, the pointing operation, and the operation of the pointing operation. A supervisory control system comprising: a supervisor display screen selection unit that selects an operator screen used by the supervisor screen generation unit based on at least one of the importance levels of the object. Using a display input integrated device that displays an operator screen, a monitoring control system capable of confirming an input operation to the operator screen on a supervisor screen different from the operator screen,
Proximity / touch information input unit for detecting the proximity state of the operating body for performing the input operation to the operator screen;
A voice information input unit for inputting a voice of an operator who performs an input operation on the operator screen;
A pointing operation determination unit that determines each of a pointing operation, a proximity operation, and a touch operation from information on a distance and coordinates of the operation body detected by the proximity / touch information input unit;
A voice recognition target data holding unit that holds voice information corresponding to an operation target that is an input operation on the operator screen, and position information on the operator screen of the operation target;
Based on the pointing operation information determined by the pointing operation determination unit, a voice recognition target data extraction unit that extracts voice information of an operation target corresponding to the pointing operation position from the voice recognition target data holding unit;
A speech recognition unit that collates speech information input from the speech information input unit with speech information extracted by the speech recognition target data extraction unit and determines whether or not they match;
Based on the determination result of the pointing operation determination unit, the operator screen corresponding to the operation of the operating body is generated, and when the operation is a pointing, the determination result of the voice recognition unit is added. An operator screen generator for generating a screen;
Based on the operator screen generated by the operator screen generation unit, a director screen generation unit that generates the director screen to which the information on the determination result in the pointing operation determination unit and the determination result of the voice recognition unit are added. Monitoring control system.

Images