JP3277658B2 - Information display device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a relates to information display device,
In particular to suitable information display device to accommodate the system with a large-screen display device and a small screen display.

[0002]

2. Description of the Related Art In recent years, in operation control of power generation plants and chemical plants, provision of information at financial institutions and securities companies, and presentations at conferences and the like, display devices having a large display surface (including mimic display devices) are used. The information provision used is planned or implemented. It is desired that information to be displayed on a large screen display device can be remotely instructed or information can be given to an instructed display target.

[0003] Techniques that can meet this demand include, for example, a pointing device and an input method disclosed in JP-A-3-179517, and a presentation device disclosed in JP-A-4-37922.

In Japanese Patent Application Laid-Open No. 3-179517, a single infrared projector remotely indicates a display surface of a large-screen display device, and a designated point is determined based on an output signal of a monitor camera monitoring the display surface. Is displayed on the large-screen display device, and a marker indicating that the designated point is designated is displayed. Furthermore, it also shows that an instruction to be input is processed using a separately prepared instruction processing means for the indicated display target.

[0005] In Japanese Patent Application Laid-Open No. Hei 4-37922, when a required portion of a screen projected on a screen or a wall surface is pointed by a pointing rod having a light-emitting device, an output signal of a television camera monitoring the screen is displayed. Based on the detected point,
It is shown that the image information is changed.

[0006]

In each of the above-mentioned conventional examples, there is only one pointer (infrared light projector, pointing rod), and when a plurality of pointers are used, which pointer is It is not possible to identify whether the part is indicated. For this reason, it is not possible to execute an instruction for an indicated point or an indicated object by using a plurality of pointers and distinguishing them.

An object of the present invention is to provide information on a display object designated for each pointer even when a plurality of pointers are used.
An object of the present invention is to provide an information display device capable of providing information easily and reliably .

[0008]

[0009]

An object of the present invention is to provide a display device.
Shooting the display device, a display device, and the display device
A plurality of points that have reached the display surface of the display device.
Shooting means for outputting a video signal including an instruction signal from a printer
The plurality of pointers based on the video signal.
Identification means for identifying each of them, and a display on a display surface of the display device.
From each of the multiple displayed objects,
Object identification means for identifying the indicated display object, and each finger
Display information about the display object specified by the
Each display device corresponding to each identified pointer
And output information output means.
It is. In addition, the above object is to provide a different indication signal from the display device.
Respectively, and displays the display surface of the display device with the instruction signal.
A plurality of pointers indicated by symbols and the display device.
The plurality of points which are shadowed and reach the display surface of the display device.
Shooting means for outputting a video signal including an instruction signal from a printer
And moving the plurality of pointers based on the video signal.
Identification means for identifying each, and display on the display surface of the display device
From multiple displayed objects with each instruction signal
Object specifying means for specifying the displayed display object and each instruction
The display information on the display object indicated by the signal,
Display on the display device corresponding to each identified pointer.
And output means for outputting display information.
You. The above object is achieved by a first display device and the first display device.
A second having a second display surface larger than the first display surface of the device;
A display device and an image of the second display device, the second display
Instruction signals from multiple pointers that reach the display surface of the device
Photographing means for outputting a video signal including:
Identification means for identifying each of the plurality of pointers based on the
A column and a plurality of display objects displayed on the second display surface
Identify the display target indicated by each instruction signal from the objects.
Object specifying means to be displayed and the display object indicated by each instruction signal
Display information on the object is associated with each of the identified pointers.
Display information output means for outputting to corresponding display devices, respectively;
This is achieved by providing: In addition, the purpose
Instruction signals from multiple pointers that have reached the display surface of the display
The plurality of pointers based on a video signal including
Identification means for identifying each of them, and a display on a display surface of the display device.
From each of the multiple displayed objects,
Object identification means for identifying the indicated display object, and each finger
Display information about the display object specified by the
A separate display device corresponding to each identified pointer
And display information output means for outputting the respective information.
Achieved.

[0010]

[0011]

Identifies the pointer outputs the instruction signal on the basis of an instruction signal reaching the display surface of the action] Display device, indicated by the command signal, the display object displayed on the display surface, identifying Therefore, a plurality of pointers can be easily identified, and the display target pointed to by each pointer can be specified with high accuracy. Therefore, even when a plurality of pointers are used, it is possible to easily and reliably provide information on the display target object designated for each pointer.

Further , since the display device on which information is to be displayed corresponding to the pointer is specified, it is possible to display the information on the display device corresponding to the pointer based on the identification information of the pointer, and to obtain the information. Is improved.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An operation / monitoring device according to a preferred embodiment of the present invention will be described with reference to FIGS.

This embodiment includes a monitor camera 1, an image processing device 2, an electronic computer 7, a large screen display device 4, CRT display devices 6A to 6G, and pointers Pa and Pb. C
The RT display devices 6A to 6G are provided on the operation monitoring panel 3. The operation monitoring panel 3 has an operation and monitoring function necessary for the operator A (generally a plurality) to monitor the operation of the plant (not shown). Operating information necessary for each of the operators A is displayed on the CRT display devices 6A to 6G. The large screen display device 4 is disposed behind the operation monitoring panel 3 and has a CR
It has a display surface larger than the display screens of the T display devices 6A to 6G.

The computer 7 has a coordinate recognition unit 6 shown in FIG.
0, an object specifying unit 61, a display information determining unit 62, and a display information creating unit 63 are executed.

With such a configuration, the operator A can grasp the display information of the CRT display devices 6A to 6G provided on the operation monitoring panel 3 and can display the large screen display device 4
, The information on the entire plant can be grasped. Further, a person B (hereinafter, referred to as a manager, not shown) in a position to manage the entire plant can grasp the information displayed on the large-screen display device 4. For this reason, the information grasp is improved, that is, the monitoring of the plant is enhanced.

The operator A uses the pointer Pa and the manager B
Uses the pointer Pb. By remote control from the pointer Pb, the information displayed on the large screen display device 4 is changed, and the detailed information on a part of the information displayed on the large screen display device 4 is displayed on the large screen display device 4. Partial display is performed. Further, by remote control from the pointer Pa, the detailed information of the information displayed on the large screen display device 4 can be displayed on any of the CRT display devices 6A to 6G and the portable display device HD. This can be achieved by associating each pointer with various display devices 4, 6A to 6G, and HD.

Which part of the large screen display device 4 is a pointer P
A monitor camera 1 and an image processing device 2 are provided to monitor whether or not an instruction is given by a and Pb. Further, in order to identify which pointer points to the large-screen display device 4, the instruction signals output from the pointers Pa and Pb, that is, the emitted lights have different shapes. In FIG. 1, the pointer Pa emits dot-shaped light as an instruction signal, and the pointer Pb
Shows an example in which a cross-shaped light is emitted as an instruction signal. The display screen of the large screen display device 4 includes a pointer P
Markers 5a and 5b corresponding to the shape of the light emitted to the position indicated by a and Pb are displayed as described later. Therefore, the operator A and the manager B can check the position indicated by the corresponding pointer by looking at the display screen of the large screen display device 4. The light emitted from the pointers Pa and Pb is an infrared laser.

First, when the manager B emits a cross-shaped infrared laser beam, which is an instruction signal, from the pointer Pb and points on the display screen of the large screen display device 4, the operation / monitoring device of the present embodiment will be described. The operation will be described below.

The operation / monitoring device of this embodiment regards the turning off of the infrared laser light output from the pointer as the input of the instruction signal. The monitor camera (infrared camera) 1 captures a display screen of the large screen display device 4. The image signal of the infrared light output from the monitor camera 1 includes the image of the cross-shaped infrared laser light, which is the instruction signal output from the pointer Pb, on the large screen display device 4, and the image processing device 2
Is input to Here, the reason for using the infrared laser is that (1) the coordinates of the pointer indicated by the pointer, which are recognized by the image processing device 2 and the computer 7, are different from the actual coordinates of the specified point due to a recognition error. This is to respond. In other words, the point recognized by the operation / monitoring device is indicated by the cursor, thereby improving the input accuracy of the instruction signal. Also, (2) pointers Pa, which output infrared light,
If Pb and the monitor camera 1 that is an infrared camera are used, the image captured by the image processing device 2 does not include visible light, and the processing in the image processing device 2 can be performed easily and at high speed.

The image processing device 2 executes the processing of FIG. An image signal of infrared light is input (process 2a). A binarization process is performed on this video signal (process 2b).
By performing binarization, noise included in the video signal is removed. Next, the feature amount of the video signal is extracted using the binary image (process 2c). In this processing, x of the binary image
The projection distribution on the axis and the y-axis is taken, and the coordinates of the point where the density of the binary image exceeds the reference value L are obtained. Projection distribution shown in FIG. 4 (d) for the pointer Pb is obtained, the coordinates of a point exceeding the reference point L is x _1, x _2, y ₁ and y _2. These coordinates x ₁ , x ₂ , y ₁ and y ₂ indicate the features of the binary image. In process 2d, the position indicated by the pointer on the screen of the large screen display device 4 is determined. That is, the coordinates x ₁ , x ₂ ,
Using y ₁ and y ₂ , the coordinates (X,
Y) (X = (x ₁ + x ₂ ) / 2, Y = (y ₁ + y ₂ ) /
2). The coordinates (X, Y) of this center position are the pointing position of the pointer. Next, the type of the pointer is determined (Process 2
e). The type of the pointer is determined by determining the area △ S (= (x ₂ −x ₁ ) (y ₂ −) using the coordinates x ₁ , x ₂ , y ₁ and y _2.
The judgment is made based on the magnitude of y ₁ )). 4 (d)
Pointer P For S ₄ for outputting a cross instruction signal
b is determined. Outputs information on the designated position of the pointer and the type of the pointer to the computer 7 (process 2).
f).

The input information of the pointed position and the pointer of the aforementioned pointer computer 7 type. Coordinate recognition unit 60
Is corrected by using a coordinate correspondence table stored in the memory 65 in advance for the input coordinates (X, Y), so that the coordinates of the actual designated position on the display screen of the large screen display device 4 ( X _i , Y _i ). By using the coordinate correspondence table, it is possible to correct the error of the indicated position caused by the distortion of the camera image and the influence of the camera position.
The coordinates (X _i , Y _i ) and the type information of the pointer are output to the display information creating unit 63, so that the coordinates (X _i , Y _I ) on the screen of the large screen display device 4 correspond to the type information of the pointer. The pointer marker (the cross-shaped marker 5b for the pointer Pb) is displayed. Since the instruction signals output from the pointers Pa and Pb are infrared rays, the administrator and the operator cannot confirm where they are pointing as they are. However, this embodiment, as described above,
Since the marker corresponding to the type of the pointer is displayed, the position pointed to by the user can be easily confirmed. The object specifying unit 61 specifies an object displayed at a position (on the screen of the large screen display device 4) indicated by the pointer Pb (or Pa). That is, the object specifying unit 61
Retrieves and inputs the data of the display information currently displayed on the large screen display device 4 from the memory 65, and inputs the coordinates (X _i , Y _I ) of the designated point, for example, JP-A-3-179517.
The coordinates (X _i , Y _i ) of the designated point are set in the same manner as the processing shown in FIG.
The object displayed at the position of is specified. The display information determination unit 62 combines the information of the specified object with the voice information of the manager B input from the headset microphone Mb (input from the headset microphone Ma in the case of the driver A). The information to be displayed is determined based on the combined information. The command signal relating to the information to be displayed is output to the display information creating unit 63. The display information determination unit 62 is described in, for example, JP-A-3-179517, page 4, upper left column 15
6 lines from the bottom, 3 lines in the lower left column (FIGS. 1, 5 to 7), and 8
5 lines from the upper left column of the page to 19 lines from the lower left column of the same page (see FIGS. 11 and 1).
3 and FIG. 14), the information to be displayed is determined using the information of the specified target object and the voice information of the manager B (or the operator A).

The display information creation unit 63 includes a display information determination unit 6
Display information corresponding to the display information to be displayed determined in step 2 is created using the information in the memory 65. In creating the display information, the type information of the pointer is also considered. The pointer type information is input to the display information creating unit 63 via the coordinate recognizing unit 60, the object specifying unit 61, and the display information determining unit 62. The display information creation unit 63 includes, for example,
The display information on the object pointed by the pointer Pb is created based on the data stored in the memory 65 corresponding to the type information of the pointer. The created information is output to a display device determined according to the type of the pointer, and is displayed there. Display information related to the object pointed by the pointer Pb is displayed on the large screen display device 4.

When the operator A indicates an object displayed on the large-screen display device 4 by the instruction signal (one dot) output from the pointer Pa, the image is displayed in the same manner as when the instruction is made by the pointer Pb. Processing device 2 and computer 7
Is executed. In the 2d and 2e the image processing apparatus 2, using the coordinates x _1, x _2, y ₁ and y ₂ of Figure 4 obtained in process 2c (a), obtains the coordinates (X, Y) of the center position, area △ determines the type of the pointer from the S ₁ (pointer Pa).
The coordinate recognizing unit 60 corrects the position of the coordinates (X, Y) with respect to the instruction signal output from the pointer Pa, and outputs the coordinates (X _i ,
Y _i ). The display information creating unit 63 displays the one-dot marker 5a on the large screen display device 4 using the coordinate values. The object specifying unit 61 and the display information determining unit 62 perform the same processing as in the case of the pointer Pb. Display information creation unit 63
Creates display information on the object pointed by the pointer Pa based on the data stored in the memory 65 corresponding to the type information of the pointer. The created information is displayed on, for example, the CRT display device 6A corresponding to the pointer Pa.

As described above, according to the present embodiment, a plurality of pointers indicating the display screen of the large screen display device 4 can be identified.
When the display screen is designated, the corresponding display information can be created and displayed on the display device. In addition, a plurality of persons can simultaneously operate the large screen display device 4 with respective pointers.
Can properly create and display the information requested by each of the display screens.

Since the display device for displaying information corresponding to the pointer is specified, the information on the object pointed by the manager B with the pointer Pb (the information to be viewed by all) is displayed on the large screen display device 4. Information on the object indicated by the operator A with the pointer Pa (information to be viewed by the individual A) can be displayed on the CRT display 6A. As described above, since the display device for displaying information is specified according to the type of the pointer, the usability of the information of the operator A and the administrator B is improved. The operator A can also access the information by touch operation of the CRT display device 6A. In particular, when the operator A points to the large-screen display device 4 with the pointer Pa, information related to the information displayed on the large-screen display device 4 can be displayed on the CRT display device 6A to be monitored by the operator A. This is made possible by identifying the pointer.

Further, a head mount display HD
("Newton", Vol. 11, No. 12, P104
107, 1991, KYOIKUSHA), the operator A points to the large-screen display device 4 with the pointer Pa while moving, and personal information necessary for the operator A Can be displayed on the head mount display HD.

Since the operator A and the manager B remotely instruct the display screen using the pointer, the operator A and the administrator B instruct the display screen of the large screen display device 4 regardless of the positions of the operator A and the administrator B. Can be. Further, regardless of the positions of the operator A and the manager B, the detection of the indicated position and the determination of the type of the pointer are accurately performed.

According to the present embodiment, it is possible to identify a plurality of pointers that output different instruction signals, and appropriately create and display information on the pointing object pointed by the identified pointer. For this reason, a plurality of users can appropriately obtain the information they need by using different pointers that output different instruction signals. In addition, even when a plurality of users simultaneously designate with the pointer, the information requested by each is displayed appropriately.

According to the present embodiment, a plurality of persons can designate an instruction target without limiting the operation place, and information for each person and common information can be respectively displayed on different display devices or a common display device. It is possible to output,
It is effective when applied to the provision of information in the central control room of a plant operated by multiple operators, the provision of information at financial institutions and securities companies, the presentation of meetings, etc., and the work efficiency and reliability of work are also effective. It can be further enhanced, and its industrial value is extremely high.

Further, since the information to be displayed is determined by combining the information of the display object indicated by the pointer and the audio information, the upper input by the pointer and the audio input means is not restricted, and they can be freely combined. Can be used. For this reason, the burden on the operator or the like for inputting information using the pointer and the voice input unit can be reduced.

4 (b) and 4 (b) using the infrared light video signal output from the monitor camera 1 even when the instruction is performed using the 2-dot type and 3-dot type pointers (emit infrared laser light). Features as shown in c) can be obtained in process 2c. When a plurality of peaks appear in the projection distribution, the center coordinate and the area ΔS of the circumscribed rectangle are calculated using the maximum value and the minimum value on the x-axis and the y-axis. For example, in the pointer 2 dot type, the projection distribution in the x-axis, determine the x ₁ and x _2, the projection distribution in the y-axis, seeking y ₁ and y _2, the area of the center coordinates and circumscribed rectangle Ask for. When the center coordinates are not determined, the minimum value of the projection distribution on the x-axis and the y-axis, for example, FIG.
The (x ₁ , y ₁ ) coordinates of (b) may be obtained.

The distinction of the pointer used for indicating the position is 2
This is performed using data such as the number of points having high luminance in the value image or the area of a circumscribed rectangle of a region having high luminance.
Also, a method of using the geometric shape of the instruction signal by the pointer, storing the shape in a memory (not shown) in the image processing device 2 in advance, and detecting the same shape by the template matching method. Good. The case where pointers are distinguished by obtaining the number of points having high luminance is a case where dot-type pointers shown in FIGS. 4A to 4C are used, and the projection distribution after binarization of the x-axis or the y-axis. In, the judgment is made based on the maximum value of the number of peaks. For example, FIG.
In the case of the three-dot type pointer (c), the number of peaks on the x-axis is the largest and the number is three, so it is determined to be the three-dot type. By the way, in the 4-dot and 5-dot pointers shown in FIG.
In some cases, the maximum value of the number of peaks in the projection distribution on the y axis and the y axis does not match the number of light emitting sources. For the pointer, the other reference value L ₂ higher level than the reference value L ₁ for extracting a feature of a binary image. That is, the total number of the number of L ₂ or more peaks of L ₁ or more peaks, x-axis, seeking the greater the y-axis, it is possible to identify the pointer by the number. For example, in the y-axis of FIG. 4 (c), the number of peaks is 2, L ₁ or more peaks is two, because L ₂ or more peaks is 1, 3 in total
Becomes x-axis, L ₁ or more peaks are three, L ₂
Since the above is zero, the total is three. Therefore, it can be determined that the pointer is pointed by the 3-dot type pointer.

Here, an example of the structure of the pointer will be described. 6 and 7 show a pointer P having three dots. Kj is a DC power supply, and Ke is a protection resistor for limiting current. The pointer P applies a voltage to the drive circuits Kd ₁ , Kd ₂ , Kd ₃ and the light emitting diode K ₄ by turning on the switch K ₅ . Therefore, the semiconductor lasers Kb ₁ , Kb ₂ , and Kb ₃ output infrared laser light, respectively. These infrared laser beams are converted into parallel beams Kc ₁ , Kc ₁ by collimating lenses Ka ₁ , Ka ₂ , and Ka ₃ .
c ₂ and Kc ₃ . Each infrared laser beam without spread,
The light is output as light emitted from the light emitting sources K ₁ , K ₂ , and K ₃ on the light emitting surface K ₆ . As a result, the infrared laser light (instruction signal) output from the pointer P indicates the display screen of the large-screen cover device 4, as shown in FIG. By the functions of the coordinate recognition unit 60 and the display information creation unit 63, three points 5
Markers having A, 5B, and 5C are displayed on the display screen of the large screen display device 4. If the switch K ₅ to OFF, the emission of the infrared laser beam from the pointer P is stopped. If the light emitting diode K ₄ is intended to indicate that the light from the light source K _1, K _2, K ₃ are output, not visible from the light source but the far-infrared light (e.g., infrared light) is output Especially effective.

FIGS. 9 and 10 show the structure of the pointer Pb used in the embodiment of FIG. The pointer Pb has two light emitting sources Kh and Ki. The light source Kh is a semiconductor laser
Kj ₁ , a focusing lens Km ₁ , a collimating lens Kn ₁ and a slit Kr ₁ . Similarly, the light emitting source Ki includes a semiconductor laser Kj ₂ , a focusing lens Km ₂ , a collimating lens Kn _2, and a slit Kr ₂ . The infrared laser light on the horizontal axis of the marker 5b is called X-axis output light, and the infrared laser light on the vertical axis is called Y-axis output light. Slit K
infrared laser light emitted from the r ₁ is the X-axis output light Ks _1, an infrared laser beam emitted from the slit Kr ₂ is a Y-axis output light Ks _2. X-axis output light Ks ₁ and Y-axis output light Ks ₂ is a light emitted respectively rodlike, are combined focus so that the cross on the display surface of the large-screen display device 4. For this purpose, focus lenses Km ₁ and Km ₂ are provided between the semiconductor laser and the collimator lens. Even if the distance between the pointer Pb and the large screen display device 4 changes,
X-axis output light Ks ₁ and Y-axis output light Ks ₂ rather than to intersect at the center point to each other, intersect at point shifted position. Thus, a cross-shaped instruction signal is obtained. The slits Kr ₁ and Kr ₂ are formed so that the X-axis output light Ks ₁ and the Y-axis output light Ks ₂ become more rod-shaped, and the width of the portion perpendicular to the axial direction becomes a constant value. It is provided to cut off.

By combining a plurality of rod-like infrared laser beams in this manner, it is possible to obtain a pointer that outputs an instruction signal for a parallel bar, a single bar, a T-shape or the like.

An operation / monitoring device according to another embodiment of the present invention will be described with reference to FIG. The same components as those in the above-described embodiment are denoted by the same reference numerals. In this embodiment, the pointers Pa, P
b, monitor camera 1, image processing device 2, coordinate conversion means 1
0, the object specifying means 11, the voice input device 16A ₁ ~ 16
A _N (headset microphones Ma and Mb in FIG. 1 are examples),
It includes a command input analysis device 15, an operation / instruction device 9, an information generating means 25, a first display device 26, and a large screen display device 4 as a second display device. Information generating means 25, the command output unit 12, the controller 13A ₁ ~13A _j, the display control unit 2
0, a memory 21, and audio output devices 24 and 24A. The first display unit 26 is provided with a CRT display device 6A~6G and head-mounted display HD ₁ ~HD _k.

The pointer Pa outputs an instruction signal similar to that in the embodiment of FIG. 1 to the display screen of the large screen display device 4 by the operation of the operator A and the pointer Pb by the operation of the manager B. The image processing device 2 captures a video signal of the display screen of the large-screen display device 4 captured by the monitor camera 1, and executes processes 2a to 2f shown in FIG.

The noise included in the video signal is removed by the binarization in the process 2b. In the process 2d, the center coordinates of the binary image are obtained as the pointer pointing position because the shape of the pointing signal formed on the large-screen display device 4 is deformed due to the difference in the location pointed by the pointer. This is to prevent the designated position from being shifted from the position of the designated signal detected on the display screen of the large screen display device 4. However, if the pointing position on the display screen is deviated, if the operator (or the administrator) points again with the pointer to solve the problem, the center coordinates of the binary image need not necessarily be obtained. What is necessary is just to find the coordinates of the specific position of the instruction signal. The image processing device 2 outputs the type of the pointer and the coordinates (X, Y) of the point indicated by the pointer to the coordinate conversion means 10.

The coordinate conversion means 10 is provided with the coordinate recognition unit 6 shown in FIG.
0 same processing as is executed, the input coordinates (X, Y) corrected obtains and outputs the actual indicated position coordinates in the large screen display device 4 of the display screen (X _i, Y _i). This correction is performed using the coordinate correspondence table. For example, infrared coordinates are embedded in the four corners of the large-screen display device 4 and the infrared light sources are allowed to emit light so that the coordinates can be corresponded. Since it is infrared, it is invisible to the user and effective. In addition, the large screen display device 4 itself outputs information of a specific pattern (for example, only the four corners of the screen are made white and the others are made black), and only when this information is output, the monitor camera 1 is temporarily turned off.
It is also possible to create coordinate conversion information by increasing the sensitivity of the pattern so that the specific pattern can be recognized.

The object specifying means 11 is based on the coordinates (X _i , Y _i ) in the large screen display device 4 obtained by the coordinate conversion means 10.
The display object on the display screen specified by the operator or the like is specified. This specification is performed by the processing shown in FIG. Coordinates (X _i, Y _i) of the position indicated on the large-screen display devices within 4 before turning off the infrared laser beam outputted from the pointer read (process 50). The designated target window is determined using the coordinates (process 51). Next, the relative coordinates (Xw, Yw) in the target window are calculated (process 52). Using the corresponding data of the designated area / character string for the figure displayed in the target window, the relative coordinates (X
(w, Yw), that is, the indicated display object is determined (process 53). The designated area / character string correspondence data is prepared for each drawing in the display screen data stored in the memory 21. The contents are data relating to the range of the designated area and the output character string. When the display object pointed by the pointer is determined, a character string corresponding to the display object is output. The information of the character string corresponding to the specified display object is displayed on the display control device 2.
0 is input to the memory 21. The transmission of information between the memory 21 and the object specifying means 11 is performed by the display control device 20.

Since the light emitted from the pointers Pa and Pb is infrared light, the operator cannot confirm where the point he is pointing is. Therefore, the display control device 20 captures the relative coordinates (X _W , X _W ) obtained by the object specifying means 11 and the type information of the pointer obtained in the processing 2 e of the image processing device 2, and sets the type of the pointer in advance. The marker corresponding to the pointer outputting the instruction signal is displayed on the large-screen display device 4 at the position of the coordinates (X _W , X _W ) using the information of the marker associated with. As a result, a marker corresponding to the pointer can be displayed at the point indicated by the pointer. Here, the display control device 20 controls the object specifying unit 11.
Instead of taking in the relative coordinates (X _W , X _W ), a character string specifying the object may be taken in. In this case, it is possible to display a marker at the coordinates of the large-screen display device 4 in the same manner as described above by determining which coordinates of the object represented by the character string are present. In this case, since the display object has been specified, the designated point is displayed by blinking the display object displayed on the large screen display device 4 or changing the display color of the object. It is possible to In this case, a blink cycle and a display color corresponding to the instruction signal can be set.

The input by the operator A is performed by voice information or an instruction using the pointer Pa. Audio information is input to the command input analyzing device 15 via the voice input device 16A ₁ ~16A _N, after being recognized is input to the display control unit 20 as speech recognition information 17. Character string information corresponding to the display target specified by the target specifying unit 11 is also input to the display control device 20. Display control device 20
Treats these pieces of input information as natural language words, stores them in the memory 21, and displays them on the large screen display device 4. For example, it is displayed in a frame below the large screen display device 4 shown in FIG. The operator or the like can monitor the input information recognized by the operation / monitoring device of this embodiment by looking at the information displayed in the frame.

When the input of the voice information from the driver or the like is completed, the display control device 20 integrates the words of the voice information stored in the memory 21 and the character string information of the display object into one sentence, Next, the sentence is analyzed, that is, the input sentence is analyzed. The display control device 20 has an input sentence analysis unit (not shown) inside.

The input sentence analysis performed by the input sentence analyzing unit is performed by the processing described in Japanese Patent Laid-Open No. 3-179517 (described in the embodiment of FIG. 1). FIG. 13 shows an outline of the process with respect to an input sentence "show a reactor water level trend". In this input sentence, "reactor" is the character string information of the display object at the position indicated by the pointer, and "show water level trend" is voice input information. First, the input sentence is divided and written with reference to the grammar and the dictionary, and a meaning and a semantic category are added (process 38). Here, the meaning category is obtained by categorizing the meaning of words used in the target world. In the example of FIG. 13, the system, the state quantity, and the like correspond to it. Next, the meaning of the input sentence is analyzed using this result (process 39).

The semantic analysis is performed in two steps as shown in FIG. First, the type of the request content is identified (). Standard sentence patterns are provided for each type of request for identification. These are described in the form of rules using meanings or semantic categories. By matching the sentence pattern with the meaning or meaning category of the input sentence, the type of request is identified. In the example shown,
“<Trend graph〉 □ present ”is matched, and the type of request is identified as trend presentation request“ trend ”, where the symbol“ <> ”means that semantic category is used for matching, and“ □ ” Indicates that any number of words may be included.

Next, information necessary for executing the request is obtained.
(). Necessary information and a method of obtaining the information are prepared as frame type knowledge. Information is acquired using this knowledge. In the example of FIG. 13, the necessary information, that is, the system device name and the state quantity name, are both obtained from the input sentence. If the necessary information is not included in the input sentence, the information is acquired by using plant-related knowledge, dialogue history, or inquiry to the operator. When the analysis of the meaning is completed, it is output as a processing instruction (processing 40). It outputs the content of the information to be displayed as a processing instruction. Such an input sentence analysis unit corresponds to the display information determination unit in the embodiment of FIG.

The display control device 20 has a display information creation unit (not shown) inside. The display information creation unit creates display information based on the data of the information to be displayed obtained by the input sentence analysis unit, and displays the display information on the display device corresponding to the type of the pointer specified by the image processing device 2. Output information. The created display information is created using information (information necessary for creating display information) corresponding to the type of the pointer read from the memory 21. FIG. 14 shows an example of the processing in the display information creation unit, taking display information creation of a system diagram as an example. In the case of a system diagram display information creation command, the system device name is obtained as necessary information in the processing of the input sentence analysis unit. The corresponding display diagram is determined based on this system device name. First, a system diagram component database configured in a partial storage area in the memory 21 is searched based on the given system device name (process 41). This database gives names of drawings and names of components such as devices included in the drawings. The drawing name and the system device name are compared (process 42), and if there is a match, a display command of the system diagram is output (process 47). This is a process corresponding to a case where the operator stores the drawing name and inputs the drawing name as it is. As a result of the search, if it is determined in the process 42 that there is no corresponding drawing, the drawing component names stored in the database are searched based on the input system device name. (Process 43). As a result of this search, when it is determined that there is no corresponding element name (process 44), the fact that there is no related drawing is output (process 48), and the operator is requested to input again. On the other hand, if the element name is found and there is only one drawing name including the element (step 45), a display command of the system diagram is output (step 49). On the other hand, if there are two or more drawing names, a drawing selection process is performed.

The instruction to present the information selected as a result of this processing is also input to the audio output device 24 and output as the audio information 23. The large screen display device 4, the CRT display device 6
A～6G, is used to control the display of the screen of the head-mounted Display HD ₁ ~HD _K. FIG. 15 shows the types of the screen display switching command stored in the memory 21 as an example. The display of the screen is switched according to these processing instructions. Which display device to switch the display on is determined based on the pointer type information obtained by the image processing device 2. That is, the type of pointer (pointer Pa,
Pb, etc.), and a display device is determined according to the type of pointer used to instruct the display screen of the large screen display device 4. Thus, the display information 26 created by the display information creation unit of the display control device 20 can be displayed on the display device determined corresponding to the pointer.

In the processing items of FIG. 15, the system diagram display is to display the system diagram, the trend diagram display is to display the trend diagram, and the window deletion is to delete the displayed window. is there. Window moving means moving the displayed window, window popping means displaying the displayed window in front of the display surface, and window push means moving the displayed window to the display surface. Display on the back.

FIG. 16 shows the concept of the operation relating to the display of this embodiment when the CRT display devices 6A to 6G and the large screen display device 4 are used. For example, FIG.
In the large screen display device 4 described in the center of the figure, the operator A indicates the feed pump portion of the whole system diagram of the plant displayed on the large screen display device 4 by the instruction signal output from the pointer Pa, and When the operator A inputs “show system diagram” by voice, as shown in (c),
A system diagram near the water supply pump is displayed on the CRT display device 6A. Pointing at the same place with an instruction signal of the pointer Pa,
When the operator A voice-inputs “flow rate trend display”, a trend diagram of the water supply pump flow rate is displayed on the CRT display device 6A as shown in (d). The manager B instructs the feed pump portion of the whole system diagram of the plant displayed on the large screen display device 4 by an instruction signal output from the pointer Pb, and the manager B inputs "show system diagram" by voice. In this case, a system diagram near the water supply pump is displayed on the large screen display device 4 as shown in FIG. When the same point is indicated by the instruction signal of the pointer Pb and the manager B inputs a voice "Flow rate trend display", a trend chart of the feed water pump flow rate is displayed on the large screen display device 4 as shown in (b). Is done. Such display is possible because the instruction signal of the pointer Pb has a cross shape and the instruction signal of the pointer Pa has a dot shape. These instruction signals can be identified, and based on the identification result, information is output to the display device corresponding to the pointer as described above. This is also done in the embodiment of FIG.

According to this embodiment, the same effect as that of the embodiment of FIG. 1 can be obtained.

The display information specified by the pointers Pa and Pb and specified by the image processing apparatus 2 and the display information 26 created based on the voice input information are stored in the pointers Pa and Pb.
It can be displayed on a display device determined according to the instruction signal output from Pb. For this reason, information required by each operator and information that all operators (including managers) need to know can be remotely instructed. Is eliminated, and information presentation with high man-machine property is possible. In addition, since the indication of the display target can be made with a plurality of pointers, there is no need to shift the timing of use between the users, and the use efficiency is high.

This embodiment not only outputs the display information 26 to the first display device 26 and the large screen display device 4, but also
As follows, it is possible to output the control information 14A ₁ to 14A _j for controlling the controlled object of the plant.

[0055] command output means 12 inputs the operation command information 18 outputted from the display control unit 20, or which outputs a which the control command signal of the original to the controller 13A ₁ ~13A _j Determine and output a control command signal to the corresponding control device.

The creation of the operation command information 18 is as follows. One device in the system diagram displayed on the large screen display device 4 is indicated by the pointer Pa. When the display control device 20 receives an output signal indicating a control command from the operation / instruction device 9 by the operation of the operator A, the display object specified by the image processing device 2 (the above-described instruction ) And the voice information of the operator A are integrated and output as operation command information 18.

Each of the control devices 13A _{1 to} 13A _j performs a control operation based on the control command signal output from the command output means 12 and the output signal from the operation / instruction device 9, and executes the corresponding controlled object. things to outputs operation information 14A ₁ to 14A _j. For example, it is assumed that the operation command information 18 of “Motor A, the rotation speed is set to 1000 rpm” is output from the display control device 20 and the command output unit 12 is input.
The command output means 12 performs an input sentence analysis and a semantic analysis on the input information in the same manner as the display control device 20, and determines a control device (for example, the control device 13A ₁ ) for controlling the motor A. relative 13A _1, and outputs a setting signal corresponding to 1000rpm, which is a target value of the rotational speed. Controller 13A _1, a signal output from the operation-indicating device 9 for the setting signal - captures (Fi from plant Dobakku signal indicating the rotational speed of the motor A is a signal 19), the setting signal and the motor The control information (control signal) is output so that the signal indicating the rotation speed of A matches.

The command output means 12 determines whether or not the type information of the pointer output from the display control device 20 and the control device for outputting the control information correspond to each other. The operation command information 18 is not executed. As described above, since the range of the control function for the pointer that outputs the instruction signal to the large screen display device 2 can be limited, the reliability of the plant operation can be improved.

In the embodiment shown in FIG.
0, object specifying means 11, display control device 20, memory 2
1 and the command input analysis device 15 can be realized by a workstation. FIG. 17 shows a configuration example of the display information generation function realized by the workstation. The workstation 250 includes a coordinate recognition unit 31, an object identification unit 32, an input control unit 33, an input sentence analysis unit 34,
A display information creation unit 36 and a memory 37 storing display screen data are included.

The coordinate recognizing unit 31 has the same function as the coordinate recognizing unit 60, and obtains the coordinates (X _i , X _i ) of the actual designated point on the display screen of the large screen display device 4. The object specifying unit 32
In the process of FIG. 12, the coordinates (X _W , X _W ) are obtained, the display object pointed by the pointer is determined, and character string information corresponding to the display object is output. The input control unit 33 combines the character string information and the voice information input from the voice input device into one sentence of a natural language, and outputs the sentence to the input sentence analysis unit 34. As in the embodiment of FIG. 1, the input sentence analyzing unit 34 is composed of 15 lines from the upper left column on page 4 to 3 lines from the lower left column on page 6 and from 5 lines from the upper left column on page 8 to the lower left column from the same page in JP-A-3-179517. The processing described in line 19, for example, the processing shown in FIG. 13 is performed. The input control unit 33 and the input sentence analyzing unit 34 constitute a display information determining unit in the embodiment of FIGS. The display information creation unit 36 has the same function as the display information creation unit in the embodiment of FIGS. With such a configuration, these functions can be realized by software processing.

In the above description, the output from the pointer is infrared light, but may be visible light. For example, in each type of pointer of FIGS. 5, 6 and 9,
Red light, blue light, and green light may be output independently, or red, blue, and green lights may be combined and output. Thus, even with a pointer that outputs an instruction signal having the same shape, it is possible to increase the types of instruction signals, that is, the types of pointers, by merely changing the color of the emitted light.

For this purpose, the monitor camera 1 is for color, and the image processing device 2 is for R (red), G (green), B
Must have a frame memory for each (blue).
Further, the image processing device 2 obtains the projection distribution for each of R, G, and B with respect to the video information fetched into each frame memory, obtains the coordinates of the designated position based on these results, and determines the type of the pointer. Identify. For example, the projection distribution after the binarization processing is such that, in the case of the instruction signal in red, the processing is performed only on the data of the frame memory of R, and in the instruction signal of green, the processing is performed only on the processing of the data of the frame memory of G. Appears. Therefore, by judging to which frame memory data the result has appeared, it is possible to easily distinguish whether the color is different for an instruction signal of the same shape. Therefore, the pointer may be identified using this information.

When the light emitted from the pointer is infrared light, the pointer can be identified even if the wavelength of the emitted light is different for each pointer. In the case of visible light, the color changes. By changing the wavelength of the infrared light emitted from the plurality of pointers and installing a plurality of monitor cameras equipped with infrared filters corresponding to the wavelength of the infrared light to be monitored, each monitor camera has a different wavelength. Only the infrared light (instruction signal) that is the light emitted from the corresponding pointer can be detected. This makes it possible to use only one light source for the pointer. Further, even if the instruction signals output by the respective pointers indicate the same location on the display screen, the image processing apparatus can distinguish and detect the instruction signals from the respective pointers.

FIG. 18 shows a pointer capable of emitting both infrared light and visible light. Two points 5A and 5B on the display screen of the large screen display device 4 are infrared light, and one point 5D
Is visible light. This is to indicate whether the indicated where the pointer P ₁ in the visible light. K6 is a visible light source, and can be configured in the same manner as in FIG. 7 using, for example, a semiconductor laser that emits visible light.

[0065] Pointer P _1, when applied to the case can not be displayed on the display device a marker described above, the effect is large. For example, a mimic display device shows a system diagram with a mosaic on a panel, for example, when the state of a valve changes, turns on a colored lamp to indicate the state, or embeds an indicator or the like in a mosaic part, and displays the state of that part Quantity (for example,
(Water level) is displayed on the indicator. In such a display device, the marker cannot be presented on the display surface. By means of a pointer, for example, the above-mentioned indicator is pointed, and detailed information and related information on
When trying to output to an RT display device, it is necessary to check whether the position of the instruction is the intended place. In such a case, if a pointer P _1, since the user of the visible light can be visually indicated is reliably and easily.

Further, by presenting the pointed object using voice information, it is possible to confirm that the pointed position is the object intended by the user, so that a more reliable instruction can be realized. In this case, the specified target object is converted into audio information based on the result obtained by the target object specifying means 11 as described in the embodiment of FIG. Information 23 can be provided.

Examples of presenting information using a mimic display device include plant monitoring in a central control room of a nuclear power plant, a thermal power plant, a steel plant, a chemical plant, etc. And monitoring of train operation status in railways, presentation of stock price information in finance and securities, and the like.

By using the above-mentioned pointer P ₁ , it is possible to point to a switch or an indicator of the operation / monitoring panel and to present information relating to this.

In these cases as well, it is necessary to determine which position of the instruction signal indicates the mimic display device and the operation / monitoring panel. However, it is possible to identify the specified target by providing infrared light emitting sources at the four corners of the mimic display device and the operation / monitoring panel and storing in advance the relative distance from these positions. It is. Alternatively, the pointer shown in FIG. 19 may be used. These all have a visible light source. In FIG.
(a) to (e) include one light source that emits red light, and a green light source is arranged around the light source. This is because when using a plurality of pointers, each user distinguishes each pointer with a green instruction signal and confirms the indicated position with a red instruction signal. (F) to (j) in FIG.
Conversely, each pointer is distinguished by a red pointing signal, and the pointing position is confirmed by a green pointing signal. As described above, the number of identifiable pointers can be increased by using one piece of optical information for indicating a designated point and a plurality of pieces of optical information for distinguishing pointers. The image processing apparatus only needs to handle the light emitted from all of these light sources as the instruction signal. That is, the pointer enables the user to visually confirm the indicated point.

The pointer is used to indicate information to be captured on the screen, to indicate panel display information, and the like, and is particularly effective when it is desired that another user grasps the point indicated by a certain user. is there. Such uses include meetings, explanations, meetings, presentations, and the like.

FIG. 20 shows the configuration of an operation / monitoring device having a large-screen display device according to another embodiment of the present invention. This embodiment is not shown, the image processing apparatus 2 of the embodiment of FIG. 11, coordinate conversion means 10, the object identification unit 11, voice input device 16A ₁ ~16A _N, command input analyzer 15,
An information generating means 25 is provided. 4A and 4B are large-screen display devices, and 40 and 41 are backup panels (so-called operation / monitoring panels) capable of performing operations related to plant operation.
It is. The backup panels 40 and 41 are provided with a system-specific alarm display device 8 at the upper part. The display device 8 has an alarm tile 8 _i on which an alarm item is stamped, and turns on a lamp attached to the alarm tile 8 _i when displaying an alarm. Therefore, it is possible to know which system has generated the alarm. However, in the system in which the alarm occurred, which of the plurality of alarm items occurred cannot be confirmed only by the system-specific alarm display device 8, and the contents thereof are displayed on the individual CRT display devices 6A to 6G. Will be displayed. In addition, the backup panels 40 and 41
Further includes a device status display lamp, a CRT display device, an EL display, operation switches, a recorder, and the like. This is the panel display. The infrared light emitting source 40A ₁
～40A _4, when an instruction to board the backup plate 40 with the pointer, coordinates indication object position indicated by the indicator signal which is captured by the monitor camera 1 or to correspond to positions on the board surface of the backup plate 40 Used for Other infrared-emitting source _{41A 1 ~41A 4, 4A 1 ~4A} 4, 4B 1
~4B ₄ is also the same.

The CRT display devices 6A to 6G provided on the operation monitoring panel 3 display information necessary for the operators A ₁ and A ₂ for plant operation monitoring. Display data to the CRT display device 6A~6G are created respectively on the basis of the result information and the operator A ₁ to A ₂ from the plant has operated from the operating means 7A-7C.

On the other hand, the information displayed on the large screen display devices 4A and 4B is monitored not only by the operator but also by the manager B who monitors the plant. The important parameters and the state of important equipment are displayed.The message and the state of important equipment change in the whole plant configuration according to the operating state, and the values of important parameters sometimes change. It is displayed every moment. Further, the situation of the entire plant is monitored using the two large screen display devices 4A and 4B.

Next, specific examples of instructions and displays will be described in detail. The example is for a nuclear plant.

[0075] illustrates an example of outputting the information operator A ₁ is involved in the control rod operation by utilizing the large-screen display device 4A on the CRT display device 6E.

[0076] First, by the pointer Pa _1, for example, instructs the core display section (the part indicated by the instruction signal 5a). As a result, the image processing apparatus 2 recognizes the pointing target and the pointer used for the pointing. As a result, FIG.
Is displayed on the CRT display device 6E corresponding to the type of the pointer. The processing for display is executed by the display control device 20 of FIG. Large screen display 4A
When the information shown in FIG. 21 is displayed in FIG. 21, instead of erasing the already displayed information and newly displaying the information, as shown in FIG. 22, new display information E is displayed on the already displayed information.
Is desirably displayed. This is because according to this display method, it is possible to grasp the outline of the information relating to the entire plant. The new display information E in FIG. 22 may be displayed on the CRT display device 6E in place of the already displayed information, or may be displayed in an overlapping manner as in the large-screen display device 4A. If there is a margin on the display screens of the large screen display device 4A and the CRT display device 6E, new display information E may be displayed there.

[0077] As shown in FIG. 22, when a new display information E is displayed on the large-screen display devices 4A, using the pointer Pa _1, the information selected by instructing the necessary portions of the display information E.
In this case, since the information relating to the control rod operation is required, by a finger difference a control rod drive system E ₁ of the display information, for example information relating to the control rod operation shown in FIG. 23 on the CRT display device 6E Can be displayed. In this case, since the information E shown in FIG. 22 is unnecessary, the display control device 20 performs a process for deleting the information E displayed on the large screen display device 4A. Also, in other to display information shown in FIG. 23 on the CRT display device 6E, a control rod drive system E ₁ in FIG. 22 in the pointer Pa ₁ instead of finger difference, using a headset microphone Ma _1, "Control "Rod drive system, operation information, show me". FIG.
In the example, new display information is displayed on the large screen display device, but whether to display the new display information on either the large screen display device or the CRT display device may be determined in advance by the type of the pointer.

If each CRT display device is capable of touch operation, the information E in FIG.
If it is output to, by touching the display portion of the control rod drive system E ₁ with a finger, the information of Figure 23 CR
It can be displayed on the T display device 6E. Further, in the same manner as described above, by using a headset microphone Ma _1, it may be input by voice. When the CRT display device 6E screen as shown in Figure 23 is displayed, the operator A ₁ is the "operating mode", "drive mode", "sequence type", the operation instruction by a touch operation to "control rod No.", etc. I do. Hatched portion in FIG. 23 displays the results of operator A ₁ has operated. Further, as the black portion in the control rod coordinates are displayed on the left side of FIG. 23 is a result of the operator A ₁ was selected for control rods, what control rod is selected, the control rod addresses (e.g., 26-27, 26-35, 3
4 to 35). FIG. 23 shows a display example of a CRT with a touch operation. By displaying the operation means, the operation and information obtained as a result of the operation (eg, information on the driving state of the control rod) are output on the same screen. Therefore, it is very convenient for the operator to check the operation state.

In this case, the display may be distributed not only to the CRT display device 6E but also to another CRT display device.

[0080] Next, an example of outputting the new information to the CRT display device 6B by utilizing backup board 40 other operator A ₂ is an operation and monitoring panel. An example in which alarm information (alarm information) is output to the CRT display device 6B as information will be described.

First, when an alarm item of a specific system is selected from the system-specific alarm display device 8 by the pointer Pa ₂ to see details, the corresponding alarm tile 8 _i is pointed. When the alarm tile _8i is instructed by the instruction signal 5b, the image processing device 2 is
Recognizes the indicated position and the type of the pointer. As a result, for example, the alarm information shown in FIG.
It is displayed on the RT display device 6B. In FIG. 24, the hatched portion indicates that the alarm (the MDRFP-A drip 8 _i1 and the MDRFP-A suction flow rate low 8 _i2 ) has occurred, but the CRT display device 6B does not. Change the display color to distinguish it from alarms that are not displayed. In FIG. 25, the time of occurrence is added to the generated alarm and displayed. 24 and 2
In each of the cases 5, information (water supply system) indicating which system is added.

FIG. 25 shows the alarm information displayed when the alarm tile _8i is designated. Of the display information shown in FIG. 24 displayed on the CRT display device 6B (touch operation enabled), the information shown in FIG. 25 may be output by touching a shaded portion. In this way, what kind of alarm items are in the system,
This has the effect that the user can easily grasp which alarm item has occurred among them and when the occurrence time has occurred.

In the above-mentioned alarm display, detailed alarm information is displayed by recognizing the indicated point without specifying the pointed object, for the following reason.

Even if the same location on the display surface of the display device is pointed out as in a large screen display device, if the contents of the display screen are different, different objects are pointed out. Therefore, when indicating the screen of the display device, it is necessary to specify the indicated display target. However, the operation
If the display target is fixed, as in a monitoring panel, the pointing point and the pointing target are always equal in a one-to-one relationship, so the coordinates of the pointing position can be determined without specifying the pointing target. And
Information on the coordinates may be output.

As described above, when the display object is always fixed, as in the case of a panel display device such as a mimic display, it is not always necessary to specify the indicated display object. By determining the points (coordinates), the coordinates (corresponding to the indicated display object) are
It is possible to output the information that has been allocated in advance to the display device corresponding to the pointer.

In the embodiment shown in FIG. 20, an instruction can be given to both the large screen display device and the operation / monitoring panel.
In the case of an instruction when the display object is fixed, such as a monitoring panel, an indication point is detected by the configuration of the operation / monitoring device according to another embodiment of the present invention shown in FIG. New information can be generated. The difference between the present embodiment and the embodiment of FIG. 11 is that the target object specifying means 11 of FIG. Since the designated point detecting means 180 only uses the output information of the coordinate converting means 10 as the information of the designated point, the coordinate converting means 10 may be considered as the designated point detecting means 180. This embodiment can achieve the same effects as the embodiment of FIG.

FIG. 27 shows the configuration of an operation / monitoring device according to another embodiment of the present invention. This embodiment is different from the embodiment of FIG. 26 in that the image processing device 2A
That is, the identification means 27 is provided instead of performing the processing 2e. The identification means 27 includes pointers Pa _{1 to} Pa _1
(N-1), identify the pointer and outputs an instruction signal based on the identification signal 27A ₁ through 27a _N output from Pb '.
This identification result is input to the display control device 20. On the other hand, the image processing device 2A obtains the position (coordinate) of the point indicated by the pointer. The obtained position coordinates are corrected by the coordinate conversion means 10, and then output to the display control device 20 via the designated point detection means 180.

This embodiment has the same effect as the embodiment of FIG. Furthermore, according to the present embodiment, the functions of the image processing apparatus 2 are simplified, so that the processing of the image processing apparatus can be shortened and the responsiveness of the entire operation / monitoring apparatus can be improved. Further, since the identification of the pointer is processed based on the identification signal and the identification by the image processing is unnecessary, it is possible to give an instruction with a pointer having the same instruction information such as the shape and the color.

Note that the identification signal output from each pointer may be an ultrasonic signal, a radio signal, or a wired signal. However, in order not to limit the location where the pointer is used, it is desirable to use an ultrasonic signal or a radio signal. In order to identify a pointer using these signals, various methods can be applied, such as a method in which the frequency and output cycle of the identification signal are different for each pointer.

An operation / monitoring device according to another embodiment of the present invention will be described. This embodiment is different from the embodiment shown in FIG. 27 in that the object specifying means 11 is provided in place of the designated point detecting means 180. Therefore, in addition to the effect of the embodiment of FIG. 27, the present invention is effective when specifying the specified target object and generating new information on the target object.

An operation / monitoring device according to another embodiment of the present invention will be described with reference to FIG. The present embodiment differs from the embodiment of FIG. 27 in that an object specifying means 11 is provided in place of the designated point detecting means 180, and an identification means 28A for identifying a pointer using voice input information in place of the identification means 27. ₁
In further comprising a ~28A _N. Identification means 28A ₁ ~ 2
Each 8A _N identifies the corresponding pointer by using the output signal of the corresponding voice input device of the voice input device 16A ₁ ~16A _N. By keeping association with the previously-one and voice information of the person voice input while using this pointer and pointer, identifying means 28A ₁ ~28A _N can identify a pointer based on the sound information. Which is an output signal identification information of the identification means 28A ₁ ~28A _N is taken into command input analyzer 15, with the analysis result of the command by the speech input, and output to the display control unit 20. As described above, it is possible to identify the pointer using the voice input information. Further, by previously in association with the previously-one pointers Pa~Pb an audio input device 16A ₁ ~16A _N, the identification means 28A ₁ ~28A _N output signals (voice information from the voice input device 16A ₁ ~16A _N ) Indicates whether or not the pointer is used. In the case of outputting the determination result as the identification information to the command input analyzer 15 is characterized in that the process of identification means 28A ₁ ~28A _N becomes very simple.

FIG. 29 shows an operation / monitoring device according to another embodiment of the present invention. This embodiment is different from the embodiment of FIG.
28 in that the object specifying means 11 in the embodiment of FIG. The embodiment of FIG. 28 can be applied to a case where the display target is fixed, such as a mimic display, but the present embodiment is also applicable to a case where the display target is variable. This embodiment can also obtain the same effect as the embodiment of FIG.

An operation / monitoring device according to another embodiment of the present invention will be described with reference to FIG. This embodiment is different from the embodiment of FIG. 11 in that the markers 5a, 5b, 5e, 5f, 5g,
.. Are displayed on the large screen display device 4 in advance, and the mark can be selected by applying an instruction signal such as the pointers Pa and Pb to one of these marks. By moving the selected mark on the display object to be instructed by operating the corresponding pointer, and eliminating the instruction signal emitted from the pointer, the display object is selected. These processes are performed by the display control device 20. The display control device 20 displays the markers 5a and the like registered in advance in a predetermined area of the large screen display device 4 (provided below the display area in FIG. 30).

In the embodiments shown in FIGS. 11 and 26 to 30, it is already possible to point to various display objects or display positions with the respective pointers and to generate new information relating to the pointed objects or the indicated positions. Stated. However, in the case where the role of each operator is determined as in the case of plant control, it may be desirable to limit the instruction range of each operator. In order to be able to cope with such a case, a pointing range is assigned to each pointer, and when pointing with a pointer, it is determined whether or not the pointing by the pointer is within the predetermined pointing range. It is necessary to judge and receive and process the instruction only when it is within the instruction range.

This determination process is executed by the display control device 20. In this case, the display control device 20 has a configuration shown in FIG.
Having. The configuration of FIG. 31 can be applied to the display control device 20 of each embodiment of FIGS. 11 and 26 to 30.

[0096] Allocation determination unit 20A has the indicated range registration information storing unit 20A ₁ and match determining means 20A _2. Coincidence determining means 20A _2, when inputted the identification information of the pointer indication position (pointer identification result information indicating a) target information (information indicating the object is located or instruction instructs) the indicated range Registration information storage means 20
From A ₁ reads the information (instruction range registration information) indicating a steerable range corresponding to the identification information. The read information is compared with the input designated position / target information, and if they match, permission information for permitting the instruction is output to the display control processing unit 20B. If they do not match, non-permission information is output. This makes it possible to limit the range of the instruction for each pointer.

The display control processing unit 20B is provided in FIG. 11 and FIG.
It has functions of an input sentence analysis unit and a display information creation unit in the display control device 20 of each embodiment of FIGS. Therefore,
When the permission information is input, the display control processing unit 20B
The corresponding display information is created by processing in the input sentence analysis unit and the display information creation unit, and is output to the corresponding display device.

In the description so far, the designated point or the designated object immediately before the designated information emitted from the pointer disappears is the designated point or the designated object designated by the pointer. However, if the user accidentally releases the switch of the pointer (K _{5 in} FIGS. 6 and 7) (OFF state) or another person crosses between the pointer and the large screen display device, the instruction information is temporarily stored. , And it may be considered that the user designates a part different from the designated point and the designated target to be designated. In contrast, this problem can be solved by assuming that the indication signal has truly disappeared only when the pointer switch is released twice within a certain period. In other words, it is assumed that the instruction signal disappears only once, is reissued within a certain period, and only when the instruction signal disappears. That is, it is determined that the instruction signal has disappeared in response to a double-click operation (a short operation of seconds or less) of the pointer switch. The processing for this is as shown in FIG. 32, and is executed by the image processing device 2 (or the display control device 20) in each of the embodiments of FIGS. 1, 11, and 26 to 30.

The processing in FIG. 32 will be specifically described. FIG.
Indicates a process when an instruction signal is detected, and this process is repeatedly executed. Output information J ₁ in FIG. 32,
This is information indicating that the instruction signal has correctly disappeared. Only when the output information J ₁ is obtained, the display control unit 20 generates a new information to instruction pointers. Output information J ₂
Only indicates the position of the instruction signal, and is used for displaying a marker or the like. In FIG. 32, when the instruction signal is issued three or more times, the output information A is not output, thereby improving the reliability of the instruction signal detection. When I actually tried the double-click operation with the pointer, I turned on the pointer switch three times several times.
I turned it off. For this reason, in FIG.
If the obtained, Soku, may be treated so as to output the output information J _1. As a result, it is possible to realize processing suitable for actual operation.

FIG. 33 is a diagram showing an output state of an instruction signal in response to a double-click operation of the pointer.

[0102] In the process of FIG. 32 detects an instruction signal in step S _1, it is determined whether or not an instruction signal has disappeared at step S _2. If the instruction signal has not disappeared, step 10 is processed, the process returns to the beginning, and steps S ₁ and S ₂ are processed. If it is determined that the instruction signal has disappeared at step S ₂ proceeds to Step S _3. This state is from t ₀ to t ₁ in FIG. 33, and is a state of ΔT ₁ .

[0102] the process proceeds to step S ₄ when the instruction signal in step S ₃ is determined not to exist. As shown in step 4, if the processing result is YES, step S is executed again.
Return to ₃ . That is, it exists during the period of ΔT ₁ . If the processing result of step 4 is NO, it means that the instruction signal has not been re-output within ΔT ₁ , and the process returns to the top after processing step S ₁₀ . That is, it is determined that the pointer has not been double-clicked. This allows
The reliability of indication signal detection is improved.

[0103] Then, the process proceeds to step S ₅ If the processing result of step S ₃ is a YES. This state corresponds to ΔT in FIG.
Indicates the status of ₂ . In step S _5,
Processing result is NO, the process proceeds to step S _6. In step S _6, △ if it is determined that T ₂ has elapsed even if the instruction signal does not disappear, it means that the double-click of the pointer is not performed, after the processing step S _10, Top. Processing result of the step S ₆ is equal YES, the flow returns to step S _5. If the processing result of step S ₅ is YES, the process proceeds to step S _7. State at step S ₇ is a state between t ₃ and t ₄ in FIG. 33. Processing result of step S ₇ is NO, and and also the processing of the step S ₈ NO
If so, the pointer has been double-clicked. Therefore, after the processing step S _9, the processing result of the step S ₈ Back to top If YES, the flow returns to step S _7. If the processing result of step S ₇ is YES, FIG. 33
Between t ₄ and t ₅ . In other words, it means that procedure was carried out over a double-click, after the process step S _10, Top.

As described above, only when the double-click operation is performed within a predetermined time, it is possible to regard that a specific location has been designated by the designation signal. However, as described above, double-click or more operations (e.g., three clicks operation) be carried out, if you like to allow the operation, if the processing result of step S ₅ is YES step and unnecessary S _7, S _8, it is sufficient to perform the steps S _9. In FIG. 33, all states after t ₃ are permitted. As a result, even if the user is unfamiliar with the operation of the pointer, it is possible to give an instruction that allows this. Note that, in step S _10, it is set to output the coordinates of the instruction signal disappears, so it is not possible to detect an instruction signal disappears, strictly a coordinates of the signal immediately before the instruction signal disappears.

Here, the above-described measure for improving the reliability of the detection of an instruction signal can be applied to any of a plurality of pointer types or a single pointer type. Further, the number of pointers may be plural or single, and may be any case. The same applies to the following embodiments.

FIG. 34 is a diagram showing another embodiment of a measure for improving the reliability of indication signal detection. The target of the description is the large screen display device 4. An invalid display area AR1 is provided on the display screen of the large screen display device 4 so that information is not displayed in the AR1 and information is displayed in the effective display area AR2 (a target range of the instruction). Is determined in advance. This is executed by the display control device 20 of FIG. By providing the invalid display area AR1 around the display screen of the display device as described above, the user does not need to point the invalid display area AR1 with a pointer. This A
Even if the instruction signal disappears in R1, the instruction may not be considered to be valid. If the AR1 is not present, as shown in FIG. 34, the instruction signal is moved to the upper right, and when the instruction signal protrudes from the large screen display device 4 and disappears, the coordinate of the instruction signal immediately before the disappearance disappears. I regard it as a position. On the other hand, this problem can be solved by handling the display screen of the large screen display device as shown in FIG. When an instruction signal is present in the invalid display area AR1, the marker is A
It is displayed in R1. For a panel display device such as a mimic display or an instruction from the operation / monitoring panel, the instruction target range is set to the effective display area AR2, and the others are the same as those in FIG. 34. This can be handled by setting AR1.

FIG. 35 is a plan view of FIG. 1 and FIG.
1 and the processing procedure executed by the image processing device 2 (or the display control device 20) in each embodiment of FIGS. Output information J ₁ in FIG. 35 is information indicating that it has been correctly instructed only when this information has been obtained, the display control unit 20 generates a new information to instruction pointers. Output information J ₂ is only indicate whether instruction signal is at any position, used for display of the marker.

In FIG. 35, an instruction signal is detected in step S ₁ , and the instruction signal is invalidated in step S ₂ in the invalid display area A.
It is determined whether it is within R1. This judgment result is YES
If, Top and executes step S _6.
That is, since the instruction signal exists in the invalid display area AR1, even if the instruction signal disappears in the invalid display area AR1, it is not regarded as valid. In step S _2, the processing result is NO, the process proceeds to step S _3. Here, if an instruction signal to the effective display area AR2, the process proceeds to step S _4, although the instruction signal has disappeared determines or not. If instruction signal disappears, the process proceeds to step S _5, and outputs the output information J _1, Top.
Otherwise all implemented the processing in step S _6, Top. Output information J ₁ in FIG. 35 is the same as that of FIG. 32, the output information J ₂ in FIG. 35 is the same as that of FIG. 32. This makes it possible to generate new information for the designated position or the designated object only when the designated signal disappears while pointing in the effective display area AR2, and the reliability of the detection of the designated signal is improved. Can be enhanced.

[0109] Further, in the processing of FIG. 35, only when the off instruction signal with double-click operation the pointer, it is also possible to as to output the output information J _1. That would be to combined processing of FIGS. 35 and 32, which comprises the steps S ₂ and subsequent processing steps in FIG. 32 (S ₂ ~
Step S ₄ subsequent processing steps in FIG. 35 a S ₁₀₎ (S
_{4 to} S ₆ ). As a result, it is possible to further enhance the reliability of the detection of the instruction signal.

In some cases, the instruction signal disappears while the instruction signal is being moved. The cause is that the instruction signal is intentionally turned off by the user's pointer operation,
There is a case where the instruction signal protrudes from the large screen display device or the like and disappears. The former is not a problem, but for the latter, the trajectory of the instruction signal is tracked, and if the instruction signal is predicted to protrude from a large screen display device or the like, even if the instruction signal disappears, Can be considered not valid.

The various processing methods described above are based on the actual operation of the pointer, and can increase the reliability of the detection of the instruction signal.

[0112]

According to the present invention, even when a plurality of pointers are used, it is possible to easily and reliably provide information on a display object designated for each pointer. For this reason, a plurality of users can appropriately obtain the information they need by using different pointers that output different instruction signals. In addition, even when a plurality of users simultaneously designate with the pointer, the information requested by each is displayed appropriately.

Further , since the display device on which the information is to be displayed is specified corresponding to the identification information of the pointer, each operator can use the information more easily.

[0114]

[0115]

According to the present invention, a plurality of persons can designate an instruction target without limiting the operation place.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an operation / monitoring device according to an embodiment of the present invention.

FIG. 2 is a detailed configuration diagram showing functions in the computer of FIG.

FIG. 3 is an explanatory diagram showing a processing procedure executed by the image processing apparatus of FIG. 1;

FIG. 4 is an explanatory diagram showing specific contents of processes 2c to 2e in FIG. 3;

FIG. 5 is a diagram illustrating a light emitting surface of various pointers.

FIG. 6 is a perspective view of an example of a pointer.

7 is an explanatory diagram of a laser light generation circuit of the pointer P in FIG.

8 is an explanatory diagram of an instruction signal output from a pointer P in FIG.

FIG. 9 is a perspective view of a pointer Pb of FIG. 1;

FIG. 10 is a configuration diagram of a light emitting source for outputting a cross-shaped instruction signal of the pointer of FIG. 9;

FIG. 11 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 12 is an explanatory diagram of a processing procedure executed by the object specifying means of FIG. 11;

13 is an explanatory diagram showing processing contents of an input sentence analysis executed by an input sentence analysis unit of the display control device of FIG. 11;

FIG. 14 is an explanatory diagram showing one processing content of display information creation executed by a display information creation unit of the display control device in FIG. 11;

FIG. 15 is an explanatory diagram of a screen display switching command.

FIG. 16 is an explanatory diagram illustrating a display example of information on a designated target object.

17 is a configuration diagram of a workstation provided with the functions of the coordinate conversion means, the object identification means, the command input analysis device, and the display control means of FIG.

FIG. 18 is a perspective view of another embodiment of the pointer.

19 is an explanatory diagram showing a configuration example of a light emitting surface of the pointer of FIG.

FIG. 20 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 21 is an explanatory diagram showing an example of information displayed on the display device when the core display section displayed on the large screen display device in FIG. 20 is pointed by a pointer.

FIG. 22 is an explanatory diagram showing an example in which new display information of FIG. 20 is displayed on a large screen display device.

23 shows a control rod drive system E in the display information shown in FIG.
FIG. 9 is an explanatory diagram illustrating an example of information displayed on a CRT display device when ₁ is selected.

FIG. 24 is an explanatory diagram showing a display example of a case where the instruction pointer alarm tile 8 _i in FIG. 20.

FIG. 25 is an explanatory diagram showing another display example when instructed by a pointer alarm tile 8 _i in FIG. 20.

FIG. 26 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 27 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 28 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 29 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 30 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 31 is a configuration diagram of another embodiment of the display control device having the allocation determination means.

FIG. 32 is an explanatory diagram of a processing procedure related to a double click applied in detecting a pointing target and a pointing point.

FIG. 33 is an explanatory diagram illustrating an output state of instruction information in response to a double-click operation of the pointer in FIG. 32;

FIG. 34 shows another embodiment of a measure for improving the reliability of indication information detection.

FIG. 35 shows another example of a processing method in which the reliability of detection is increased when detecting a pointing target or a pointing point.

[Explanation of symbols]

HD, HD _1- HD _K … Head mount display, P
a, Pb pointer, 1 monitor camera, 2, 2A image processing device, 2c feature extraction unit, 2d designated position detection unit, 2e pointer identification unit, 3, 40, 41 operation / monitoring panel, 4 , 4A, 4B ... large screen display device, 5a, 5b ...
Markers, 6A to 6G: CRT display device, 7: Electronic computer, 10: Coordinate conversion means, 11: Object specifying means, 15
... command input analyzer, 16A ₁ ~16A _N ... voice input device, 20 ... display controller, 21,37,65 ... memory,
25 ... information generating means, 27 ... identifying means, 31, 60 ... coordinate recognizing section, 32,61 ... object specifying section, 34 ... input sentence analyzing section, 36,63 ... display information creating section, 62 ... display information determining section, 180: designated point detecting means, AR1: invalid display area, AR2: valid display area.

──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Hiroyuki Yuchi 7-2-1, Omikacho, Hitachi City, Ibaraki Pref. Energy Research Laboratory, Hitachi, Ltd. (72) Inventor Hiroshi Seki 7-2 Omikacho, Hitachi City, Ibaraki Prefecture No. 1 Hitachi, Ltd. Energy Research Laboratory (72) Inventor Yukio Nagaoka 7-2, Omika-cho, Hitachi City, Ibaraki Pref. Hitachi, Ltd. Energy Research Laboratory (56) References JP-A-4-37922 (JP, A) JP-A-1-257935 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 3/033 310 G05B 23/02 301

Claims (11)

(57) [Claims] And 1. A display device, photographs the display device, the display and imaging means for outputting a video signal including an instruction signal from a plurality of pointers having reached the display surface of the device, on the basis of the said video signal Multiple pointers each
Identification means for identifying, and a plurality of display objects displayed on a display surface of the display device
Display information about each instruction signal display object designated by a and the object specifying means for identify the display object indicated by the instruction signal from the object
And display information output means for outputting the information to the display device corresponding to each of the identified pointers . 2. A display device, which outputs different instruction signals, respectively, and displays the same on the display device.
A plurality of pointers for pointing a surface by the instruction signal, and the display device is photographed, and reaches the display surface of the display device.
A video signal including an instruction signal from the plurality of pointers.
Shooting means for outputting, and each of the plurality of pointers based on the video signal.
Identification means for identifying, and a plurality of display objects displayed on a display surface of the display device
Identify the display target indicated by each instruction signal from the objects.
Object identification means to be displayed, and display information on the display object indicated by each instruction signal
To the display device corresponding to each of the identified pointers.
An information display device , comprising: display information output means for outputting the respective information. 3. A first display device, and a second display surface larger than the first display surface of the first display device.
A second display device having: a display surface of the second display device, the second display device being photographed;
Video signals including instruction signals from multiple pointers that have reached
And a plurality of pointers based on the video signal.
Identification means for identifying, and the identification means are displayed on the second display surface
Display indicated by each instruction signal from multiple display objects
Object specifying means for specifying the object, and display information on the display object specified by each instruction signal
To the display device corresponding to each of the identified pointers.
An information display device , comprising: display information output means for outputting the respective information. 4. The display information output means according to each instruction signal.
Display information relating to the indicated display object,
Output to separate display devices corresponding to each pointer
The information according to any one of claims 1 to 3, wherein
Display device. 5. The display information output means according to each of the instruction signals.
Of the display information for the indicated display object,
The finger input corresponding to the pointer that output the signal
Outputting display information corresponding to the indication.
The information display device according to any one of claims 1 to 4. 6. An instruction input corresponding to the pointer.
Is input by voice.
Information display device. 7. The display device according to claim 7 , wherein
Find the designated position on the display surface with each designated signal.
Means, and each indication signal based on these indication positions
Said object specifying means for specifying the indicated display object;
The information display device according to any one of claims 1 to 6, further comprising:
Place. 8. The image capturing means is an infrared camera.
The information display device according to any one of claims 1 to 7. 9. The display device photographed by the photographing means.
On the display surface of the
To display a marker indicating that each instruction signal indicates
Information display device of 請 Motomeko 8. 10. The display photographed by the photographing means.
Areas where information is not displayed in the peripheral area on the display surface of the device
The information display device according to any one of claims 1 to 9, further comprising:
Place. 11. A plurality of points reaching a display surface of a display device.
Based on the video signal including the instruction signal from the printer.
Identification means for respectively identifying number pointers, and a plurality of display objects displayed on the display surface of the display device
Identify the display target indicated by each instruction signal from the objects.
Object identification means to be displayed, and display information on the display object indicated by each instruction signal
A separate display device corresponding to each of the identified pointers.
And a display information output means for outputting the information to each of the devices.