JPH07200160A - Information providing device and information display device - Google Patents

Abstract

PURPOSE:To properly display information regarding a display object indicated with a pointer even when plural pointers are used. CONSTITUTION:Pointers Pa and Pb indicated the display screen of a large-screen display device 4 remotely. The display screen is photographed by a monitor camera 1. An image processor 2 extracts the feature quantities of pointers used for the indication on the basis of the obtained video signal and the positions of the points indicated with the pointers and the pointers are discriminated on the basis of the feature quantities. An electronic computer 7 generates the information regarding the display object at the position indicated with the corresponding pointer. The generated information is displayed on a display device predetermined with the pointer (CRT display devices 6A-6G or large- screen display device 4).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information providing apparatus and an information display apparatus, and more particularly to an information providing apparatus and an information display apparatus suitable for adapting to a system having a large screen display apparatus and a ginger surface display apparatus. .

[0002]

2. Description of the Related Art In recent years, a display device (including a mimic display device) having a large display surface is used in operation control of power plants and chemical plants, information provision at financial institutions and securities companies, and presentations at conferences and the like. The information provision used is planned or implemented. It is desired to be able to remotely instruct the information displayed on the large screen display device and to give the information to the instructed display target.

Techniques that can meet this demand include, for example, a pointing device device and an input method disclosed in Japanese Patent Laid-Open No. 3-179517 and a presentation device disclosed in Japanese Patent Laid-Open No. 4-37922.

In Japanese Unexamined Patent Publication (Kokai) No. 3-179517, a single infrared projector remotely indicates a display surface of a large-screen display device, and an indication point is based on an output signal of a monitor camera that monitors the display surface. Is detected and a marker indicating that the pointing point has been pointed to is displayed on the large screen display device. Further, it is also shown that an instruction to be input is processed with respect to the instructed display object by using a separately prepared instruction processing means.

In Japanese Laid-Open Patent Publication No. 4-37922, when an intended portion of a screen projected on a screen or a wall surface is pointed by a pointing rod equipped with a light emitter, an output signal of a television camera monitoring the screen is output. Based on the
It is shown to change the image information.

[0006]

In each of the above-mentioned conventional examples, there is only one pointer (infrared projector, pointing rod). It is not possible to identify if the part is indicated. Therefore, it is impossible to distinguish the pointers using a plurality of pointers and execute the instruction on the designated point or the designated object.

An object of the present invention is to provide an information providing device which can obtain corresponding information even when a plurality of pointers are used.

Another object of the present invention is to provide an information display device which can improve usability for obtaining information.

[0009]

The above-mentioned object is to provide a display device, a plurality of pointers that output different indication signals and indicate the display surface of the display device by the indication signals, and an identification means for identifying the pointers. , The display object displayed on the display surface, which is indicated by the pointer,
This is achieved by providing an object specifying unit for specifying and a unit for outputting the identification information of the pointer and the information obtained based on the display object.

Another object of the present invention is to output a different instruction signal from the display device, to provide a plurality of pointers for indicating the display surface of the display device with the instruction signal, and to shoot a video signal by photographing the display surface. A photographing means for outputting, an identification means for identifying the pointer pointing to the display surface based on the video signal, and a display object pointed to by the pointer and displayed on the display surface, Based on the object specifying means for specifying and the information of the specified display object, the display information determining means for determining the display information, the determined display information is created, and the display information is identified. It is created by outputting to the display device corresponding to the pointer.

[0011]

Since the pointer pointing the display surface of the display device can be identified and the pointing position can be detected, it is possible to distinguish the pointers by the plurality of pointers and remotely point the display surface. . Further, since it is possible to associate the designated point or the designated object with the designated pointer, it is possible to generate new display information, operation information, voice information, etc. based on these. In particular, it is possible to divide the generated information according to the purpose of use and the target of use by designating the range and the target of the generated information for each pointer in advance. That is, it is possible to separate the generated information for each pointer.

According to another feature of the present invention, since the display device for displaying information corresponding to the pointer is specified,
It can be displayed on the display device corresponding to the pointer based on the identification information of the point, and the usability for obtaining the information is improved.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An operation / monitoring device according to a preferred embodiment of the present invention will be described with reference to FIGS.

This embodiment comprises a monitor camera 1, an image processing device 2, an electronic computer 7, a large screen display device 4, CRT display devices 6A to 6G, and pointers Pa and Pb. C
The RT display devices 6A to 6G are provided on the operation monitoring panel 3. The operation monitoring panel 3 has the operation and monitoring functions required by the operator A (generally a plurality) to monitor the operation of the plant (not shown). Operating information required for each operator A is displayed on the CRT display devices 6A to 6G. The large screen display device 4 is arranged behind the operation and monitoring panel 3 and has a CR
It has a display surface larger than the display screens of the T display devices 6A to 6G.

The electronic computer 7 is a coordinate recognition unit 6 shown in FIG.
0, the object specifying unit 61, the display information determining unit 62, and the display information creating unit 63 are executed.

With such a configuration, the operator A can grasp the display information of the CRT display devices 6A to 6G provided on the operation monitoring panel 3 and the large screen display device 4
Information related to the entire plant displayed in can be grasped. Further, a person (hereinafter, referred to as an administrator, not shown) B who manages the entire plant can grasp the information displayed on the large screen display device 4. Therefore, the information grasp is improved, that is, the monitorability of the plant is improved.

The operator A uses the pointer Pa and the operator B
Uses the pointer Pb. By the remote operation from the pointer Pb, the information displayed on the large screen display device 4 can be changed and more detailed information on a part of the information displayed on the large screen display device 4 can be displayed on the large screen display device 4. It is displayed on a part of the screen. Further, by remote control from the pointer Pa, detailed information of the information displayed on the large screen display device 4 can be displayed on any of the CRT display devices 6A to 6G and the portable display device HD. This can be done by associating each pointer with various display devices 4, 6A to 6G, HD.

Which part of the large screen display device 4 is the pointer P
A monitor camera 1 and an image processing device 2 are provided to monitor whether or not the instruction is given by a or Pb. Further, in order to identify which pointer is pointing the large screen display device 4, the pointing signals output from the pointers Pa and Pb, that is, the emitted lights have different shapes. In FIG. 1, the pointer Pa emits dot-shaped light as an instruction signal, and the pointer Pb
Shows an example in which cross-shaped light is emitted as an instruction signal. A pointer P is displayed on the display screen of the large screen display device 4.
Markers 5a and 5b corresponding to the shape of the light emitted to the position indicated by a and Pb are displayed as described later. Therefore, the operator A and the administrator B can confirm the pointing position with the corresponding pointer by looking at the display screen of the large screen display device 4. The light emitted from the pointers Pa and Pb is an infrared laser.

First, the operation / monitoring apparatus of this embodiment when the administrator B emits a cross-shaped infrared laser beam, which is an instruction signal, from the pointer Pb to instruct on the display screen of the large screen display apparatus 4 The operation will be described below.

In the operation / monitoring apparatus of this embodiment, turning off the infrared laser beam output from the pointer is regarded as input of the instruction signal. The monitor camera (infrared camera) 1 photographs the display screen of the large screen display device 4. The image signal of the infrared light output from the monitor camera 1 includes the image of the cross-shaped infrared laser light that is the instruction signal output from the pointer Pb in the large screen display device 4, and the image processing device 2
Entered in. Here, the reason for using the infrared laser is that (1) the coordinates of the pointing point by the pointer recognized by the image processing apparatus 2 and the electronic computer 7 are different from the actual pointing point coordinates due to a recognition error. This is to correspond. In other words, by indicating the point recognized by the operation / monitor device with the cursor, the input accuracy of the instruction signal is improved. Further, (2) a pointer Pa that outputs infrared light,
If Pb and the monitor camera 1 which is an infrared camera are used, the image captured by the image processing apparatus 2 does not include visible light, and the processing in the image processing apparatus 2 can be performed easily and at high speed.

The image processing apparatus 2 executes the processing shown in FIG. An image signal of infrared light is input (process 2a). Binarization processing is performed on this video signal (processing 2b).
By binarizing, noise included in the video signal is removed. Next, the feature amount of the video signal is extracted using the binary image (process 2c). In this process, x of the binary image
The projection distribution on the axes and the y-axis is obtained, and the coordinates of the point where the density of the binary image exceeds the reference value L are obtained. The projection distribution shown in FIG. 4D is obtained for the pointer Pb, and the coordinates of the points beyond the reference point L are x ₁ , x ₂ , y ₁ and y ₂ . These coordinates x ₁ , x ₂ , y ₁ and y ₂ indicate the features of the binary image. In the process 2d, the pointing position of the pointer on the screen of the large screen display device 4 is obtained. That is, the coordinates x ₁ , x ₂ ,
Using y ₁ and y ₂ , the coordinates (X,
Y) is calculated (X = (x ₁ + x ₂ ) / 2, Y = (y ₁ + y ₂ ) /
2). The coordinates (X, Y) of this center position become the position designated by the pointer. Next, the type of pointer is determined (process 2
e). The determination of the type of the pointer is made by using the areas ΔS (= (x ₂ −x ₁ ) (y ₂ −) obtained by using the coordinates x ₁ , x ₂ , y ₁ and y _2.
y ₁ )) is determined. △ for Figure 4 (d)
Pointer P that outputs a cross-shaped instruction signal to S ₄
It is determined to be b. Information about the designated position of the pointer and the type of the pointer is output to the computer 7 (Process 2
f).

Information on the above-mentioned designated position of the pointer of the electronic computer 7 and the type of the pointer is inputted. Coordinate recognition unit 60
Is corrected by using a coordinate correspondence table stored in advance in the memory 65 with respect to the input coordinates (X, Y), so that the coordinates (X) of the actual pointing position on the display screen of the large screen display device 4 are corrected. _i , Y _i ) is obtained and output. By using the coordinate correspondence table, it is possible to correct the pointing position error caused by the distortion of the camera image and the influence of the camera position.
The coordinates (X _i , Y _i ) and the pointer type information are output to the display information creation unit 63, so that the coordinates (X _i , Y _i ) on the screen of the large screen display device 4 correspond to the pointer type information. The marker of the pointer (the cross-shaped marker 5b for the pointer Pb) is displayed. Since the instruction signals output from the pointers Pa and Pb are infrared rays, the manager and the operator cannot confirm where they are pointing as they are. However, in this embodiment, as described above,
Since the marker corresponding to the type of the pointer is displayed, the position pointed to by the user can be easily confirmed. The object identifying unit 61 identifies the object displayed at the position (on the screen of the large screen display device 4) pointed by the pointer Pb (or Pa). That is, the object specifying unit 61
Is used to retrieve the display information data currently displayed on the large-screen display device 4 from the memory 65 and input the coordinates (X _i , Y _i ) of the designated point, for example, JP-A-3-179517, page 3, lower part. The object displayed at the position of the coordinates (X _i , Y _i ) of the designated point is specified in the same manner as the processing shown in the right column, line 18 to page 4, upper left column, line 3 and FIG. Display information determination unit 6
2 includes the information of the specified object and the voice information of the manager B input from the headset microphone Mb (in the case of the operator A, input from the headset microphone Ma),
The information to be displayed is determined based on this combined information.
A command signal related to the information to be displayed is output to the display information creating unit 63. The display information determination unit 62, for example,
Japanese Laid-Open Patent Publication No. 3-179517, page 4, upper left column, line 15 to page 6, lower left column, line 3 (FIGS. 1, 5 to 7), and page 8 upper left column, line 5 to lower page left column, line 19 of the same page ( By the processing described in FIG. 11, FIG. 13 and FIG. 14), the information to be displayed is determined using the information of the specified target and the voice information of the manager B (or the operator A).

The display information creating section 63 is provided with the display information determining section 6
The display information corresponding to the display information to be displayed determined in 2 is created using the information in the memory 65. When creating this display information, pointer type information is also taken into consideration. The pointer type information is input to the display information creating unit 63 via the coordinate recognizing unit 60, the object specifying unit 61, and the display information determining unit 62. The display information creation unit 63, for example,
The display information regarding the object pointed by the pointer Pb is created based on the data stored in the memory 65 corresponding to the type information of the pointer. The created information is output to and displayed on a display device that is defined according to the type of pointer. Display information related to the object pointed by the pointer Pb is displayed on the large screen display device 4.

Even when the operator A indicates the object displayed on the large screen display device 4 by the instruction signal (one dot) output from the pointer Pa, the image is displayed in the same manner as when the pointer Pb is used. Processor 2 and computer 7
The process is executed in. In 2d and 2e of the image processing apparatus 2, the coordinates (X, Y) of the center position are calculated using the coordinates x ₁ , x ₂ , y ₁ and y ₂ of FIG. The type of pointer (pointer Pa) is determined from ΔS ₁ .
The coordinate recognizing unit 60 corrects the position of the coordinate (X, Y) with respect to the instruction signal output from the pointer Pa to obtain the coordinate (X _i ,
Y _i ). The display information creating unit 63 displays the 1-dot marker 5a on the large screen display device 4 using this coordinate value. The object specifying unit 61 and the display information determining unit 62 perform the same processing as in the case of the pointer Pb. Display information creation unit 63
Creates display information relating to the object pointed by the pointer Pa based on the data stored in the memory 65 corresponding to the type information of the pointer. The created information is displayed on, for example, the CRT display device 6A corresponding to the pointer Pa.

As described above, according to the present embodiment, since it is possible to identify a plurality of pointers which point the display screen of the large screen display device 4, it is possible for a plurality of persons to make a large screen display device 4.
When the display screen is designated, the corresponding display information can be created and displayed on the display device. In addition, a plurality of people can simultaneously use the respective pointers to display the large screen display device 4
Even when the display screen is designated, the information required by each can be appropriately created and displayed.

Since the display device which should display the information corresponding to the pointer is specified, the information on the object pointed by the manager B with the pointer Pb (the information which should be seen by all) is displayed on the large screen display device 4. Information about the object pointed to by the operator A with the pointer Pa (information that the individual operator A should see) is C
It can be displayed on the RT display device 6A. In this way, since the display device for displaying information is specified according to the type of pointer, the usability of the information of the operator A and the manager B is improved. The operator A can also access the information by the touch phase difference of the CRT display device 6A. In particular, when the operator A points the large screen display device 4 with the pointer Pa, the information related to the information displayed on the large screen display device 4 can be displayed on the CRT display device 6A that is the monitoring target of the operator A. This is possible by identifying the pointer.

Furthermore, a head mount display HD
("Newton", Vol. 11, No. 12, P104
~ 107, 1991, KYOIKUSHA), the operator A points the large-screen display device 4 with the pointer Pa while moving, and the personal information necessary for the operator A. Can be displayed on the head mount display HD.

Since the operator A and the administrator B point the display screen remotely using the pointer, the operator can point the display screen of the large screen display device 4 regardless of the positions of the operator A and the administrator B. You can Further, regardless of the positions of the operator A and the manager B, the detection of the designated position and the determination of the type of the pointer are accurately performed.

According to this embodiment, it is possible to identify a plurality of pointers that output different pointing signals, and appropriately create and display information about the pointing object pointed by the identified pointers. Therefore, a plurality of users can appropriately obtain the information they need by using different pointers that output different instruction signals. In addition, even when a plurality of users point at the same time with the pointer, the information requested by each of them is properly displayed.

In the present embodiment, a plurality of persons can designate an instruction target without limiting the operation place, and information for each person or common information can be displayed on different display devices or common display devices. Can be output,
It is effective when applied to the information provided in the central control room of a plant operated by multiple operators, the information provided at financial institutions and securities companies, and the presentations at conferences, etc., and also the work efficiency and work reliability. It is possible to further increase it, and its industrial value is extremely high.

Further, since the information to be displayed is determined by combining the information of the display object designated by the pointer and the voice information, the upper input of the pointer and the voice input means is not restricted, and they can be freely combined. Can be used. Therefore, it is possible to reduce the burden on the operator or the like for inputting information using the pointer and the voice input means.

Even when pointing is made by using a 2-dot type pointer and a 3-dot type pointer (emits infrared laser light), the image signals of infrared light output from the monitor camera 1 are used to generate the images shown in FIGS. The characteristics as shown in c) can be obtained in the process 2c. When a plurality of peaks appear in the projected distribution, the maximum and minimum values on the x-axis and y-axis are used to calculate the center coordinates and the area ΔS of the circumscribed quadrangle. For example, in a 2-dot type pointer, x ₁ and x ₂ are calculated for the projection distribution on the x axis, and y ₁ and y ₂ are calculated for the projection distribution on the y axis to determine the center coordinates and the area of the circumscribed rectangle. Ask for. In addition, when the center coordinates are not obtained, the minimum value of the projection distribution of the x-axis and the y-axis, for example, in FIG.
The (x ₁ , y ₁ ) coordinates in (b) may be obtained.

The distinction of the pointer used to indicate the position is 2
This is performed using data such as the number of high-luminance points in the value image or the area of the circumscribed quadrangle of the high-luminance region.
Alternatively, the geometrical shape of the pointing signal by the pointer is used, this shape is stored in advance in the memory (not shown) in the image processing apparatus 2, and the same shape is detected by the template matching method. Good. When the number of high-brightness points is obtained to distinguish the pointers, the dot-type pointers shown in FIGS. 4A to 4C are used, and the projection distribution after binarization of the x-axis or the y-axis is used. In, the judgment is made by the maximum value of the number of peaks. For example, in FIG.
In the 3-dot type pointer of (c), the number of peaks on the x-axis is the maximum and is three, so it is determined to be a 3-dot type. By the way, in the 4-dot and 5-dot type pointers shown in FIG.
In some cases, the maximum number of peaks of the projection distribution on the axes and the y-axis does not match the number of light emitting sources. With respect to the pointer, the feature of the binary image is extracted by another reference value L ₂ having a higher level than the reference value L ₁ . That is, it is possible to sum the number of peaks of L ₁ or more and the number of peaks of L ₂ or more to obtain the larger one of the x-axis and the y-axis, and identify the pointer by the number. For example, on the y-axis in FIG. 4 (c), the number of peaks is 2, but the number of peaks of L ₁ or more is 2, and the number of peaks of L ₂ or more is 1, so the total is 3
Becomes On the x-axis, there are three peaks of L _{1 and} above, and L ₂
Since the above is zero, the total is 3. Therefore, it can be determined that the pointer is designated by the 3-dot type pointer.

Here, an example of the structure of the pointer will be described. 6 and 7 show a pointer P having three dots. Kj is a DC power supply, and Ke is a protection resistor for limiting current. The pointer P applies a voltage to the drive circuits Kd ₁ , Kd ₂ , Kd ₃ and the light emitting diode K ₄ by turning on the switch K ₅ . Therefore, the infrared laser light is output from each of the semiconductor lasers Kb ₁ , Kb ₂ , and Kb ₃ . These infrared laser lights are collimated by the collimating lenses Ka ₁ , Ka ₂ , and Ka ₃ into parallel lights Kc ₁ and Kc.
c ₂ and Kc ₃ . Each infrared laser light without spread,
The light is emitted from the light emission sources K ₁ , K ₂ , and K ₃ on the emission surface K ₆ . As a result, the infrared laser light (instruction signal) output from the pointer P indicates the display screen of the large screen cover device 4, as shown in FIG. By the functions of the coordinate recognition unit 60 and the display information creation unit 63, three points 5
Markers having A, 5B and 5C are displayed on the display screen of the large screen display device 4. When the switch K ₅ is turned off, the emission of the infrared laser light from the pointer P is stopped. The light emitting diode K ₄ indicates that light is being emitted from the light emitting sources K ₁ , K ₂ , and K ₃ , and when far-infrared light (for example, infrared light) is output from the light emitting source, which is invisible to the eye. Especially effective.

The structure of the pointer Pb used in the embodiment of FIG. 1 is shown in FIGS. 9 and 10. The pointer Pb has two light emission sources Kh and Ki. The light source Kh is a semiconductor laser
Kj ₁ , a focusing lens Km ₁ , a collimating lens Kn ₁ and a slit Kr ₁ are provided. The light emission source Ki also includes a semiconductor laser Kj ₂ , a focusing lens Km ₂ , a collimating lens Kn ₂ and a slit Kr ₂ . The infrared laser light on the horizontal axis of the marker 5b is called X-axis output light, and the infrared laser light on its vertical axis is called Y-axis output light. Slit K
The infrared laser light emitted from r ₁ is the X-axis output light Ks ₁ , and the infrared laser light emitted from the slit Kr ₂ is the Y-axis output light Ks ₂ . The X-axis output light Ks ₁ and the Y-axis output light Ks ₂ are rod-shaped emitted lights, respectively, and are focused so as to form a cross shape on the display surface of the large screen display device 4. For this purpose, focus lenses Km ₁ and Km ₂ are provided between the semiconductor laser and the collimator lens. Even if the distance between the pointer Pb and the large screen display device 4 changes,
The X-axis output light Ks ₁ and the Y-axis output light Ks ₂ do not intersect each other at the center point, but intersect at a position that is displaced. In this way, a cross-shaped instruction signal is obtained. The slits Kr ₁ and Kr ₂ spread the light so that the X-axis output light Ks ₁ and the Y-axis output light Ks ₂ become more rod-shaped so that the width of the portion perpendicular to the axial direction becomes a constant value. It is provided to block the

By combining a plurality of rod-shaped infrared laser beams in this way, it is possible to obtain a pointer for outputting an instruction signal such as a parallel rod, a single rod or a T-shaped rod.

An operating / monitoring device according to another embodiment of the present invention will be described with reference to FIG. The same components as those in the above-described embodiment are designated by the same reference numerals. In this embodiment, the pointers Pa and P
b, monitor camera 1, image processing device 2, coordinate conversion means 1
0, the object specifying means 11, the voice input device 16A ₁ ~ 16
A _N (the headset microphones Ma and Mb in FIG. 1 are examples),
A command input analysis device 15, an operation / instruction device 9, an information generating means 25, a first display device 26, and a large screen display device 4 which is a second display device are provided. The information generating means 25 includes a command output means 12, control devices 13A _{1 to} 13A _j , and a display control device 2.
0, a memory 21, and audio output devices 24 and 24A. The first display unit 26 is provided with a CRT display device 6A~6G and head-mounted display HD ₁ ~HD _k.

The pointer Pa is operated by the operator A, and the pointer Pb is operated by the administrator B to output an instruction signal similar to the embodiment of FIG. 1 to the display screen of the large screen display device 4. The image processing device 2 takes in a video signal of the display screen of the large screen display device 4 captured by the monitor camera 1 and executes processes 2a to 2f shown in FIG.

By the binarization in the process 2b, the noise contained in the video signal is removed. In the process 2d, the central coordinates of the binary image are obtained as the pointing position of the pointer, because the shape of the pointing signal imaged on the large screen display device 4 is deformed due to the different pointed position by the pointer. This is to prevent the designated position and the position of the designation signal detected on the display screen of the large screen display device 4 from shifting. However, if the pointed position on the display screen is deviated and the operator (or the administrator) points the pointer again, it is not necessary to find the center coordinates of the binary image. The coordinates of the specific position of the instruction signal may be obtained. The image processing device 2 outputs the type of the pointer and the coordinates (X, Y) of the point designated by the pointer to the coordinate conversion means 10.

The coordinate conversion means 10 is a coordinate recognition section 6 of FIG.
The same processing as 0 is executed, the input coordinates (X, Y) are corrected, and the coordinates (X _i , Y _i ) of the actual designated position on the display screen of the large screen display device 4 are obtained and output. This correction is carried out using the coordinate correspondence table, but for example, the infrared light emission sources are embedded in the four corners of the large screen display device 4 and are made to emit light, so that the above coordinate correspondence can be achieved. The infrared rays are effective because they are invisible to the user. In addition, the large screen display device 4 itself outputs information of a specific pattern (for example, whitening only the four corners of the screen and blacking the other), and the monitor camera 1 is temporarily output only when this information is output.
It is also possible to create coordinate conversion information by increasing the sensitivity of the so that the specific pattern can be recognized.

The object specifying means 11 is based on the coordinates (X _i , Y _i ) in the large screen display device 4 obtained by the coordinate converting means 10.
The display target on the display screen specified by the operator etc. is specified. This identification is performed by the processing shown in FIG. The coordinates (X _i , Y _i ) of the designated position in the large screen display device 4 before turning off the infrared laser light output from the pointer are read (process 50). Using this coordinate, the designated target window is determined (process 51). Next, the relative coordinates (Xw, Yw) in the target window are calculated (process 52). Using the corresponding data of the designated area / character string for the figure being displayed in the target window, the relative coordinates (X
(w, Yw), that is, the designated display object is determined (process 53). The designated area / character string correspondence data is prepared for each drawing in the display screen data stored in the memory 21. The contents are data regarding the range of the designated area and the output character string. When the display target pointed by the pointer is determined, the character string corresponding to this display target is output. The information of the character string corresponding to the specified display object is displayed by the display control device 2
It is input to 0 and stored in the memory 21. The display controller 20 transmits information between the memory 21 and the object specifying means 11.

Since the emitted lights of the pointers Pa and Pb are infrared rays, the operator cannot confirm where the point he is pointing is. Therefore, the display control device 20 takes in the relative coordinates (X _W , X _W ) obtained by the object identifying means 11 and the type information of the pointer obtained in the process 2e of the image processing device 2, and pre-determines the type of the pointer. Using the information of the marker associated with, the marker corresponding to the pointer outputting the instruction signal is displayed on the large screen display device 4 at the position of the coordinates (X _W , X _W ). As a result, a marker corresponding to the pointer can be displayed at the point designated by the pointer. Here, the display control device 20 uses the object specifying means 11
Therefore, instead of taking in the relative coordinates (X _W , X _W ), a character string specifying the object may be taken in. In this case, it is possible to display the marker at the coordinate of the large screen display device 4 in the same manner as described above by determining the coordinate of the object indicated by the character string. In this case, since the display target has been specified, the display target displayed on the large-screen display device 4 is blinked or the display color of the target is changed to display the designated point. It is possible to In this case, the blink cycle and display color associated with the instruction signal can be used.

The input of the operator A is performed by voice information or an instruction using the pointer Pa. The voice information is input to the command input analysis device 15 via the voice input devices 16A _{1 to} 16A _N , recognized, and then input to the display control device 20 as the voice recognition information 17. Character string information corresponding to the display target specified by the target specifying unit 11 is also input to the display control device 20. Display controller 20
Treats all of these input information as natural language words, stores them in the memory 21, and displays them on the large screen display device 4. For example, it is displayed in the lower frame of the large screen display device 4 shown in FIG. The operator or the like can monitor the input information recognized by the operation / monitor device of this embodiment by looking at the information displayed in the frame.

When the input of the voice information from the operator or the like is finished, the display control device 20 integrates the words of the voice information and the character string information of the display object stored in the memory 21 into one sentence, Next, this sentence is analyzed, that is, the input sentence is analyzed. The display control device 20 has an input sentence analysis unit (not shown) inside.

The input sentence analysis performed by this input sentence analysis unit is performed by the process described in Japanese Patent Laid-Open No. 3-179517 (described in the embodiment of FIG. 1). FIG. 13 shows an outline of the processing for the input sentence “show reactor water level trend”. In this input sentence, "nuclear reactor" is the character string information of the display object at the position pointed by the pointer, and "show water level trend" is voice input information. First, referring to the grammar and the dictionary, the input sentence is divided into words and a meaning and a meaning category are added (process 38). Here, the meaning category is a category of meanings of words used in the target world. In the example of FIG. 13, the system and the state quantity correspond to that. Next, the meaning of the input sentence is analyzed using this result (process 39).

The semantic analysis is carried out in two steps, as shown in FIG. First, the type of request content is identified (). A standard sentence pattern is prepared for each type of request for identification. These are described in the form of rules using meanings or meaning categories. The type of request is identified by matching the meaning of the sentence pattern or the meaning category of the input sentence. In the example shown,
"<Trend It is identified that the type of request is the trend presentation request “trend” by matching “graph〉 □ Present” ”, where the symbol“ 〈〉 ”uses the semantic category for matching,“ □ ” Indicates that any number of words may be included.

Next, the information necessary for executing the request is acquired.
(). Necessary information and its acquisition method are prepared as frame-type knowledge. Information is acquired using this knowledge. In the example of FIG. 13, both the system device name and the state quantity name, which are necessary information, are obtained from the input sentence. If the necessary information is not included in the input sentence, the information is acquired by using plant-related knowledge, dialogue history, or inquiring to the operator. When the meaning analysis is completed, it is output as a processing instruction (processing 40). The content of information to be displayed is output as a processing instruction. Such an input sentence analysis unit corresponds to the display information determination unit in the embodiment of FIG.

The display control device 20 has a display information creating section (not shown) inside. The display information creation unit creates display information based on the data of the information to be displayed obtained by the input sentence analysis unit, and displays it on the display device corresponding to the type of pointer specified by the image processing device 2. Output information. The created display information is created using the information corresponding to the type of the pointer (information necessary to create the display information) read from the memory 21. FIG. 14 shows an example of processing in the display information creation unit, taking creation of display information in a system diagram as an example. In the case of the display information creation command of the system diagram, the system device name is obtained as necessary information by the processing of the input sentence analysis unit. The corresponding display diagram is determined based on the system device name. First, the system diagram constituent element database configured in a part of the storage area in the memory 21 is searched based on the given system device name (process 41). This database gives the name of a drawing and the names of components such as devices included in the drawing. The drawing name and the system device name are compared (process 42), and if there is a match, a display command for the system diagram is output (process 47). This is a process corresponding to the case where the operator stores the drawing name and inputs it as it is. As a result of the search, when it is determined in step 42 that there is no corresponding drawing, the drawing component name stored in the database is searched based on the input system device name. (Processing 43). As a result of this search, when it is determined that there is no corresponding element name (process 44), the fact that there is no related drawing is output (process 48), and the operator is requested to re-input. On the other hand, when the element name is found and the number of drawing names including the element is one (process 45), a display command for the systematic diagram is output (process 49). On the other hand, if there are two or more drawing names, a drawing selection process is performed.

The information presentation command selected as a result of this processing is also input to the audio output device 24 and output as audio information 23. The large screen display device 4 and the CRT display device 6 are also provided.
It is used to control the display of the screens of A to 6G and head mount displays HD _{1 to} HD _K. FIG. 15 shows the types of screen display switching commands stored in the memory 21 as an example. The screen display is switched according to these processing instructions. Which display device the display is switched to is determined based on the type information of the pointer obtained by the image processing device 2. That is, the type of pointer (pointer Pa,
Pb, etc.) are assigned to display devices, and the display device is determined according to the type of pointer used to instruct the display screen of the large screen display device 4. As a result, the display information 26 created by the display information creating unit of the display control device 20 can be displayed on the display device defined corresponding to the pointer.

In the processing items of FIG. 15, the systematic diagram display means to display the systematic diagram, the trend diagram display means to display the trend diagram, and the window deletion means to erase the displayed window. is there. Moving a window is to move the displayed window, window pop is to display the displayed window in front of the display surface, and window push is to display the window being displayed on the display surface. It is to be displayed on the back.

FIG. 16 shows the concept of the display-related operation of this embodiment when the CRT display devices 6A to 6G and the large screen display device 4 are used. For example, in FIG.
In the large screen display device 4 described in the center of the above, the operator A instructs the water supply pump portion of the overall system diagram of the plant displayed on the large screen display device 4 by the instruction signal output from the pointer Pa, and When the operator A inputs “show system diagram” by voice, as shown in (c),
A system diagram near the water supply pump is displayed on the CRT display device 6A. The same signal is indicated by the instruction signal of the pointer Pa,
When the operator A voice-inputs "flow trend display", a trend diagram of the water supply pump flow rate is displayed on the CRT display device 6A as shown in (d). The administrator B instructs the water supply pump portion of the entire system diagram of the plant displayed on the large screen display device 4 by the instruction signal output from the pointer Pb, and the administrator B inputs "show system diagram" by voice. In this case, as shown in (a), a system diagram near the water supply pump is displayed on the large screen display device 4. When the same signal is indicated by the instruction signal of the pointer Pb, and the administrator B inputs a voice message "flow trend display", a trend diagram of the water supply pump flow rate is displayed on the large screen display device 4 as shown in (b). To be done. Such a display is possible because the pointing signal of the pointer Pb has a cross shape and the pointing signal of the pointer Pa has a dot shape. These instruction signals can be identified, and information is output to the display device corresponding to the pointer as described above based on the identified result. This is also done in the embodiment of FIG.

According to this embodiment, the same effect as that of the embodiment of FIG. 1 can be obtained.

The display information 26 created based on the display object designated by the image processing device 2 and the voice input information is designated by the pointers Pa, Pb.
It is possible to display on a display device that is determined corresponding to the instruction signal output from Pb. For this reason, it is possible to remotely instruct the information required by each operator and the information that all operators (including managers) must know. It is possible to present information with high man-machine characteristics. In addition, since it is possible to instruct the display target with each of a plurality of pointers, it is not necessary to shift the timing of use between the users, and there is an effect of high utilization efficiency.

The present embodiment not only outputs the display information 26 to the first display device 26 and the large screen display device 4, but also
As follows, it is possible to output the control information 14A ₁ to 14A _j for controlling the controlled object of the plant.

When the command output means 12 receives the operation command information 18 output from the display control device 20, the command output means 12 determines which of the control devices 13A _{1 to} 13A _j the control command signal is to be output based on this. Then, the control command signal is output to the corresponding control device.

The operation command information 18 is created as follows. The pointer Pa points to one device in the system diagram displayed on the large screen display device 4. When the output signal indicating the control command is input from the operation / instruction device 9 by the operation of the operator A, the display control device 20 receives the display object specified by the image processing device 2 as described above (the above-mentioned instruction is given). Device) and voice information of the operator A are integrated and output as operation command information 18.

Each of the control devices 13A _{1 to} 13A _j performs a control calculation based on the control command signal output from the command output means 12 and the output signal from the operation / instruction device 9 and the corresponding controlled object. The operation information 14A _{1 to} 14A _j is output for the object. For example, it is assumed that the display control device 20 outputs, as an example, the operation command information 18 “motor A, rotation speed 1000 rpm” and the command output means 12 inputs the operation command information 18.
The command output means 12 performs the same input sentence analysis and semantic analysis on the input information as the display control device 20, determines the control device (for example, the control device 13A ₁ ) that controls the motor A, and determines this control device. A setting signal corresponding to 1000 rpm, which is the target value of the rotation speed, is output to 13A ₁ . The control device 13A ₁ takes in a signal output from the operation / instruction device 9 in response to the setting signal (a signal indicating the number of revolutions of the motor A which is the feedback signal 19 from the plant), and sets the setting signal and the motor. The control information (control signal) is output so that the signals indicating the number of revolutions of A match.

The command output means 12 determines whether the type information of the pointer output from the display control device 20 corresponds to the control device outputting the control information, and if they do not correspond in advance, then The operation command information 18 is not executed. In this way, the range of control functions for the pointer that outputs an instruction signal to the large screen display device 2 can be limited, so that the reliability of plant operation can be improved.

In the embodiment of FIG. 11, the coordinate conversion means 1
0, object specifying means 11, display control device 20, memory 2
1 and the command input analysis device 15 can be realized by a workstation. FIG. 17 shows a configuration example of the display information generating function realized by the workstation. The workstation 250 includes a coordinate recognition unit 31, an object identification unit 32, an input control unit 33, an input sentence analysis unit 34,
It includes a display information creation unit 36 and a memory 37 that stores display screen data.

The coordinate recognizing section 31 has the same function as the coordinate recognizing section 60 and obtains the coordinates (X _i , X _i ) of the actual designated point on the display screen of the large screen display device 4. The object specifying unit 32
In the process of FIG. 12, the coordinates (X _W , X _W ) are obtained, the display object pointed by the pointer is determined, and the character string information corresponding to this display object is output. The input control unit 33 combines the character string information and the voice information input from the voice input device into one sentence of a natural language, and outputs this sentence to the input sentence analysis unit 34. The input sentence analysis unit 34 is the same as the embodiment of FIG. 1 in Japanese Patent Laid-Open No. 3-179517. The process described in the 19th line, for example, the process shown in FIG. 13 is performed. The input control unit 33 and the input sentence analysis unit 34 configure the display information determination unit in the embodiments of FIGS. 1 and 11. The display information creating unit 36 has the same function as the display information creating unit in the embodiments of FIGS. 1 and 11. With such a configuration, these functions can be realized by software processing.

In the above description, the output from the pointer is infrared light, but it may be visible light. For example, in each type of pointer of FIGS. 5, 6 and 9,
The red light, the blue light, and the green light can be individually output, or the red, blue, and green lights can be combined and output. As a result, even with a pointer that outputs an instruction signal of the same shape, it is possible to increase the type of instruction signal, that is, the type of pointer, simply by changing the color of the emitted light.

For this purpose, the monitor camera 1 is for color, and the image processing device 2 is for R (red), G (green), B.
Each (blue) frame memory must be provided.
Further, the image processing apparatus 2 obtains the projection distribution for each of R, G, and B with respect to the video information loaded in each frame memory, obtains the coordinates of the designated position based on these results, and determines the pointer type. Identify. For example, the projection distribution after the binarization processing results only in the processing for the data of the R frame memory in the instruction signal in red, and in the processing for the data in the G frame memory in the instruction signal in green. Appears. Therefore, by determining which frame memory data the result appears in, it is possible to easily distinguish whether the colors are different for the instruction signals of the same shape. Therefore, this information may be used to identify the pointer.

When the emitted light of the pointer is infrared light, the pointer can be identified even if the wavelength of the emitted light is made different for each pointer. In the case of visible light, the color changes. By changing the wavelengths of infrared light emitted from a plurality of pointers and installing a plurality of monitor cameras equipped with infrared filters corresponding to the wavelengths of infrared light to be monitored, Only infrared light (instruction signal) that is the light emitted from the corresponding pointer can be detected. Thereby, it is possible to use only one light source for the pointer. Further, even if the instruction signals output by the respective pointers indicate the same location on the display screen, the image processing apparatus can detect the instruction signals from the respective pointers by distinction.

FIG. 18 shows a pointer capable of emitting both infrared light and visible light. The two points 5A and 5B on the display screen of the large screen display device 4 are infrared light, and one point 5D
Is visible light. This is because the visible light indicates where the pointer P ₁ is pointing. K6 is a visible light source, and can be configured in the same manner as in FIG. 7 using, for example, a semiconductor laser that emits visible light.

The pointer P ₁ has a great effect when it is applied when the above-mentioned marker cannot be displayed on the display device. For example, the mimic display device shows a systematic diagram in a mosaic on the panel, for example, when the state of the valve changes, a colored lamp is turned on to show the state, or an indicator or the like is embedded in the mosaic place, and the state of the place is displayed. Quantity (eg,
The water level) is displayed on the indicator. In such a display device, the marker cannot be presented on the display surface. For example, the pointer is used to point to the above-mentioned indicator, and detailed information and related information relating to the terminal can be displayed in the individual C
When trying to output to the RT display device, it is necessary to check whether or not the position of the instruction is the intended place. In such a case, if the pointer P ₁ is used, the user can see the visible light, so that the instruction can be given reliably and easily.

Further, by presenting the instructed object using the voice information, it is possible to confirm that the instructed position is the object intended by the user, so that a more reliable instruction can be realized. In this case, the specified target object is converted into voice information based on the result obtained by the target object specifying means 11 as described in the embodiment of FIG. It can be provided as information 23.

Examples of presenting information using the mimic display device include plant monitoring in a central control room of a nuclear power plant, a thermal power plant, a steel plant, a chemical plant, etc., and a power system monitoring showing a power transmission path, a voltage, and a tidal current. , Monitoring train operation status in railways, presenting stock price information in finance and securities, etc.

If the above-mentioned pointer P ₁ is used, it is also possible to point the switch or the indicator of the operation / monitor panel and present the information related thereto.

Also in these cases, it is necessary to judge which position of the mimic display device and the operation / monitor board the instruction signal indicates. However, it is possible to identify the instructed object by providing infrared light emission sources at the four corners of the mimic display device and operation / monitor panel and storing in advance what the relative distance from these positions is. Is. The pointer shown in FIG. 19 may be used as the pointer. They all have a visible light source. Of FIG.
In (a) to (e), one light source that emits red light is provided, and a green light source is arranged in the periphery. This is because when using a plurality of pointers, each user distinguishes each pointer with a green instruction signal and confirms the indicated position with a red instruction signal. 19 (f) to (j)
On the contrary, the respective pointers are distinguished by the red pointing signal and the pointing position is confirmed by the green pointing signal. In this way, the number of identifiable pointers can be increased by using one optical information for indicating the pointing point and by using a plurality of optical information for distinguishing the pointers. The image processing device may handle light emitted from all of these light sources as an instruction signal. That is, this pointer enables the user to visually check the designated point.

This pointer is used particularly for instructing information displayed on the screen, for instructing panel display information, etc., and is particularly effective when it is desired to let another user grasp the instructed point of one user. is there. Such applications include meetings, explanations, meetings, and presentations.

FIG. 20 shows the configuration of an operation / monitor device having a large screen display device according to another embodiment of the present invention. Although this embodiment is not shown, the image processing apparatus 2, the coordinate conversion means 10, the object specifying means 11, the voice input devices 16A _{1 to} 16A _N , the command input analysis device 15, of the embodiment of FIG.
The information generating means 25 is provided. 4A and 4B are large-screen display devices, and 40 and 41 are backup boards (so-called operation / monitor boards) that can be operated in connection with plant operation.
Is. The backup panels 40 and 41 are provided with system-specific alarm display devices 8 on their upper portions. The display device 8 has an alarm tile 8 _i on which an alarm item is marked, and when displaying an alarm, a lamp attached to the alarm tile 8 _i is turned on. Therefore, it is possible to know in which system the alarm has occurred. However, which of the plurality of alarm items has occurred in the system in which the alarm has occurred cannot be confirmed only by this system-based alarm display device 8, and the content can be displayed in the individual CRT display devices 6A to 6G. Will be displayed. The backup boards 40 and 41
Is further equipped with a device status display lamp, a CRT display device, an EL display, operation switches, a recorder and the like. This part is the panel display section. Note that the infrared light emission source 40A ₁
40A to 40A ₄ are for making the coordinates of the indicated position of the instruction signal photographed by the monitor camera 1 correspond to the pointing object or the position on the board surface of the backup board 40 when the board surface of the backup board 40 is indicated by the pointer. Used for. Other infrared-emitting source _{41A 1 ~41A 4, 4A 1 ~4A} 4, 4B 1
The same applies to ~ 4B ₄ .

Now, the CRT display devices 6A to 6G provided on the operation monitoring panel 3 display information necessary for the operators A ₁ and A ₂ in plant operation monitoring. The display data on the CRT display devices 6A to 6G are created based on the information from the plant and the results of the operators A _{1 to} A ₂ operating the operating means 7A to 7C, respectively.

On the other hand, the information displayed on the large-screen display devices 4A and 4B is monitored not only by the operator but also by the manager B who monitors the plant. The configuration is such that the important parameters and the status of important equipment are displayed.The message and the status of the important equipment change according to the operating status in the entire plant configuration, and the values of the important parameters sometimes change. It is displayed moment by moment. Furthermore, the status of the entire plant is monitored using the two large screen display devices 4A and 4B.

Next, an example of specific instructions and displays will be described in detail. The example is for a nuclear plant.

An example in which the operator A ₁ uses the large screen display device 4A to output information relating to control rod operation to the CRT display device 6E will be described.

First, for example, the pointer Pa _{1 is used} to indicate the core display portion (the portion indicated by the instruction signal 5a). As a result, the image processing device 2 recognizes the pointing target and the pointer used for pointing. As a result, FIG.
The information indicated by is displayed on the CRT display device 6E corresponding to the type of pointer. The display processing is executed by the display control device 20 shown in FIG. 11. Large screen display device 4A
When the information of FIG. 21 is displayed in FIG. 21, the already displayed information is not erased and newly displayed, but new displayed information E is displayed on top of the already displayed information as shown in FIG.
It is desirable to display. This is because, according to this display method, it is possible to grasp the outline of the information related to the entire plant. The new display information E shown in FIG. 22 may be displayed on the CRT display device 6E instead of the information already displayed, or may be displayed in an overlapped manner like the large screen display device 4A. Further, if there is a margin on the display screens of the large screen display device 4A and the CRT display device 6E, new display information E may be displayed there.

As shown in FIG. 22, when the new display information E is displayed on the large screen display device 4A, the pointer Pa ₁ is used to indicate a necessary portion of the display information E to select the information.
In this case, since information related to control rod operation is required, by pointing the control rod drive system E ₁ out of this display information, for example, information related to control rod operation shown in FIG. 23 is displayed on the CRT display device 6E. Can be displayed. In this case, since the information E shown in FIG. 22 is unnecessary, the display control device 20 performs a process for erasing the information E displayed on the large screen display device 4A. When the information shown in FIG. 23 is displayed on the CRT display device 6E, instead of pointing the control rod drive system E ₁ of FIG. 22 with the pointer Pa ₁ , the headset microphone Ma ₁ is used for “control It is also possible to input "drive rod system, operation information, show" by voice. FIG. 22
In the above example, the new display information is displayed on the large screen display device, but whether to display on the large screen display device or the CRT display device may be predetermined depending on the type of the pointer.

If each CRT display device can be touch-operated, the information E in FIG. 21 is the CRT display device 6E.
23, by touching the display portion of the control rod drive system E ₁ with a finger, the information of FIG.
It can be displayed on the T display device 6E. Further, similarly to the above, the headset microphone Ma ₁ may be used to input by voice. When the screen as shown in FIG. 23 is displayed on the CRT display device 6E, the operator A ₁ gives an operation instruction by operating the “operation mode”, “driving mode”, “sequence type”, “control rod No.”, etc. To do. The shaded portion in FIG. 23 shows the result of the operation by the operator A ₁ . In addition, the black-painted portion in the control rod coordinates displayed on the left side of FIG. 23 indicates which control rod has been selected as a result of the operator A ₁ selecting the control rod. 26-27, 26-35, 3
4 to 35) and displayed. Note that FIG. 23 is a display example of a CRT with a touch operation. By displaying the operation means, the operation and information obtained as a result of the operation (eg, information on the drive state of the control rod) are output on the same screen. Therefore, it is very convenient for the operator to confirm the operation state.

In this case, not only the CRT display device 6E but also other CRT display devices may be distributed and displayed.

Next, an example in which another operator A ₂ outputs new information to the CRT display device 6B by using the backup panel 40 which is an operation / monitor panel will be described. An example of outputting alarm information (alarm information) to the CRT display device 6B as information will be described.

First, when the user wants to view the details by selecting an alarm item of a specific system from the system-specific alarm display device 8 with the pointer Pa ₂ , he points to the corresponding alarm tile 8 _i . When the alarm tile 8 _i is instructed by the instruction signal 5b, the image processing device 2
Recognizes the designated position and the type of pointer. As a result, for example, the alarm information shown in FIG. 24 or FIG.
It is displayed on the RT display device 6B. In FIG. 24, the shaded portions indicate that the alarms (MDRFP-A drip 8 _i1 and MDRFP-A suction flow rate low 8 _i2 ) are generated, but the alarm is generated on the CRT display device 6B. Change the display color to distinguish from the alarms that are not displayed. In FIG. 25, the time of occurrence is added to the generated alarm and displayed. 24 and 2
In each case of 5, the information indicating which system (water supply system) is added.

Note that FIG. 25 shows the alarm information displayed when the alarm tile 8 _i is instructed. Of the display information of FIG. 24 displayed on the CRT display device 6B (touchable), the information of FIG. 25 may be output by touching the shaded area. In this way, what kind of alarm items are in the system,
There is an effect that the user can easily grasp which alarm item has occurred and when it occurred.

In the above-mentioned alarm display, detailed alarm information is displayed by recognizing the pointing point without specifying the pointing object, but this is due to the following reason.

Even if the same portion of the display surface of the display device is instructed like a large screen display device, if the contents of the display screen are different, different objects are instructed. Therefore, when instructing the screen of the display device, it is necessary to specify the instructed display object. However, the operation
When the display target is fixed like a monitoring panel, the pointing point and the pointing target are always in a one-to-one relationship, so the coordinates of the pointing position can be determined without specifying the pointing target. Then
Information about the coordinates may be output.

As described above, when the display object is always fixed like a panel display device such as a mimic display, it is not always necessary to specify the instructed display object, and By determining the point (coordinates), the coordinates (corresponding to the indicated display object) are
Information assigned in advance can be output to the display device corresponding to the pointer.

In the embodiment shown in FIG. 20, it is possible to give instructions to both the large screen display device and the operation / monitor panel.
In the case where an object to be displayed is fixed as in the case of a monitoring board, a pointing point is detected by the configuration of the operation / monitoring device according to another embodiment of the present invention shown in FIG. 26, and based on this pointing point, Can generate new information. The difference between this embodiment and the embodiment of FIG. 11 is that the object specifying means 11 of FIG. 11 is replaced with the pointing point detecting means 180. Since the pointing point detecting means 180 only uses the output information of the coordinate converting means 10 as the pointing point information, the coordinate converting means 10 may be considered as the pointing point detecting means 180. This embodiment can obtain the same effect as the embodiment of FIG.

FIG. 27 shows the configuration of an operation / monitoring device according to another embodiment of the present invention. The difference between this embodiment and the embodiment of FIG. 26 is that the image processing device 2A
That is, the identification means 27 is provided instead of performing the process 2e. The identification means 27 uses the pointers Pa _{1 to} Pa.
_The pointer outputting the instruction signal is identified based on the identification signals 27A _{1 to} 27A _N output from _(N-1) and Pb '.
The identification result is input to the display control device 20. On the other hand, the image processing apparatus 2A obtains the position (coordinates) of the point designated by the pointer. The obtained position coordinates are corrected by the coordinate conversion means 10 and then output to the display control device 20 via the designated point detection means 180.

This embodiment produces the same effect as the embodiment of FIG. Furthermore, according to the present embodiment, the function of the image processing apparatus 2 is simplified, so that the processing of the image processing apparatus can be shortened and the responsiveness of the entire operation / monitoring apparatus can be improved. Further, since the identification of the pointer is processed based on the identification signal and the identification by the image processing is unnecessary, it is possible to instruct by the pointer having the same instruction information such as the shape and the color.

The identification signal output from each pointer may be an ultrasonic signal, a wireless signal, or a wired signal. However, it is desirable to use an ultrasonic signal or a wireless signal in order not to limit the use place of the pointer. In order to identify the pointer using these signals, various methods such as a method in which the frequency and the output cycle of the identification signal are different for each pointer can be applied.

An operation / monitoring device according to another embodiment of the present invention will be described. The present embodiment is different from the embodiment of FIG. 27 in that the object specifying means 11 is provided instead of the pointing point detecting means 180. Therefore, in addition to the effect of the embodiment of FIG. 27, it is effective when the instructed object is specified and new information regarding this object is generated.

An operating / monitoring device according to another embodiment of the present invention will be described with reference to FIG. The difference from the embodiment of FIG. 27 of this embodiment is that the object specifying means 11 is provided instead of the pointing point detecting means 180, and the identifying means 28A that identifies the pointer by using the voice input information instead of the identifying means 27. ₁
In further comprising a ~28A _N. Identification means 28A ₁ ~ 2
Each of the 8A _N identifies a corresponding pointer using the output signal of the corresponding voice input device among the voice input devices 16A _{1 to} 16A _N. By previously associating the pointer and the voice information of the person who inputs a voice while using the pointer, the identification means 28A _{1 to} 28A _N can identify the pointer based on the voice information. The identification information, which is the output signal of the identification means 28A _{1 to} 28A _N , is captured by the command input analysis device 15 and is output to the display control device 20 together with the analysis result of the command by voice input. As described above, the pointer can be identified using the voice input information. Further, by previously in association with the previously-one pointers Pa~Pb an audio input device 16A ₁ ~16A _N, the identification means 28A ₁ ~28A _N output signals (voice information from the voice input device 16A ₁ ~16A _N ), Whether or not the pointer is used can be determined. When the determination result is output as the identification information to the command input analysis device 15, the processing of the identification means 28A _{1 to} 28A _N is extremely simple.

FIG. 29 shows an operating / monitoring device according to another embodiment of the present invention. This embodiment is similar to the embodiment of FIG.
28 is that the object specifying means 11 in the embodiment of FIG. 28 is replaced by the pointing point detecting means 180. The embodiment of FIG. 28 is applicable to the case where the display object is fixed, such as mimic display, but this embodiment is applicable to the case where the display object is variable. This embodiment can also obtain the same effects as the embodiment of FIG.

An operation / monitoring apparatus according to another embodiment of the present invention will be described with reference to FIG. The present embodiment differs from the embodiment of FIG. 11 in that the markers 5a, 5b, 5e, 5f, 5g,
... is displayed on the large screen display device 4 in advance, and the mark can be selected by applying an instruction signal such as the pointer Pa or Pb to one of these marks. The display target is selected by moving the selected mark onto the display target desired to be pointed by operating the corresponding pointer and erasing the pointing signal emitted from the pointer. The display controller 20 performs these processes. The display control device 20 displays the previously registered markers 5a and the like in a predetermined area of the large screen display device 4 (provided below the display area in FIG. 30).

In the embodiments of FIGS. 11 and 26 to 30, it is already possible to point out various display objects or display positions with the respective pointers and generate new information about the pointed objects or the specified positions. Stated. However, in the case where the role of each operator is fixed as in plant control, it may be desirable to limit the range instructed by each operator. In order to be able to deal with such a case, an instruction range is assigned for each pointer, and when the pointer is used, whether the instruction by the pointer is within the predetermined instruction range or not is determined. It is necessary to accept and process the instruction only when it is determined and within the instruction range.

This determination processing is executed by the display control device 20. In this case, the display control device 20 has the configuration shown in FIG. 31, especially the allocation determination means 20A for performing the determination processing.
Have. The configuration of FIG. 31 can be applied to the display control device 20 of each of the embodiments of FIGS. 11 and 26 to 30.

The allocation determination means 20A has an instruction range registration information storage means 20A ₁ and a match determination means 20A ₂ . When the identification information of the pointer (information indicating the identification result of the pointer) and the pointed position / target information (information indicating the pointed position or the pointed object) are input, the coincidence determination means 20A ₂ indicates the pointing range. Registration information storage means 20
Information indicating the instructable range corresponding to the identification information (instruction range registration information) is read from A ₁ . The read information is compared with the input designated position / target information, and if they match, the permission information for permitting the designation is output to the display control processing unit 20B. If they do not match, the disapproval information is output. This makes it possible to limit the range of instructions for each pointer.

The display control processing unit 20B is shown in FIGS.
6 to 30 have the functions of the input sentence analysis unit and the display information creation unit in the display control device 20 of each embodiment. Therefore,
The display control processing unit 20B, when the permission information is input,
Corresponding display information is created by the processing in the input sentence analysis unit and the display information creation unit, and this is output to the corresponding display device.

In the above description, the pointing point or pointing target pointed immediately before the pointing information emitted from the pointer disappears is the pointing point or pointing target pointed by the pointer. However, if the user accidentally releases the pointer switch (K _{5 in} FIGS. 6 and 7) (OFF state), or another person crosses between the pointer and the large screen display device, the instruction information is temporarily lost. It is conceivable that the user may consider that the user has instructed a part different from the point to be pointed and the pointed object to be pointed by the user. On the other hand, this problem can be solved by assuming that the instruction signal has truly disappeared only when the pointer switch is released twice within a certain period. That is, it is considered that the instruction signal disappears only when the instruction signal disappears once and is reissued within a certain period, and the instruction signal disappears. That is, it is determined that the instruction signal has disappeared in response to a double-click operation of the switch of the pointer (operation of a second or shorter time). The process for this is as shown in FIG. 32, and is executed by the image processing device 2 (or the display control device 20) in each of the embodiments of FIGS. 1, 11, and 26 to 30.

The processing of FIG. 32 will be specifically described. Figure 32
Shows the processing when the instruction signal is detected, the entire processing is repeatedly executed. The output information J _{1 in} FIG. 32 is
This is information indicating that the instruction signal has disappeared correctly. Only when the output information J ₁ is obtained, the display control device 20 generates new information in response to the pointer instruction. Output information J ₂
, It only indicates the position of the instruction signal, and is used for displaying a marker or the like. In FIG. 32, when the instruction signal is issued three times or more, the output information A is not output to improve the reliability of the instruction signal detection. When I actually tried double-clicking with the pointer, I turned the pointer switch on and off three times.
I turned it off. Therefore, in FIG. 32, the state D
If is obtained, the output information J ₁ may be immediately output. As a result, it is possible to realize the processing suitable for the actual operation.

FIG. 33 is a diagram showing how an instruction signal is output in response to a double-click operation of the pointer.

In the process of FIG. 32, the instruction signal is detected in step S ₁ and it is determined in step S ₂ whether the instruction signal has disappeared. If the instruction signal has not disappeared, step 10 is processed, the process returns to the beginning again, and steps S ₁ and S ₂ are processed. If it is determined in step S ₂ that the instruction signal has disappeared, the process proceeds to step S ₃ . This condition leads to t ₁ from t ₀ in FIG. 33 is a state that is a period △ T _1.

When it is determined in step S ₃ that the instruction signal does not exist, the process proceeds to step S ₄ . As shown in step 4, if the result of this processing is YES, step S
Return to ₃ . That is, it exists in the period of ΔT ₁ . If the processing result of step 4 is NO, it means that the instruction signal has not been re-output within ΔT ₁ , and therefore the processing returns to the beginning after the processing of step S ₁₀ . That is, it is determined that the pointer is not double-clicked. This allows
The reliability of detection of the instruction signal is improved.

Next, if the processing result in step S ₃ is YES, the process advances to step S ₅ . This state is ΔT in FIG.
Indicates that the state is ₂ . In step S ₅ ,
If the processing result is NO, the process proceeds to step S ₆ . If it is determined in step S ₆ that the instruction signal does not disappear even after ΔT _{2 has} elapsed, it means that the pointer has not been double-clicked. Therefore, after step S ₁₀ , the process returns to the beginning. Processing result of the step S ₆ is equal YES, the flow returns to step S _5. If the processing result of step S ₅ is YES, the process proceeds to step S _7. The state of step S _{7 is} a state between t ₃ and t ₄ in FIG. The processing result of step S ₇ is NO, and the processing result of step S ₈ is also NO.
If so, the pointer was double-clicked. Therefore, after the processing step S _9, the processing result of the step S ₈ Back to top If YES, the flow returns to step S _7. If the processing result in step S ₇ is YES, the process in FIG.
Is between t ₄ and t ₅ . That is, since the operation of double-clicking or more has been carried out, the process returns to the beginning after the processing of step S ₁₀ .

As described above, only when the double-click operation is performed within the predetermined time, it is possible to consider that the specific portion is designated by the designation signal. However, as described above, even if a double-click operation or more (for example, three click operations) is performed, if the operation is allowed, if the processing result of step S ₅ is YES, the step and unnecessary S _7, S _8, it is sufficient to perform the steps S _9. In FIG. 33, all the states after t ₃ are allowed. As a result, even if the operator is unfamiliar with the operation of the pointer, it is possible to give an instruction allowing the operation. Note that, in step S _10, it is set to output the coordinates of the instruction signal disappears, so it is not possible to detect an instruction signal disappears, strictly a coordinates of the signal immediately before the instruction signal disappears.

The measures for improving the reliability of the instruction signal detection described above can be applied to any of a plurality of pointer types and a single pointer type. Furthermore, the number of pointers may be plural or single, or in any case. The same applies to the following examples.

FIG. 34 is a diagram showing another embodiment for improving the reliability of instruction signal detection. The subject of explanation is the large-screen display device 4. An invalid display area AR1 is provided on the display screen of the large screen display device 4 so that no information is displayed in this AR1, and information is displayed in the valid display area AR2 (which is the target range of the instruction). Is determined in advance. This is executed by the display control device 20 of FIG. By thus providing the invalid display area AR1 in the periphery of the display screen of the display device, the user no longer points to the invalid display area AR1 with a pointer. Therefore, if an instruction signal is input to this AR1, This A
Even if the instruction signal disappears in R1, the instruction may not be considered valid. If this AR1 does not exist, as shown in FIG. 34, the instruction signal is moved to the upper right, and when the instruction signal overflows and disappears from the large screen display device 4, the instruction signal disappears from the coordinates of the instruction signal immediately before the disappearance. Think of it as a position. On the other hand, by handling the display screen of the large screen display device as shown in FIG. 34, this problem can be solved. When the instruction signal is present in the invalid display area AR1, the marker is A
Displayed in R1. For a panel display device such as a mimic display or an instruction from the operation / monitor panel, the instruction target range is set as the effective display area AR2, and the others are the same as those in FIG. 34. This can be dealt with by setting AR1.

FIG. 35 is a schematic diagram of FIG. 1 and FIG.
1 and the processing procedure executed by the image processing apparatus 2 (or the display control apparatus 20) in each of the embodiments of FIGS. 26 to 30. The output information J _{1 in} FIG. 35 is information indicating that the instruction has been correctly made, and only when this information is obtained, the display control device 20 generates new information in response to the pointer instruction. The output information J ₂ only indicates the position of the instruction signal and is used for displaying a marker or the like.

[0108] In FIG. 35, detects an instruction signal in step S _1, an instruction signal is invalid display area A in Step S ₂
It is determined whether or not it is within R1. This judgment result is YES
If so, the process of step S ₆ is executed and the process returns to the beginning.
That is, since the instruction signal exists in the invalid display area AR1, even if the instruction signal disappears in the invalid display area AR1, it is not considered valid. If the processing result is NO in step S ₂ , the process proceeds to step S ₃ . Here, if an instruction signal to the effective display area AR2, the process proceeds to step S _4, although the instruction signal has disappeared determines or not. If the instruction signal disappears, the process proceeds to step S ₅ , the output information J ₁ is output, and the process returns to the beginning.
In all other cases, the process of step S ₆ is performed and the process returns to the beginning. The output information J ₁ of FIG. 35 is the same as that of FIG. 32, and the output information J ₂ of FIG. 35 is the same as that of FIG. This makes it possible to generate new information for the pointing position or the pointing object only when the pointing signal disappears while pointing in the effective display area AR2, and the reliability of the detection of the pointing signal can be improved. It is possible to improve the sex.

Further, in the processing of FIG. 35, it is possible to output the output information J ₁ only when the instruction signal is erased by double-clicking the pointer. That would be to combined processing of FIGS. 35 and 32, which comprises the steps S ₂ and subsequent processing steps in FIG. 32 (S ₂ ~
Step S ₄ subsequent processing steps in FIG. 35 a S ₁₀₎ (S
_{4 to} S ₆ ) and replace it. As a result, it is possible to further improve the reliability of the instruction signal detection.

Also, the instruction signal may disappear while the instruction signal is being moved. The reason is that the pointer signal is intentionally erased by the user's pointer operation,
The instruction signal may run off from the large screen display device and disappear. The former is not a problem, but for the latter, the movement locus of the instruction signal is tracked, and if the instruction signal is predicted to be out of the large screen display device or the like, even if the instruction signal disappears, You can prevent it from being considered valid.

The various processing methods described above are based on the actual operation of the pointer, and it is possible to improve the reliability of detection of the instruction signal.

[0112]

According to the present invention, it is possible to identify a plurality of pointers that output different pointing signals, and appropriately create and display information about the pointing object pointed by the identified pointers. Therefore, a plurality of users can appropriately obtain the information they need by using different pointers that output different instruction signals. In addition, even when a plurality of users point at the same time with the pointer, the information requested by each of them is properly displayed.

According to another feature of the present invention, since the display device for displaying the information is specified corresponding to the identification information of the pointer, the usability of the information is improved for each operator.

Further, according to another feature of the present invention, since the pointing point or the pointing object and the pointing pointer are associated with each other, new display information, operation information and voice information are based on these. Etc. can be generated. In particular, by setting the range of the created information and the display device for each pointer in advance, it becomes possible to divide the display information for each purpose of use and object of use.

Pointers with different pointing signals point to a part of the device where the display object is fixed, such as the mimic display device and operation panel, and the information related to the pointed part is presented to the display device corresponding to the pointer. It is also possible.

According to the present invention, a plurality of persons can designate an instruction target without limiting the operation place.

[Brief description of drawings]

FIG. 1 is a configuration diagram of an operation / monitoring device according to an embodiment of the present invention.

FIG. 2 is a detailed configuration diagram showing functions in the electronic computer of FIG.

FIG. 3 is an explanatory diagram showing a processing procedure executed by the image processing apparatus of FIG.

FIG. 4 is an explanatory diagram showing specific contents of processes 2c to 2e of FIG.

FIG. 5 is a diagram illustrating light emitting surfaces of various pointers.

FIG. 6 is a perspective view of an example of a pointer.

7 is an explanatory diagram of a laser light generation circuit of the pointer P of FIG.

8 is an explanatory diagram of an instruction signal output from the pointer P of FIG.

9 is a perspective view of the pointer Pb in FIG. 1. FIG.

10 is a configuration diagram of a light emitting source for outputting a cross-shaped instruction signal of the pointer of FIG.

FIG. 11 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 12 is an explanatory diagram of a processing procedure executed by the object identifying means of FIG. 11.

13 is an explanatory diagram showing the processing contents of input sentence analysis executed by the input sentence analysis unit of the display control device of FIG. 11. FIG.

14 is an explanatory diagram showing one processing content of display information creation executed by the display information creation unit of the display control device of FIG. 11. FIG.

FIG. 15 is an explanatory diagram of a screen display switching command.

FIG. 16 is an explanatory diagram illustrating a display example of information about an instructed target object.

17 is a configuration diagram of a workstation having the functions of the coordinate conversion means, the object specifying means, the command input analysis device, and the display control means of FIG.

FIG. 18 is a perspective view of another embodiment of the pointer.

19 is an explanatory diagram showing a configuration example of a light emitting surface of the pointer of FIG.

FIG. 20 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 21 is an explanatory diagram showing an example of information displayed on the display device when the core display unit displayed on the large screen display device in FIG. 20 is pointed by a pointer.

22 is an explanatory diagram showing an example in which the new display information of FIG. 20 is displayed on a large screen display device.

23 is a control rod drive system E of the display information shown in FIG.
It is explanatory drawing which shows an example of the information displayed on a CRT display device, when ₁ is selected.

FIG. 24 is an explanatory diagram showing a display example when the alarm tile 8 _i in FIG. 20 is designated by a pointer.

FIG. 25 is an explanatory diagram showing another display example when the pointer is used to point to the alarm tile 8 _i in FIG. 20.

FIG. 26 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 27 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 28 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 29 is a configuration diagram of an operation / monitoring device according to another embodiment of the present invention.

FIG. 30 is a configuration diagram of an operation / monitor device according to another embodiment of the present invention.

FIG. 31 is a configuration diagram of another embodiment of the display control device having an allocation determination means.

FIG. 32 is an explanatory diagram of a processing procedure relating to double-clicking applied in detecting a pointing target and a pointing point.

33 is an explanatory diagram showing an output situation of instruction information with respect to a double-click operation of the pointer of FIG. 32. FIG.

FIG. 34 is another example of a measure for improving the reliability of instruction information detection.

FIG. 35 is another example of the processing method in which the reliability of detection is increased when detecting the pointing target or the pointing point.

[Explanation of symbols]

HD, HD _{1 to} HD _K ... Head mount display, P
a, Pb ... Pointer, 1 ... Monitor camera, 2, 2A ... Image processing device, 2c ... Feature extraction unit, 2d ... Pointed position detection unit, 2e ... Pointer identification unit, 3, 40, 41 ... Operation / monitor board, 4 , 4A, 4B ... Large screen display devices, 5a, 5b ...
Markers, 6A to 6G ... CRT display device, 7 ... Electronic calculator, 10 ... Coordinate conversion means, 11 ... Object specifying means, 15
... command input analyzer, 16A ₁ ~16A _N ... voice input device, 20 ... display controller, 21,37,65 ... memory,
25 ... Information generating means, 27 ... Identification means, 31, 60 ... Coordinate recognition section, 32, 61 ... Target specifying section, 34 ... Input sentence analysis section, 36, 63 ... Display information creating section, 62 ... Display information determining section, 180 ... Point detection means, AR1 ... Invalid display area, AR2 ... Valid display area.

Front Page Continuation (72) Inventor Hiroshi Seki 7-2-1 Omika-cho, Hitachi City, Ibaraki Prefecture Hitachi Energy Research Laboratory Ltd. (72) Inventor Yukio Nagaoka 7-2-1 Omika-cho, Hitachi City, Ibaraki Prefecture Hitachi, Ltd. Energy Research Institute

Claims (15)

[Claims] 1. A display device, a plurality of pointers that output different indication signals and indicate the display surface of the display device by the indication signals, identification means for identifying the pointers, and an instruction by the pointers. An information providing apparatus comprising: an object specifying unit that specifies a display object displayed on the display surface; and a unit that outputs the identification information of the pointer and the information obtained based on the display object. 2. The information providing apparatus according to claim 1, further comprising: photographing means for photographing the display surface and outputting a video signal, and the identifying means for identifying the pointer based on the video signal. 3. A display device, a plurality of pointers that output different instruction signals and point the display surface of the display device with the instruction signals, and a photographing means that photographs the display surface and outputs a video signal. Identification means for identifying the pointer pointing to the display surface based on the video signal, and object identifying means for identifying the display object displayed on the display surface pointed by the pointer. And a display information determining means for determining display information based on the information of the identified display object, and a display device for creating the determined display information and corresponding the display information to the identified pointer. An information display device characterized by outputting to. 4. A means for determining a position indicated by the indication signal on the display surface based on the video signal, and an object specifying unit for specifying the display object based on the indicated position. Item 3 information display device. 5. The information display device according to claim 3, further comprising the pointer that outputs infrared light that is the instruction signal. 6. The information display device according to claim 3, wherein said photographing means is an infrared camera. 7. The information display device according to claim 4, wherein said photographing means is an infrared camera. 8. The information display device according to claim 7, further comprising means for creating display information of a mark indicating that the pointing signal is pointing at the pointing position and outputting the display information. 9. A first display device and a first display device of the first display device.
A second display device having a second display surface that is larger than the display surface, a plurality of pointers that output different instruction signals and point the second display surface by the instruction signals, and the second display surface is photographed. A photographing means for outputting a video signal, an identification means for identifying the pointer pointing to the second display surface based on the video signal, and a display means for displaying on the second display surface designated by the pointer. A display object determining unit that determines display information based on information of the display target object that specifies the display target object and the information of the display target object that is specified, and creates the determined display information. An information display device, wherein display information is output to the display device corresponding to the identified pointer. 10. A first display device and a second display device having a second display surface that is larger than the first display surface of the first display device, and outputs different instruction signals to the second display surface. A plurality of pointers designated by a designation signal, a photographing means for photographing the second display surface and outputting a video signal, and identifying the pointer pointing the second display surface based on the video signal. Based on the identification means and the object specifying means for specifying the display object displayed on the second display surface, which is pointed by the pointer, based on the information of the specified display object and the voice input information. An information display device, comprising: display information determining means for determining display information; and creating the determined display information, and outputting the display information to the display device corresponding to the identified pointer. 11. A display device, which outputs an instruction signal different from that of the display device,
A plurality of pointers for pointing the display surface of the display device by the pointing signal, and an identification means for identifying the pointers;
An object specifying means for specifying a display object displayed on the display surface, which is pointed by the pointer, and means for outputting control information obtained based on the identification information of the pointer and the display object. Operation control device equipped with. 12. The operation control device according to claim 1, further comprising: photographing means for photographing the display surface and outputting a video signal, and the discrimination means for discriminating the pointer based on the video signal. 13. A display device, which outputs an instruction signal different from that of the display device,
A plurality of pointers for pointing the display surface of the display device by the instruction signal; a photographing means for photographing the display surface and outputting a video signal; and pointing the display surface based on the video signal. Identification means for identifying a pointer, and an object specifying means for specifying the display object displayed on the display surface, which is pointed by the pointer, and is displayed based on information on the specified display object. An operation control device comprising: a display information determining means for determining information; and the display information thus determined is created, and the display information is output to a display device corresponding to the identified pointer. 14. A pointer for pointing a display surface of a display device, a feature extracting means for extracting a feature of an instruction signal from each pointer focused on the display surface, and a point indicated by the instruction signal. Pointing point detecting means, output of the feature extracting means and the pointing point detecting means whether the pointing signal is the pointing signal assigned to the pointing range when the pointing signal is pointed to a predetermined pointing range Allocation determining means for determining based on information, and when inputting coincidence information indicating that the instruction signal and the instruction range match from the allocation determining means, the information assigned to the instruction range is generated. An information providing device including means. 15. The information display device according to claim 3, wherein in the display device, a specific area in which information is not displayed is provided in a peripheral area in the display surface.