JP3375644B2 - Processing execution method and valve remote operation method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a man-machine interface (hereinafter, simply referred to as "man-machine") using video images (hereinafter, simply referred to as "images"), and more particularly to images input from a video camera (hereinafter, simply referred to as "man-machine interfaces"). A video operation method and device suitable for monitoring and operating a remote object, a video search method and device, a video processing definition method and device, and
The present invention relates to a remote operation monitoring method and system for a power generation plant, a chemical plant, a steel rolling mill, a building, a road, a water and sewer system, and the like.

[0002]

2. Description of the Related Art In order to safely operate a large-scale plant system represented by a nuclear power plant, an operation monitoring system having an appropriate man-machine is indispensable. The plant is operated and maintained by the operator's three tasks of “monitoring”, “decision” and “operation”. The operation monitoring system needs to be equipped with a man-machine to facilitate these three tasks of the operator. In the "monitoring" work, it is necessary to be able to grasp the state of the plant immediately and accurately. At the time of “judgment”, the operator must be able to quickly refer to the information and the information that is the basis for the judgment. At the time of "operation", an operation environment in which the operation target and the result of the operation can be intuitively understood and the operation intended by the operator can be executed accurately and quickly is indispensable.

The man-machine of the conventional operation monitoring system is
Let's take a look at each work of “monitoring”, “judgment” and “operation”.

(1) Monitoring The state inside the plant can be grasped by monitoring data from various sensors for detecting pressure and temperature and images from video cameras arranged at various places in the plant. Values from various sensors are displayed in various forms on a graphic display or the like. Trend graphs and bar graphs are also widely used. On the other hand, the image from the video camera is often displayed on a dedicated monitor different from the graphic display. 4 in the plant
It is not uncommon for zero or more cameras to be installed. The operator monitors the various parts of the plant while switching cameras and controlling the camera direction and lens. In normal surveillance work, the operator rarely sees the image from the camera, and at present the utilization rate of the camera image is low. (2) Judgment When something abnormal occurs in the plant, the operator comprehensively examines a large amount of information obtained from sensors and cameras,
You have to quickly and accurately determine what is happening in your plant. In the current operation monitoring system, since data from various sensors and images from cameras are managed independently, it is difficult to refer to them while associating them with each other, which puts a heavy burden on the operator.

(3) Operation The operation is performed using the buttons and levers on the operation panel. Recently, there are many systems that can be operated by combining a graphic display and a touch panel and selecting menus and figures displayed on the screen. However, the buttons and levers on the control panel,
Also, the menus and figures on the display are
It is an abstract form that is separated from the things on site, and it may be difficult for the operator to remember their functions and the results of the operation. That is, there is a problem that it is not immediately known which lever can be operated to perform a desired operation, and it is difficult to intuitively grasp what operation command is sent to which device in the plant when a certain button is pressed. Further, since the operation panel is configured independently of a monitor such as a camera, there is a problem that the device becomes large.

[0006]

As described above, the prior art has the following problems.

(1) Remote operation using keys, buttons, levers on the operation panel, or remote operation using menus or icons on the screen makes it difficult to convey the sense of presence in the field, and therefore the actual operation target and operation result can be displayed. It is difficult to grasp intuitively and there is a high possibility that it will be operated incorrectly.

(2) The operator must directly switch the camera or perform remote control, and when monitoring is performed using a large number of cameras, it is not possible to easily select a camera that shows what he or she wants to see. In addition, it takes a lot of time to operate a remote camera so that the desired portion can be clearly seen.

(3) A screen for displaying an image from the video camera, a screen for referring to other data, and a screen or device for operating are separate, and the entire device becomes large. There are problems such as difficulty in cross-reference with data.

(4) Although the camera image has a high effect of transmitting a sense of realism, it has a drawback that the operator cannot intuitively grasp the structure in the camera image because it has a large amount of information and is not abstracted. On the other hand, the graphic display makes it possible to emphasize important parts, simplify unnecessary parts, and abstract only essential parts to display, but it is an expression separated from actual things and events. , The operator may not be able to easily recall the relationship between the graphic expression and the actual thing or event.

(5) Image information from the camera and other information (for example, data such as pressure and temperature) are managed independently of each other, and mutual reference cannot be easily performed. Therefore, it is difficult to judge the situation comprehensively.

An object of the present invention is to achieve at least one of the following (1) to (3) .

(1) In a remote operation monitoring system, etc.,
To allow the operator to intuitively understand the operation target and operation result.

[0014]

(2) To reduce the size of the remote operation monitoring system and save space.

(3) Utilizing the advantages of each of the camera image and the graphics and compensating for the disadvantages .

[0017]

[0018]

According to the invention, said object is solved by a method comprising the steps of:

(1) Object designation step An object in the video image displayed on the screen (hereinafter, referred to as an object) is a pointing device (hereinafter, referred to as an object).
It is specified using an input means such as (PD). Video images can be input from a remotely located video camera or played from a storage medium (optical video disc, video tape recorder, computer disc, etc.). As the pointing device, for example, a touch panel, tablet, mouse, eye tracker, gesture input device, or the like is used. Before designating the object, the specifiable object in the video may be clearly displayed by the composite display of graphics.

(2) Processing Execution Step Processing is executed based on the object designated by the object designation step. The contents of the processing are as follows, for example.

Sending an operation command that causes the specified object to move or results similar to when the specified object is operated. For example, when the designated object is a button, the button is actually pushed down, or an operation command that gives the same result as when the button is pushed down is sent.

[0022]

The graphics are compositely displayed on the video so that the specified object is clearly displayed.

Display information related to the specified object. For example, the manual, maintenance information, structure diagram, etc. of the specified object are displayed.

Display a list of processes that can be executed on the specified object by using a menu. The menu can also be expressed graphically. That is, some figures are composite-displayed on the image, the composite-displayed figures are selected by the PD, and the next process is executed based on the selected figures.

[0026]

[0027]

[0028]

Operation: The object in the video image on the screen is directly specified, and the operation command is sent to the specified object. The operator gives an operation instruction while looking at a video image of the object actually taken. If the operation instruction causes a visible movement of the object, the movement is directly reflected in the camera image. In this way, by directly operating the live-action image, the operator can perform remote operation as if he / she was actually working on site. As a result, the operator can intuitively understand the operation target and the operation result, and can reduce erroneous operations.

[0029]

When a direct operation is applied to an object in a video, graphics are appropriately displayed on the video. For example, when the user designates an object, graphics display is performed so as to clearly indicate which object has been designated. As a result, the operator can confirm that the operation intended by the operator is being performed reliably. Also,
If a plurality of processes can be executed on the specified object, a menu for selecting a desired process is displayed. This menu may be composed of figures. By selecting the figure displayed as the menu, the operator can have a stronger sense of actually operating the object.

Information is displayed based on the object in the image designated on the screen. With this, the information related to the object in the video can be referred to only by designating the object. The situation can be easily determined while referring to the video and other information at the same time.

[0032]

[0033]

[0034]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A plant operation monitoring system according to an embodiment of the present invention will be described with reference to the drawings .

The overall structure of this embodiment will be described with reference to FIG. In FIG. 1, 10 is a display serving as display means for displaying graphics and images, 12 is a pressure-sensitive touch panel serving as input means mounted on the entire surface of the display 10, 14 is a speaker for outputting voice, and 20 is an operation. A man-machine server that provides a man-machine for a person to monitor and operate the plant, a switcher 30 for selecting one video input and a voice input from a plurality of video inputs and a plurality of voice inputs, and 50 A control computer that controls equipment in the plant and collects data from sensors. Reference numeral 52 is an information system local area network that connects the control computer 50 to the man-machine server 20, other terminals, and computers. -KU (hereinafter referred to as LAN)
(For example, LA as specified in IEEE802.3
N) and 54 are control system LANs (for example, LANs defined by IEEE802.4) that connect the control computer 50 to various devices to be controlled and various sensors.
Reference numerals 60, 70, 80 are placed in various places in the plant, and industrial video cameras (hereinafter simply referred to as cameras) for capturing and inputting an object to be controlled, 62, 72,
Reference numeral 82 is a controller for controlling the orientations and lenses of the cameras 60, 70, 80, respectively, according to commands from the control computer 50, 64, 74, 84 are microphones attached to the cameras 60, 70, 80, respectively. 90,92
Are various sensors for knowing the state of each part of the plant, 94,
An actuator 96 controls various devices in the plant in accordance with commands from the control computer 50.

The pressure-sensitive touch panel 12 is a type of PD. When an arbitrary position on the touch panel 12 is pressed with a finger, the coordinates of the pressed position and the pressed pressure are reported to the man-machine server. The touch panel 12 is attached to the entire surface of the display 10. The touch panel 12 is transparent, and the display content of the display 10 behind the touch panel 12 can be seen. This allows the operator to display 1
It is possible to specify by touching the object displayed on 0 with a finger. In this embodiment, as the operation of the touch panel 12, three types of (1) lightly pressing, (2) strongly pressing, and (3) dragging are used. To drag means to move the finger while pressing the touch panel 12 with the finger.
Although a pressure sensitive touch panel is used as the PD in this embodiment,
Other devices may be used. For example, a non-pressure sensitive touch panel, tablet, mouse, light pen, eye tracker, gesture input device, keyboard, etc. may be used.

A plurality of images taken by the cameras 60, 70, 80 are selected by the switcher 30 and displayed on the display 10 via the man-machine server 20. The man-machine server 20 controls the switcher 30 via a communication port such as RS232C to select an image from a desired camera. When the image is selected, the input sound from the microphones 64, 74, 84 and the like are also selected at the same time. That is, when the camera is selected, the sound is also switched to the input from the microphone attached to the selected camera. The voice input from the microphone is output from the speaker 14. Of course, you can also switch the input from the microphone and camera independently. The man-machine server 20 can combine graphics with the image from the camera. The man-machine server 20 also sends an operation command to the control computer via the information system LAN 52 to specify the orientation of the camera and the control of the lens. This camera orientation and lens control will be referred to as camera work.

Further, the man-machine server, in accordance with the instruction of the operator, through the control computer 50, the sensor 90,
Input data from 92, etc.
Remotely operate 4, 96, etc. Man-machine server 20
Controls the movements and functions of various equipment in the plant by sending operation commands to the actuator.

The configuration of the man-machine server will be described with reference to FIG. In FIG. 2, 300 is a CPU, 310 is a main memory, 320 is a disk, 330 is a PD, an I / I for connecting the touch panel 12, the switcher 30, and the like.
Reference numerals O and 340 denote graphics frame buffers for storing display data generated by the CPU 300, and 36.
0 is a digitizer that digitizes the input analog video information, 370 is a video frame buffer that stores the digitized video information that is the output of the digitizer 360, and 380 is a graphic frame buffer 340.
Is a blending circuit for synthesizing the contents of the video frame buffer and displaying them on the display 10.

The video information input from the camera is displayed on the display 10 after being combined with the graphics generated by the man-machine server 20. The graphic frame buffer 340 has data of R, G, and B colors and data called α value corresponding to each pixel on the display 10. The α value specifies, for each pixel of the display 10, how to combine the image information in the video frame buffer 370 and the graphic display data in the graphic frame buffer 340. The function of the blend circuit 380 can be expressed by d = f (g, v, α). Here, g and α are color information and an α value of one pixel in the graphic frame buffer 340, v is color information of a pixel corresponding to g in the video frame buffer 370, and d is g and v. Is the color information of the pixel as a result of combining. In this system, the following formula is used as the function f.

[0041] f (g, v, α) = 255 {αg + (1-α) v} Of course, another function may be used as the function f.

The graphic frame buffer 340 has a so-called double buffer structure.
The double buffer has a buffer for two screens, and the buffer to be displayed on the display 10 can be selected at any time. The buffer displayed on the display 10 is called a front buffer, and the buffer not displayed is called a back buffer. Switching between the front and back buffers can be done instantly. Flickering at the time of drawing can be eliminated by drawing graphics in the back buffer and switching the back buffer to the front buffer when the drawing is completed. The contents of either buffer can be read or written from the CPU at any time.

FIG. 3 shows an example of the display screen structure of the display 10. In FIG. 3, 100 is a display 10.
Display screen, 110 is a menu area for designating commands relating to the entire system, 150 is a data display area for displaying data from sensors and various materials and data relating to the plant, 130 is the entire plant and each part. 2 is a drawing display area for displaying a configuration diagram, a structural drawing, a design drawing, and the like, and 200 is an image display area for displaying an image input from a camera.

FIG. 4 shows an example of the display form of the drawing display area 130. In FIG. 4, 132 is a menu for issuing a command that clearly indicates the position of the sensor, and 134 is one object on the drawing designated by the operator.
When the operator selects an object in the drawing displayed in the drawing display area 130, information from the sensor related to the object is displayed on the data display area 150 or the video display area 200. For example, if a camera is defined as a sensor associated with the specified object, the image input from the camera is displayed in the image display area 20.
Displayed at 0. Also, for example, when a hydraulic pressure sensor is defined as a sensor related to the specified object, graphics that clearly indicate the current hydraulic pressure value,
A trend graph showing the change of the hydraulic pressure value so far is displayed in the data display area 150. When the finger is strongly pressed on the touch panel 12, the object on the drawing displayed at the pressed position is designated. Nothing happens if the sensor associated with the specified object is not defined.
In FIG. 4, the display position of the object 134 is strongly pressed with a finger. When the object is pressed with a finger, it is highlighted so that the operator can confirm that the object has been designated. In the example shown in FIG. 4, a camera 60 that reflects the object 134 is displayed on the object 134.
And a microphone 64 for inputting a sound around the object 134 is defined as an associated sensor, and when the object 134 is designated, the image display area 200 is displayed.
An image showing the object 134 is displayed on the screen, and the sound around the object 134 is output from the speaker 14.

FIG. 5 shows one display form of the image display area 200 when the object 134 is designated on the drawing display area 130, and the object 13 arranged in the plant.
4 shows the correspondence relationship with 4. In FIG. 5, 202 to 210
Is a menu for setting the camerawork of the camera that is currently shooting the video, and 220 is a menu for clearly designating the object that can be designated in the video. 202 is a menu for setting the orientation of the camera. By selecting the menu 202, the camera can be panned left and right or up and down. Two
Reference numeral 04 denotes a menu for controlling the lens of the camera to zoom in the image, 206 denotes a menu for controlling the lens of the camera to zoom out the image, and 208 denotes resetting the camera work to the previous camera work. , 210 is a menu for resetting to the first camera work.

Reference numerals 400 to 424 are various objects attached to or around the object 134. Four
00 is a valve, and 410 and 420 are objects 134.
The above-mentioned character display, 412 is a voltage meter, 4
Reference numeral 14 is a button for turning on the power, 416 is a button for turning off the power, 422 is a meter showing the hydraulic pressure, and 424 is a slider knob for adjusting the hydraulic pressure. Valve 400,
The buttons 414, 416 and the knob 424 are not only an operating device that can be actually operated by hand, but also an operating device that can be remotely operated by issuing an operation command from the man-machine server 20.

FIG . 6 shows a region 20 as a reference example of the present invention.
At 0, it shows that camera work is set so that the meter 412 is in the center of the image when the meter 412 is lightly touched with a finger. When the meter 412 is designated by the operator as shown in the left diagram of FIG. 6, the orientation of the camera 60 is set so that the meter 412 appears in the center of the image.
The lens of the camera 60 is controlled so as to zoom in, and the image changes as shown in the right diagram of FIG. By simply touching the object on the screen, the operator can set the camera work so that the object can be seen more clearly, and the remote operation of the camera is not bothered. In FIG. 6, reference numeral 502 is a graphic echo for clearly indicating that the meter 412 has been designated. The graphic echo 502 is erased when the operator removes the finger from the touch panel 12. In this way, the man-machine can be improved by synthesizing the graphic display on the camera image.

FIG. 7 shows, as a reference example of the present invention, how the camera work is set so that the bulb 400 is in the center of the image when the bulb 400 is lightly touched with the finger in the image display area 200. . Valve 40
When 0 is designated by the operator, the image changes so that the valve 400 is largely projected in the center as shown in the right diagram of FIG. In FIG. 7, reference numeral 504 is a graphic echo for clearly indicating that the valve 400 has been designated. The graphic echo 504 is erased when the operator releases the finger from the touch panel 12. Other objects 410,414,
The same operation can be performed for 416, 420, 422, and 424.

When the operator strongly presses the inside of the image display area 200 with the finger, the object displayed at the position of the finger can be operated. 8 to 10 show examples of operating an object. FIG. 8 shows an example of operating the button 414. When the position where the button 414 is displayed on the image display area 200 is strongly pressed with a finger, the button 414 is pushed down from the man-machine server 20 to the actuator for operating the button 414 remote via the control computer 50. Is sent, and the button 414 at the remote site is actually pushed down. Button 414 is depressed,
As a result, how the needle of the meter 412 touches is displayed on the image display area 200 by the camera 60. As a result, the operator feels as if he or she actually pressed the button on the screen.

FIG. 9 shows an example of operating the slider knob 424 by dragging the finger on the touch panel 12. On the image display area 200, when the position where the button 414 is displayed is strongly pressed with the finger and the finger is moved sideways, the knob 424 displayed in the image also moves in accordance with the movement of the finger. As a result of the movement of the knob 424, the meter 422
The needle also moves. At this time, the man-machine server 20 issues a command to the actuator for controlling the knob 424 via the control computer 50 each time the finger moves, and actually moves the knob 424 in accordance with the movement of the finger. As a result, the operator feels as if he / she is actually moving the knob 424 with his / her finger.

FIG. 10 shows an example in which, when a plurality of operations can be performed on an object, a desired operation is selected by selecting a figure compositely displayed on the video. Figure 1
At 0, 510 and 520 are valves 400 and
When the display position of is strongly pressed with a finger, it is a figure composited and displayed on the image. When the operator strongly presses the figure 510 with his finger,
The man-machine server 20 uses the control computer 50 to
An operation command is issued to the actuator to rotate the valve 400 to the left. On the contrary, when the figure 512 is strongly pressed with a finger, the man-machine server sends an operation command for rotating the valve 400 to the right to the actuator. The rotating state of the valve 400 is photographed by the camera 60 and displayed on the image display area 200. Along with the rotation of the valve 400, the graphic 51
You may rotate the display of 0,512.

FIG. 11 shows a method of specifying an operable object. Since not all objects shown in the image can be operated, a means for clearly indicating operable objects is required. In FIG. 11, 514 to 524 are
Objects 400, 412, 414, 41 respectively
6, 422 and 424 are graphics displayed to clearly show that they are operable. In the case of this embodiment,
The extrapolation rectangle of the object is displayed. Of course, various other display methods such as displaying more realistic graphics can be considered for the explicit display of the object.

FIG. 12 shows a reference example in which text is input and a video in which the text is displayed is searched. In FIG. 12, reference numeral 530 is a graphic composited and displayed on an image, 600 is a search sheet for executing a text search, 610 is a next menu for searching for another image that matches with the search key, and 620 is the end of the search. An end menu 630 for designating to perform is a text input area for inputting to the search key. Menu area 1
When a menu for designating a search is selected in 10, a search sheet 600 is displayed on the display screen 100. When a text serving as a search key is input to the text input area 630 from the keyboard and the return key is pressed, the search is started.
The man-machine server searches for a camera that can project an object including the search key, sets the searched camera in camera work so that the search key is clearly visible, and displays the image from the searched camera in the video display area 200. Graphics 530 are composite-displayed on the part that matches the search key in the video to clearly indicate the part that matches the search key. The video search using the text as the search key allows the operator to reflect the desired object by words. With this method, it is possible to quickly find a target to be monitored without switching cameras or operating the camera remotely. Although the keyboard is used for inputting text in this embodiment, other input means such as a voice recognition device or a handwritten character recognition device may be used. Further, in the present embodiment, the text is used as the search key, but a graphic may be used as the search key to search for an image in which a graphic that matches the graphic of the search key is displayed.

[0054] From Figure 13 with reference to FIG. 24, by specifying the object in the image, describing the function of executing the operation based on the object. FIG. 17 shows a flow chart of a program that realizes this function. When the touch panel 12 is pressed on the video display area 200, the object displayed at the pressed position (the position on the screen designated by the operator using the PD such as the touch panel is called an event position) is displayed. Identify (step 1000). When the object can be identified (when the object exists at the event position) (step 1010), the operation defined corresponding to the object is executed (step 1020).

The object shown at the event position is identified by referring to the model of the object to be photographed and the camera information. The model to be photographed is data on the shape and position of the object to be photographed. The camera information is data on how the camera is shooting the object to be shot, that is, the position, orientation, angle of view, orientation, etc. of the camera.

Various methods can be considered for modeling the object to be photographed. In this specification , (1) three-dimensional model,
(2) Two two-dimensional models are used together. 2 above
Below is a summary of the two models and their advantages and disadvantages.

(1) Three-dimensional model A model in which the shape and position of the object to be photographed are defined in a three-dimensional coordinate system. The advantage is that the object can be identified corresponding to any camera work. That is, the object can be operated while freely operating the camera. The disadvantage is that since the model has to be defined in a three-dimensional space, the process of creating the model and identifying the object is more complicated than the 2D model. However, recently, CAD is often used for designing plants and designing and arranging devices in the plant,
Utilizing these data makes it easy to create a three-dimensional model.

(2) Two-dimensional model A model that defines the shape and position of an object in a two-dimensional coordinate system (display plane) for each specific camera work. The advantage is that it is easy to create a model. You can define a model by drawing a figure on the screen. The disadvantage is that you can only operate on camerawork images for which a model has been defined in advance. In order to increase the degree of freedom in camera operation, it is necessary to define the shape and position of an object on the corresponding plane for each more camera work. In many operation monitoring systems, there are many cases where the places to be monitored are decided beforehand. In that case, the number of camera works is determined, so that the disadvantage of the two-dimensional model does not matter.

A method of identifying an object based on a three-dimensional model will be described with reference to FIGS. 13 to 16. Figure 5
The object to be photographed by the camera 60 shown in FIG.
FIG. 13 shows an example of modeling with yz (called the world coordinate system). Here, the shape of each object is modeled by a plane, a rectangular parallelepiped, a cylinder, or the like. Of course, more kinds of three-dimensional basic shapes such as a sphere and a tetrahedron may be used. Further, not only a combination of basic shapes but also a more precise shape model may be used. Objects to be operated 400, 410, 412, 414, 416, 420, 4
22 and 424 are planes 800 and 8 on the model, respectively.
10,812,814,816,820,822,82
It is modeled as 4.

Correspondence between the image captured by the camera and the three-dimensional model will be described with reference to FIG. Imaging by a camera is an operation of projecting an object arranged in a three-dimensional space onto a two-dimensional plane (image display area 200). That is, the image displayed in the image display area 200 is a perspective projection of an object arranged in a three-dimensional space onto a two-dimensional plane. When the two-dimensional orthogonal coordinate system XsYs taken on the screen is called the screen coordinate system, the image taken by the camera is one point (x,
It can be formulated as an equation (1) that maps y, z) to one point (Xs, Ys) on the screen coordinate system.

[0061]

[Equation 1]

The matrix T in equation (1) is called a view transformation matrix. Each element of the view conversion matrix T includes camera information (camera position, posture, orientation, angle of view) and video display area 2
Given a size of 00, it is uniquely determined. Camera information is given in the world coordinate system. In FIG. 14, the position of the camera corresponds to the coordinates of the center Oe of the lens, the orientation of the camera corresponds to the vector OeYe, and the orientation of the camera corresponds to the vector OeZe.

The object identifying process is a process for determining which point on the world coordinate system is projected onto the point P on the screen coordinate system when one point P on the screen coordinate system is designated. As shown in FIG. 15, all points on the extended line of the straight line connecting the center Oe of the camera lens and the point P on the screen coordinate system are projected onto the point P. Among the points on the straight line, the point actually projected on the image display area 200 by the camera is the intersection of the straight line with the object 1 closest to the center Oe of the lens. In FIG. 15, an intersection point P1 between the object 1 and the straight line 840 is projected by the camera onto a point P on the image display area 200. That is, assuming that the event position is P, the object 1 will be identified.

Techniques for obtaining a view transformation matrix T from camera information and techniques for perspectively projecting and displaying a model defined in the world coordinate system on the screen coordinate system based on the view transformation matrix T are well known in the graphics field. Technology. In perspective projection, the process of projecting the surface of the object close to the camera onto the screen and not projecting the surface hidden from the camera by other objects onto the screen is called hidden surface removal (hi
dden surfaceelimination) or visible surface determination (Visible
-Surface Determination), and many algorithms have already been developed. These techniques are, for example,
Foley.vanDam.Feiner.Hughes, “Computer Graphics
Published by Principles and Practice "Addison Wesley (199
0), and Newman.Sproull, “Principles of Interacti
ve Computer Graphics ”, published by McGraw-Hill (197
It is explained in detail in 3). Also, in many so-called graphic workstations, graphic functions such as view transformation matrix setting from camera information, perspective projection, and hidden surface removal are pre-installed by hardware and software, and these can be processed at high speed. .

Object identification processing can be performed using these graphic functions. In the three-dimensional model, the surface of the object to be operated is color-coded in advance, and it is possible to identify which object the surface belongs to by the color. For example, in FIG.
3, planes 800, 810, 812, 814, 8
Different colors are set to 16, 820, 822 and 824 respectively. The color set for each object will be called an ID color. FIG. 16 shows the procedure of the identification processing using this ID-colored three-dimensional model. First, the current camera information is inquired (step 1300), and the view conversion matrix is set based on the inquired camera information (step 1310). The man-machine server 20 always manages the current camera state, and when a camera information inquiry is made, camera information is returned according to the current camera state. Of course, the current camera state may be managed by the camera control controller. In step 1320,
Based on the view conversion matrix set in step 1310, the color-coded model is drawn in the back buffer of the graphic frame buffer 340. In this drawing, perspective projection and hidden surface removal processing are performed. Since the drawing is performed in the back buffer, the drawn result is not displayed on the display 10. When the drawing is completed, the pixel value of the back buffer corresponding to the event position is read (step 1330). The pixel value is the ID color of the object projected at the event position. The ID color has a one-to-one correspondence with the object, and the object can be identified.

A method of identifying an object based on a two-dimensional model will be described with reference to FIGS. 18 to 24. Two
The dimensional model defines the position and shape of the object after being projected from the world coordinate system to the screen coordinate system. When the orientation and the angle of view of the camera change, the position and shape of the object projected on the screen coordinate system change. Therefore, in the two-dimensional model, it is necessary to have data on the shape and position of the object for each camera work. In this embodiment, the object is modeled by a rectangular area. That is, an object in a camerawork is represented by the position and size of the rectangular area in the screen coordinate system. Of course, you may model by using other figures (for example, a polygon or a free curve).

FIG. 18, FIG. 19 and FIG. 20 show the correspondence between the camerawork and the two-dimensional model. The left view of each drawing shows the display form of the image display area 200 for each camera work. In addition, the right diagram of each figure shows a two-dimensional model of an object corresponding to each camera work. In the left diagram of FIG. 18, objects 410, 412, 414 and 414 on the image
Numerals 416, 420, 422, and 424 are rectangular areas 710, 712, 714, and 71 in the two-dimensional model shown on the right.
6,720,722,724. Corresponding to one camera work, a set of rectangles modeling an object is called an area frame. The area frame 1 corresponding to the camerawork 1 is rectangular areas 710, 712, 71.
4, 716, 720, 722, 724.
19 and 20 show examples of area frames for different camera works. In FIG. 19, the area frame 2 corresponding to the camera work 2 is a rectangular area 740, 742, 74.
It is composed of 6,748. Rectangular areas 740, 742
746 and 748 are objects 412 and 41, respectively.
6, 424, 422. Similarly, in FIG. 20, the area frame 3 corresponding to the camera work 3 is composed of a rectangular area 730. The rectangular area 730 corresponds to the object 400. Even the same object corresponds to another rectangular area if the camera work is different. For example,
The object 416 corresponds to the rectangular area 716 in the case of camerawork 1 and corresponds to 742 in the case of camerawork 2.

The data structure of the two-dimensional model is shown in FIGS. 22, 23 and 24. In FIG. 22, reference numeral 1300 is a camera data table that stores data corresponding to each camera. The camera data table 1300 stores data of camera work that can be operated on an object in a video and data of a region frame corresponding to each camera work.

In FIG. 23, 1320 is camerawork data. The camerawork data includes a horizontal angle that is the horizontal direction of the camera, a vertical angle that is the vertical direction of the camera, and an angle of view that indicates the degree of zooming. Here, it is assumed that the posture of the camera and the position of the camera are fixed. When the posture of the camera and the position of the camera can be remotely controlled, data for controlling them can be added to the camera work data 1320. The camerawork data 1320 is used to set the camera to a predefined camerawork. That is, the man-machine server 20 sends the camera work data to the camera control controller to remotely operate the camera. The camerawork data 1320 is not directly necessary for the object identification process.

FIG. 24 shows the data structure of the area frame. The area frame data is composed of the number of areas forming the area frame and data on each rectangular area. The area data includes the position of the rectangular area in the screen coordinate system (x,
y), the size of the rectangular area (w, h), the active state of the object, the action, and additional information. The active state of an object is data indicating whether the object is active or inactive. When an object is inactive, it is not identified. Only active objects are identified. A pointer to the event / motion correspondence table 1340 is stored in the motion. In the event / action correspondence table 1340, the object is PD
The action to be performed when specified by is stored in pairs with the event. Here, the event is PD
The operation type is designated. For example, the event is different when the pressure-sensitive touch panel 12 is strongly pressed and when it is lightly pressed. When an event occurs, the object at the event location is identified and the action paired with the event / action pair defined in the object that matches the event that occurred occurs.
The additional information of the area frame stores a pointer to additional information 1350 of the object that cannot be represented only by the rectangular area. There are various types of additional information. For example, there is a text drawn on an object, a color, an object name, and related information (for example, a device manual, maintenance information, design data, etc.). This allows the object to be searched for based on the text drawn on the object and the related information of the specified object to be displayed.

FIG. 21 shows a procedure for identifying an object using a two-dimensional model. First, the area frame corresponding to the current camera work is set to the camera data table 130.
Search from 0 (step 1200). Next, the area including the event position is searched from the areas forming the area frame. That is, the position and size data of each area stored in the area frame data is compared with the event position (step 1220), and when the area at the event position is found, the number is returned to the upper processing system. The upper processing system checks whether or not the found area is in the active state, and if it is in the active state, executes the operation defined corresponding to the event. Step 1220 is repeated until either a region containing the event location is found or all regions in the region frame have been examined (step 1210).

The two-dimensional model is defined using a two-dimensional model definition tool. The two-dimensional model definition tool has the following functions.

(1) Camera selection function A function of selecting an arbitrary camera arranged in the plant and displaying an image from the camera on the screen. There are the following methods for selecting the camera.

By designating an object on the layout plan of the plant displayed on the screen, the camera that projects the object is designated.

Specify the location of the camera on the plant layout displayed on the screen.

Designate an identifier such as a camera number or name.

(2) Camera work setting function A function of remotely controlling the camera selected by the camera selection function to set the orientation and angle of view of the camera.

(3) Graphic drawing function A function of drawing a graphic on the image displayed on the screen. Drawing is done by combining basic graphic elements such as rectangles, circles, polygonal lines, and free-form curves. With this function, the image of the object is used as the underlay and the outline of the object is drawn.

(4) Event / action pair definition function A function of designating at least one figure drawn by the figure drawing function and defining an event / action pair for it.
The event is defined by selecting a menu or entering the event name as text. The operation is described by selecting a predefined operation from a menu or using a description language. As such a description language, for example, the description language UIDL described in “User Interface Construction Support System with Meta User Interface” described in Information Processing Society of Japan, Volume 30, No. 9, pages 1200 to 1210. To use.

The two-dimensional model definition tool creates a two-dimensional model by the following procedure.

Procedure 1: Designation of camera and camera work (1) Select a camera using the camera selection function,
Display the image on the screen. Next, using the camera work setting function (2), camera work is set and an image of a desired place is obtained.

Step 2: Defining the outline of the object The outline of the object to be defined as an object in the object in the image displayed in the procedure 1 is drawn by using the (2) graphic drawing function.

Procedure 3: Define a pair of event and action Procedure 2 is performed using the event / action pair definition function (4) above.
Select at least one figure drawn in, and define a pair of event and action.

Step 4: Saving the definition contents The definition contents are saved if necessary. Definition contents are shown in Figure 22,
It is stored in the data structure shown in FIGS. When it is desired to create a two-dimensional model for another camera or another camera work, steps 1 to 4 are repeated.

By having an object model, it is possible to know where and how an object is displayed in the image. As a result, it is possible to display information related to the object on the basis of the position and shape of the object in the image, and to search the image of the object. An example is given below.

The object name, function, operation manual, maintenance method, etc. are compositely displayed on or near the object. For example, issuing a command will display the name of each device in the video.

An object created by graphics is synthesized and displayed on a live-action image as if it were actually viewed by a camera.

Searching the additional information of the object based on the input keyword, and setting the camera and camera work so as to reflect the corresponding object.

The internal structure of the object that cannot be displayed by the camera is composite-displayed on the object in the video. For example, the state of the water flow in the pipe is simulated based on the data obtained from other sensors, and the result is synthetically displayed on the pipe shown in the live-action image. Similarly, graphics showing the state of the flame in the boiler (for example, the temperature distribution map created based on the information obtained from the sensor) are composite-displayed on the boiler in the image.

An object to be noticed at present is specified by graphics. For example, when an anomaly is detected by the sensor, graphics are composited and displayed on the object in the image. Also, graphics are combined with the object in the image related to the data displayed in the trend graph so that the relationship between the data and the object in the image can be immediately understood.

[0091]

According to the present invention, (1) a remote driving monitoring system
On the stem, etc., the operator can directly set the operation target and the operation result.
I can grasp it sensibly. Also, (2) Remote operation monitoring system
It is possible to reduce size and save space. Change
(3) Advantages of camera images and graphics
Can be used to make up for the shortcomings.

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[Brief description of drawings]

FIG. 1 is an overall configuration of a plant operation monitoring system according to the present invention.

FIG. 2 is a hardware configuration of a man-machine server.

FIG. 3 is a screen configuration example of a plant operation monitoring system according to the present invention.

FIG. 4 is a screen display form of a drawing display area.

FIG. 5 is a diagram showing a correspondence between a screen display form of a video display area and a site.

FIG. 6 is a diagram showing an example of camera work setting by object designation.

FIG. 7 is a diagram showing an example of camera work setting by object designation.

FIG. 8 is a diagram showing an example of a button operation by specifying an object.

FIG. 9 is a diagram showing an example of a slider operation by specifying an object.

FIG. 10 is a diagram showing an example of an operation by graphic selection.

FIG. 11 is a diagram showing an explicit example of an operable object.

FIG. 12 is a diagram showing an example of video search using a search key.

FIG. 13 is a diagram showing an example of a three-dimensional model.

FIG. 14 is a diagram showing a relationship between a three-dimensional model and an image on a screen.

FIG. 15 is a diagram showing the relationship between points on the screen and objects.

FIG. 16 is a diagram showing a procedure of object identification processing using a three-dimensional model.

[Fig. 17] Designating an object in a video and selecting its object
It is in the function to execute the operation based on the project. This machine
Flow chart of the program that realizes Noh .

FIG. 18 is a diagram showing a relationship between a two-dimensional model and camera work.

FIG. 19 is another diagram showing a relationship between a two-dimensional model and camera work.

FIG. 20 is another diagram showing the relationship between the two-dimensional model and camera work.

FIG. 21 is a diagram showing a procedure of object identification processing using a two-dimensional model.

FIG. 22 is a diagram showing the structure of a camera data table.

FIG. 23 is a diagram showing a structure of camera work data.

FIG. 24 is a diagram showing a data structure of a region frame.

[Explanation of symbols]

10 ... Display, 12 ... Pressure sensitive touch panel, 14 ...
Speakers, 20 ... Man-machine server, 30 ... Switcher, 40 ... Keyboard, 50 ... Control computer, 60 ... Video camera, 62 ... Camera control controller, 64 ...
Microphone, 90, 92 ... Sensor, 94, 96 ... Actuator, 100 ... Display screen, 130 ... Drawing display area, 15
0 ... data display area, 200 ... video display area, 600 ...
Text search sheet.

─────────────────────────────────────────────────── ─── Continued Front Page (72) Inventor Koichiro Tanikoshi 4026 Kuji Town, Hitachi City, Hitachi, Ibaraki Prefecture Hitachi, Ltd.Inside Hitachi Research Laboratory (72) Masayasu Futagawa 4026 Kuji Town, Hitachi City, Ibaraki Hitachi Ltd. Hitachi, Ltd. Inside the laboratory (72) Shinya Tanito 4026 Kuji-cho, Hitachi, Hitachi, Ibaraki Prefecture Hitachi, Ltd. Inside the Hitachi Research Laboratory (72) Inventor Atsuhiko Nishikawa 5-2-1 Omika-cho, Hitachi City, Ibaraki Hitachi Ltd. Omika Plant (56) References JP-A 61-187480 (JP, A) JP-A 61-1137 (JP, A) JP-A 60-194691 (JP, A) JP-A 62-136991 (JP, A) Kaihei 4-39691 (JP, A) JP 62-31272 (JP, A) JP 62-81887 (JP, A) JP 58-58686 (JP, B2) JP 60-48951 (JP, B2) (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06F 3/00 G06F 3/14-3/153

Claims (2)

(57) [Claims] 1. A device in a plant is photographed by a camera.
Display the image on the display and display position of the image of the device displayed on the display
The figure clearly indicates the operable direction of the device when is specified.
When a shape is composite-displayed on the image of the device and the display position of the composite-displayed figure is designated,
Processing to operate the device in the direction specified by the specified figure
A method of executing a process characterized by executing. 2. A camera for photographing a valve in a plant
Displayed on the display, the display position of the valve image displayed on the display
Diagram that clearly indicates the rotatable direction of the valve when is specified
The shape is composite-displayed on the valve image , and when the display position of the composite-displayed figure is designated,
The process of rotating the valve in the direction specified by the specified figure.
A remote valve operation method characterized by executing a process.