JPH0659646A - Sign displaying method and device on screen - Google Patents

Abstract

PURPOSE:To provide a plant monitoring control system or the like with a means for giving a sign on an apparatus on the site. CONSTITUTION:This device comprises a display screen 500 and the like having a sign putting ICON 530, sign ICON 700-703, pointer 501, plant configuration illustration 510, video image 520. It is thus possible to issue memorandums, messages and the like to add and see them in such a feeling as to put signs on apparatuses, resulting in improvement of work efficiency.

Description

Detailed Description of the Invention

[0001]

[Industrial application] Video images (hereinafter referred to simply as "images")
A man-machine interface (hereinafter, also simply referred to as “man-machine”) system that utilizes such a system, more specifically, an image input from a video camera (hereinafter, simply as “camera image”) A method and an apparatus for providing a means for associating preset various information with a subject constituting (sometimes referred to as).

[0002]

2. Description of the Related Art As a conventional plant monitoring method, for example, a memo is affixed to a device, an operation panel, a display, a system monitoring panel and the like existing on the site to call the attention of a plant user. At this time, various contents are written on the paper to which the adhesive is attached on the back side, and the paper is attached to the target device. For example, state the failure status of the equipment, precautions for use, indication that the test is in progress, and various changes on the paper.
These pieces of paper (memo) are used for contacting other workers, memorandums for themselves, and the like. Sticking a memo on a device in this way is called a "tag rack", and a memo is called a "tag". On the other hand, in plant supervisory control systems, remote control and centralization of supervisory control are in progress. Various equipment in the plant is centrally monitored and controlled from the operation room located at a remote point, and the operator performs most of the operation and monitoring work through the display in the operation room and the input device such as various commands. To do. In such a work form, the conventional bill holder is very inconvenient. This is because it is necessary to actually go to the site in order to put a tag on the device on the site or refer to a tag hung on the device on the site. In addition, there is a case in which the object to be hung up is only present in the display screen. For example, the conventional system monitoring board is being replaced by the system diagram display on the display screen, and in that case, it is also necessary to carry out the bill hanging on the system diagram displayed on the screen.
However, since the display content of the display is changed as necessary, even if a memo is pasted on the display screen, it does not mean that the memo is pasted on the display object on the screen. Hypert
ext '89 Proceedings (Pittsburgh, Pennsylvania, No
v. 5-7, 1989) ACM, New York, 1989, `` Inter Note: Extend '' on pages 365-378
ing a Hypermedia framework to support annotative c
ollaboration '' (Catlin, T., Bush, PE, Yankelovic
h, N.) discloses a technique for annotating a document displayed on the screen. This technique can be interpreted as a type of technique for electronically tagging a selected portion of a document displayed on the screen. However, the above-mentioned document only discloses a technique for tagging document data, and screen components excluding editing targets such as document data, for example, components of a man-machine interface displayed on the screen (menu,
No technology is disclosed for tagging objects such as icons, graphs, meters, etc., and subjects shown in video images.

[0003]

As described above, the conventional techniques have the following problems. First, when tagging equipment on site, it is necessary to actually go to the site. Similarly, when referring to tags on equipment on site, it is also necessary to actually go to the site, which makes workability easier. Extremely bad. Secondly, annotations, memos, etc. cannot be added to screen constituent elements other than the data to be edited (for example, document data) (that is, "tagging" cannot be performed). Therefore,
An object of the present invention is to solve the above-mentioned problems of the prior art and achieve the following items. That is, firstly, it is necessary to be able to refer to the tags placed on the equipment on the spot while in the operation room. Secondly, it is necessary to provide a device capable of tagging screen components other than the data to be edited. In particular, it is to be able to add notes, annotations, etc. to the subject shown in the video image.

[0004]

[Means for Solving the Problems] In order to solve the above problems, the following means can be considered. First, when a video image is displayed, necessary information is created in advance by at least one of a medium including audio and video, and the information is memorized as being related to a predetermined part in the video image of the subject. Alternatively, a screen tagging method may be considered in which graphics (tags) that clearly indicate that the previously created information is associated with the subject based on the stored information are displayed. Further, in the above method, a screen tagging method is conceivable in which, when the displayed tag is instructed by the instructing means, the information generated by any one of the previously created media including at least audio and video is output. Further, in the above method, a screen tagging method may be used in which a plurality of scenes of a subject are photographed, stored as video images, and a tag is applied in association with each scene. Further, when displaying a video image of a subject captured by the image capturing means, at least audio,
Necessary information is created in advance by any of the media including the image, and the information includes at least an imaging means number indicating which of the plurality of imaging means is used and an imaging means setting parameter including a setting position of the imaging means. A screen tagging device that remembers that there is a relation, and creates and stores graphics (tags) that clearly show that the previously created information is associated with the subject based on the stored information. A method is also possible. In this case, when a certain imaging means setting parameter is selected, a tag related to the imaging means setting parameter is displayed, and when the displayed tag is instructed by the instructing means, at least the sound and the image created in advance are displayed. A screen tagging method of outputting information by any of the included media is also conceivable. Further, when displaying a video image of a subject photographed by the image pickup means, necessary information is created in advance by at least one of a medium including audio and video, and the information is related to a designated area on the display image. A screen tagging method in which a graphic (tag) that clearly indicates that the previously created information is associated with the subject is created and stored based on the stored information. Can also be considered. In this case, when a designated area on a certain display image is selected, a tag related to the designated area on the display image is displayed, and when the displayed tag is instructed by the instructing means, it is created in advance. It is also conceivable that the screen tagging method for outputting information by any one of the above-described media including at least audio and video. Further, when a subject is photographed by the image pickup means and a video image of the subject is displayed, necessary information is created in advance by at least one of a medium including a sound and an image, and the information is predetermined in a video image of the subject. Screen tagging method for storing graphics indicating that the object is associated with the previously created information based on the stored information. That is, first, only the video image of the subject is displayed, and then, only when a part of the subject is instructed, it is determined whether or not a tag is put on the instructed part, and the tag is displayed. When hanging, a screen tagging method of outputting information related to the tag is also conceivable. Furthermore, the following means can be considered as an apparatus that realizes the above method. A screen tagging device comprising a display means for displaying a picked-up image and a tagging processing means, wherein the tagging processing means is an information creating means for creating necessary information, and the created information is predetermined. And a means for storing the captured image in association with a part of the captured image.
A screen tagging device having a function of clearly showing a graphic (tag) when an image associated with the information created by the information creating means exists on the display screen. The above-mentioned device further includes means for instructing a part of the captured image, and the display means for displaying the captured image indicates a graphic (tag) clearly indicated by the display means by means for instructing a part of the captured image In this case, a screen tagging device having a function of outputting the created information stored in association with the tag is also conceivable.

[0005]

[Operation] The operation will be described below. First, information (hereinafter referred to as "tag" or "memo") is created using various media such as text, graphics, voice, and video. Next, on the video image displayed on the screen, a subject or a region to be associated with the above information is designated. The means for making such correspondence is the designated subject, or
The information corresponding to the position on the image is stored. Next, on the currently displayed image, the subject with a tag, or
If there is an area, it is clearly indicated that the tag is hung by a method such as displaying the graphics on the video image in a composite manner. Further, when the designated location is designated by the means for clearly indicating the bill hanging state, the bill information is output. If the information is text, drawing, video image, etc., it is displayed on the screen, and if it is sound, the necessary information is output from the speaker or the like. Of course, even if the tagging state is not clearly indicated, if the tagged object or area is designated, as long as there is information to be referenced, information (items written on the tag) is output. As described above, the present invention provides a device and a method for tagging devices in the field while in the operation room and making it possible to refer to the tag applied, and for tagging screen components other than the data to be edited. This makes it possible to add notes, annotations, etc., to the subject displayed on the video image.

[0006]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A plant operation monitoring system according to an embodiment of the present invention will be described with reference to FIGS. The present invention is a power plant, a chemical plant, a steel rolling mill, a building,
It goes without saying that it can be applied to various remote operation monitoring systems such as roads and water and sewer systems. The tagging function in the present embodiment is a function of attaching a memo (hereinafter sometimes referred to as "tag") to a display object on the screen. The tag can be described using various media such as voice, video, text, and graphics. The bill holder has various uses such as a message to other operators and a memorandum for oneself.
Not only graphic objects on the screen but also objects displayed in video images can be hung. If the tagging is performed on the surveillance video on site, the actual tagging on site can be simulated and replaced. FIG. 1 shows an example of the overall configuration of a plant monitoring control system. Each operator has a workstation (hereinafter simply referred to as "W
Sometimes referred to as "S"), the operation of the plant is performed as necessary while monitoring the state of the plant. One workstation 10 used by operators for operation monitoring
0 is shown in the figure.

The workstation 100 comprises voice processing hardware 120, amplifier built-in speakers 121 and 122, and a microphone 123 for inputting, storing, and reproducing voice. The audio processing hardware 120 has an A / D conversion function for converting an analog audio signal into a digital signal and a D / D conversion function for converting a digital audio signal into an analog signal.
It has an A conversion function and a signal processing function for compressing / decompressing a digital audio signal. The voice signal from the microphone 123 or the microphone 211 at the site where the plant is present is converted into a digital signal by the voice processing hardware 120, and further compressed, and the workstation 10
It is recorded in the memory of 0, a disk, or the like. Conversely, the recorded digital voice is expanded by the voice processing hardware 120, A / D converted, and then output from the speakers 121 and 122. The microphone 123 is mainly used by an operator for recording his / her own voice and inputting an operation command. The microphone 123 is arranged at a position where an operator can easily input a voice. If voice input is desired from a location remote from the workstation 100, a wireless microphone may be used, for example. The image composition hardware 110 in FIG. 1 is a means for combining and displaying the image signal of the workstation 110 and the television signal. The camera 111 is for the operator to mainly record his or her image. As a device for inputting a television signal to the video composition hardware 110, a camera 111, a group of cameras (for example, a camera 210) that monitors a site where a plant exists, a laser disc player (LD) 220, a video tape recorder (VTR) 22.
1st grade is considered. The video signal of the workstation 110 has a higher resolution than these television signals. Therefore, the two cannot be combined as they are. Therefore, the video compositing hardware 110 converts a television signal having a low resolution so as to match the resolution of the workstation 100, and then synthesizes the two. The video composition hardware 110 also has a so-called chroma key function. The chroma key function is a function of imposing a video signal only on a part of a specified color in the video signal of the workstation 110. For example, if "black" is specified as the chroma key, the video signal is displayed only in the black portion on the screen. The video signal is not displayed in areas other than black,
As a result, the image of the scene taken by the camera 210 is displayed on the display screen of the workstation 100, and the workstation 1 is displayed on the image.
The graphics drawn by 00 can be combined and displayed. In this system, it is the same as the workstation 100 so that multiple operators can operate in cooperation with each other.
Alternatively, workstations having equivalent functions are connected to each other via the network 10. The states of various devices in the plant (including tagging information, of course) are transmitted to each workstation via the network 10. On the contrary, the operation command from each workstation to each device is also transmitted via the network 10.
Such a network can be easily configured with an information transmission medium such as an optical fiber. The workstation 100 has a local driver 130 and a shared media server 1400 as software for accessing various media devices. Here, as the media device, general devices (camera, LD, VTR, CD-ROM driver (CD), etc.) for inputting, recording, processing, and outputting video, audio, etc. can be considered. Local driver 1
30 and the shared media server 140 are both communication ports (RS232C, SCSI) of the workstation 100.
Etc.) to control each media device. The local driver 130 is a media device connected to each workstation (a media device that is not shared with other workstations, for example, the video composition hardware 110, the audio processing hardware 120, the CD-ROM driver 131, etc.).
It is the software that controls and is installed on each workstation. Shared media server 1
Reference numeral 40 is software that controls media devices shared by a plurality of workstations. For example, shared media server 140 is installed only on workstation 100. Other workstations are connected to the shared media server 1 via the network 10.
40 to access the shared media device.
The shared media device includes a camera (for example, a camera 210 or the like), a microphone (for example, a microphone 211 or the like), an LD 220, a VTR 22 arranged at various places on the site where the plant exists.
1. Matrix switcher 20 for data switching
0, control switcher 250, camera controller 25
There are 1, 252 and so on. The camera 210, the microphone 211, and the like are provided to visually and aurally check the inside of the plant from the operation room. Data matrix switcher 200
Is a switching means that can be connected in an arbitrary matrix state in an M input to N input port configuration using each of an audio signal and a video signal as an input signal. Therefore, the data matrix switcher 200 can distribute the signal from one input terminal to a plurality of output terminals. Thereby, for example, the image of the camera 210 can be distributed to a plurality of displays and can be viewed from each display.

The LD 220 stores manual video information. The VTR 221 is connected to both the input side and the output side of the switcher 200. As a result, the image of the camera monitoring the site and the image of the camera 111 can be recorded, and the recorded image can be output to the display screen of the workstation 200. The operator can use the camera 111 to capture an image of his / her talking and record the image in the VTR 221, thereby transmitting the message by the image to other operators. Of course, instead of recording the video in the VTR 221, it may be digitized and then compressed and stored in the memory, disk, or the like of the workstation in the same manner as the audio. Further, in FIG. 1, camera controllers 251 and 252 are means for controlling a camera (camera 210 or the like) installed on the site, and a control switcher 250 is installed on the site together with the camera controllers 251 and 252. It is a means to switch the connection with the camera. Camera controller 25
1, 252 and the control switcher 250 are connected to the communication port of the workstation 100, respectively.
The camera controllers 251 and 252 are means for freely setting the orientation and the angle of view of the cameras connected via the control switcher 250. Shared media server 140
Is the client process (C
P1) to a camera (eg, camera 2) on site
10)), the control right of the camera 210 is assigned to CP1 in the following procedure. First, the first procedure will be described. It is checked whether or not the control right of the camera 210, which has been requested to be controlled, is assigned to another client process. If the control right has already been assigned, the camera 210 informs the CP 1 that the camera is in use, and ends the control right assigning process. When the control right is not assigned, the procedure proceeds to the second procedure. The second procedure searches for a camera controller that is not used by another client process. When all the camera controllers are in use, the fact is notified to CP1 and the control right allocation processing is ended. If there is a camera controller that is not in use, assign it to CP1. Finally, in step 3, a command is output to the control switcher 250 to connect the controller assigned to CP1 (referred to as the controller 251) and the control line to the camera 210. After the control right is obtained by the above procedure, for example, C
When there is a request from P1 to pan the camera 210 to the left (referred to rotating the camera horizontally), the shared media server 140 sends an operation command for panning left to the camera controller 251. Upon receiving the operation command, the camera controller 251 drives and controls the motor of the pan head (referred to as a mounting base of the camera including the drive unit) of the camera 210 to pan the camera 210 to the left. C
When P1 finishes using the camera 210, P1 notifies the shared media server 140 of the fact and releases the control right of the camera 210. By controlling the cameras in the above-described procedure, it is possible to control an arbitrary camera from a plurality of client processes with a camera controller that is smaller in number than the number of cameras installed in the field. In other words, it is only necessary to prepare camera controllers for the number of cameras to be operated simultaneously. FIG. 2 shows an example of the monitor screen displayed on the display screen of the workstation 100. In FIG. 2, a display screen 500 of the workstation 100 is a plant configuration diagram 510, a graphic display of control data and the like, a camera, an LD 22.
0, a video image 520 displaying a video signal from the VTR 221, a tag icon 530, a tag menu 5
31 and a pointer 501 displayed in conjunction with a pointing device such as a mouse. In this embodiment, a mouse is used as an example of the pointing device. The mouse is provided with two buttons, which are called a left button and a right button, respectively. In particular, when it is not necessary to distinguish between the left and right buttons, they will be simply referred to as buttons. Of course, other pointing devices such as a tablet, a touch panel, and a trackball may be used instead of the mouse.

In the present embodiment, the following operation is performed using a mouse. The operation of "clicking" is an operation of pressing the button of the mouse and releasing the button that has been pressed without moving the position of the mouse. This operation is mainly used for designating a position within the screen 500, designating a display object, and the like. The operation of "drag" is an operation of pressing down the mouse button and moving the mouse as it is. The drag operation ends by releasing the button that was being pressed. 2 on the screen 500 by dragging
You can specify points. That is, there are two points: the position of the pointer 501 at the start of dragging (when the button is pressed down) and the position of the pointer 501 at the end of dragging (when the button is released). Operate the mouse to display on the screen 500
Placing the pointer 501 and clicking (or dragging) the mouse is referred to as clicking (or dragging) the display object. The billing menu 531 has, for example, “pulling sheet”, “link”, “icon display”, “search”, “cut”, “paste” and the like as selection items.

The function of each item is as follows. The item "tagging sheet" is a process of opening a tagging sheet which is a tool for creating and editing tags. Item "link"
Is a process of associating a bill with an object (referred to as a link). The item "icon display" is a process of displaying an icon (referred to as a bill icon) indicating that the bill is hung. When this item is selected, the item name changes to "Hide icon". If you select "Hide Icon",
The display of the tag icon disappears and the item name changes to "icon display". The item "search" is a keyword search process based on the content of the tag. When a keyword is input, the tag containing the keyword and the object to which the tag is linked are displayed. The item “cut” is a process of deleting the tag corresponding to the currently selected tag icon and storing the data of the tag in a kind of buffer called a paste buffer. The item "paste" is a process of linking the tag stored in the paste buffer to the currently selected object. The following outlines a standard procedure for hanging objects on the screen. First, select the item “Payment sheet” of the panel menu 531 to open the panel sheet 600 shown in FIG. The billing menu 531 is called by clicking the billing icon 530.

To select an item on the billing menu 531, a desired item may be clicked on with a mouse. (The above is "Procedure 1") Next, a bill is created and edited using the bill hanging sheet 600. The tag includes graphic (including text) data, voice data, video data, and the like. The bill hanging sheet 600 has a graphic editing function, a voice recording function, a video recording function, and the like in order to edit the data. (The above "procedure 2") Next, the created bill is saved and the bill hanging sheet 600 is ended. When the billing sheet 600 is closed, the bill icon is displayed at the default position on the screen 500.

Next, the created tag is linked to the object. Drag the displayed tag icon to move it over the object you want to tag, and click the tag menu 53.
When the item of "link" of 1 is selected, the tag is linked to the object. (The above "procedure 3") In the above procedure, the objects include the following. First, the components of the graphics display 703. Examples include user interface tools such as graphics, menus, icons, graphs, images and the like. Further, a group of a plurality of constituent elements can be an object. Next, the subject of the video image 520 and the devices and objects shown in the video image are also included. In addition, the video image itself is also included. Each scene of the video image can also be an object. If the camera parameter, that is, the shooting direction of the camera (changes by a pan operation or the like), the angle of view (changes by a zoom operation), or the like changes, the target scene also changes. A scene corresponding to a certain camera parameter also becomes one object (referred to as a scene object). Further, a video image stored in the LD 220, VTR 221, etc. at a certain time also becomes a scene object. The scene object shot by the camera is called a live video object, and the scene object of the stored video image is called a stored video object. Therefore, an area on the video image (scene object) and an area (including points) in the scene object can also be objects. The area can be defined by, for example, the shape of a figure. Next, with reference to FIG. 3, a method of creating and editing a tag using a tag-hanging sheet will be described.
In FIG. 3, an area 610 for inputting and displaying a tag title, an area 611 for inputting and displaying the name of the creator of the tag, an area 612 for inputting and displaying the address of the tag, and a date and time display at the time of creating the tag 613, figure editing area 620, figure editing area 62
Scroll bars 621 and 622 for scrolling the drawing drawn in 0 to the left, right, up and down, a figure menu 625 for selecting a figure object to be drawn in the figure editing area 620, and enlarging or reducing the drawing drawn in the figure editing area 620. A menu 626 for displaying, a menu 630 for specifying an object (audio or video) to be recorded or reproduced, an icon 631 for controlling recording or reproduction of audio or video.
~ 635, management menu 64 for managing bill data
0, a list display 650 of object names to which the currently created tag is linked, and a scroll bar 651 for scrolling the list display 650 are shown. First, the items and functions of the graphic menu 625 will be described below. Item "character"
Is a function for drawing a character. After selecting this item,
When one point in the edit area 620 is clicked with the mouse and characters are input from the keyboard, the input character is drawn from the clicked position. When the pointer 501 is moved to a place other than the character string currently being input and the mouse is clicked, the character string input ends.

The item "straight line" is a function for drawing a straight line. When this item is selected and the mouse is dragged in the editing area 620, a straight line connecting the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging is drawn. The item “rectangle” is a function of drawing a rectangle. After selecting this item, if you drag the mouse in the edit area 620, the pointer 50
A rectangle whose diagonal vertices are the position 1 and the position of the pointer 501 at the end of dragging is drawn. Item "oval"
Is a function for drawing an ellipse. After selecting this item,
When the mouse is dragged in the editing area 620, an ellipse inscribed in a rectangle whose diagonal is the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging is drawn. The item “polyline” is a function for drawing a polyline. After selecting this item, repeatedly moving and clicking the mouse in the editing area 620, and finally clicking twice at the same position, the pointer 50 when clicked
A polygonal line formed by connecting the positions of 1 with a straight line is drawn. The item “polygon” is a function of drawing a polygon.
After selecting this item, repeatedly moving and clicking the mouse in the editing area 620, and finally clicking twice at the same position, the position of the pointer 501 at the time of clicking is sequentially connected by a straight line, and the last point and the first point. A polygon is created by connecting the points of with a straight line. The item “handwriting” is a function for drawing a line by freehand. When the mouse is dragged, a line connecting the passing points of the pointer 501 is drawn. Edit area 62
The figure drawn on 0 can be selected and moved using the mouse. When the mouse is moved to move the pointer 501 to a desired figure and the left button is clicked, the figure is selected. When the mouse is dragged in the editing area 620, all the figures included in the rectangular area whose diagonal vertices are the position of the pointer 501 at the start of dragging and the position of the pointer 501 at the end of dragging are selected. become.
On the other hand, when the mouse is moved to move the pointer 501 to a desired figure, the left button is pressed down and the mouse is dragged, the figure moves together with the pointer 501. When the right button of the mouse is pressed while the pointer 501 is on the edit area 620, a graphic object edit menu (not shown) is displayed. With the right button held down, move the mouse to move the pointer 501 to the desired item in the graphic object edit menu, and release the right button to select the desired item. When the right button is released, the display of the graphic object edit menu disappears.
The items and functions of the graphic object edit menu are shown below. The item “delete” is an instruction to delete the selected figure and at the same time store the figure in the paste buffer. The item “copy” is a command to copy the selected figure to the paste buffer. The item "Paste" is
This is an instruction to draw the contents of the paste buffer at the position of the pointer 501 when the mouse was most recently clicked.
The item “rotate” is an instruction to rotate the figure selected by a desired angle around the desired position. The item “enlarge / reduce” is an instruction for enlarging or reducing the figure selected to have a desired size. The item “composite” is an instruction to group a plurality of selected figures. The grouped figures can be edited and operated as one figure. The item “decomposition” is a command to decompose the grouped figures into individual figures. Next, with reference to FIG. 3, a method for leaving a memo by voice using a tag will be described. First, the item “voice” of the menu 630 is selected. Next, when the icon 635 is clicked, the display of the icon 635 is highlighted. The highlighted display of the icon 635 indicates that the audio recording mode or the video recording mode is set. When the voice recording mode is entered, the bill hanging sheet 600 requests the shared media server 140 to connect the output port of the microphone 123 and the input port of the voice processing hardware 120. Upon receiving the request, the shared media server 140 performs the switching process of the data switching matrix switcher 200 via the communication port provided in the workstation 100. Next, when the icon 633 is clicked, recording is started. If you speak into the microphone 123, it will be recorded as the voice data of the bill. The voice input from the microphone 123 is so-called A / D converted by the voice processing hardware 120, further compressed and recorded in a memory (not shown), a disk (not shown), etc. in the workstation 100. . To end recording the voice, click the icon 632. When the recording is finished, the highlighting of the icon 635 is canceled and the playback mode may be entered. When confirming the recorded contents, first, the icon 631 is clicked to move the pointer to the beginning of the recorded data. Here, the "pointer" means a position on the data where recording or reproduction is started. When the icon 633 is clicked, the recorded voice data is reproduced from the position of the pointer. That is, the audio data stored in the memory, the disk or the like is read out from the position of the pointer, the audio processing hardware 120 performs the expansion processing and the D / A conversion processing, and the audio is output from the speakers 121 and 122. It Furthermore, in the case of additionally recording voice, first, the icon 634 may be clicked to move the pointer to the end of the recorded data. Next, if the icons 635 and 633 are sequentially clicked, additional recording will be started. Next, we will explain how to use a video on a tag to leave notes. First, the item “video” of the menu 630 is selected. Next, when the icon 635 is clicked, the display of the icon 635 is highlighted. At this time, the billing sheet 600 connects (1) the output port of the microphone 123 and the input port of the audio processing hardware 120 to the shared media server 140, and (2) the output port of the camera 121 and image composition. Hardware 110 input port and V
Request to connect with the input port of TR221.
Upon receiving the request, the shared media server 140 transmits the data switching matrix switcher 200 via the communication port provided in the workstation 100.
Connection process. Next, when the icon 633 is clicked, recording will be started. At this time, the bill hanging sheet 600 is transferred to the shared media server 140 by the VTR 22.
Request to start recording 1. Upon receiving the request, the shared media server 140 instructs the VTR 221 to start recording via the communication port provided in the workstation 100. At the same time, a bill object (described later) is generated on the memory in the workstation 100, and the recording start address of the VTR 221 is stored in the bill information storage field of the bill object. The address is designated by a frame number, for example. Of course, designation by a time code or the like is also possible. Camera 111 and microphone 123
When the user speaks to, the video and audio of the speaker will be recorded in the VTR 221. To end the video recording, click the icon 632. When the recording is finished, the highlighted display of the icon 635 is canceled and the mode is changed to the reproduction mode.

At the end of recording, a recording end address is newly stored in the tag information storage field of the tag object generated at the start of recording. When confirming the recorded contents, first, the icon 631 is clicked to move the pointer to the beginning of the recorded data. That is, the shared media server 140 is requested to rewind the VTR 221 to the recording start address stored in the tag object. When the icon 633 is clicked, the reproduction of the VTR 221 is started from the rewound portion. Furthermore, when additionally recording video information, first, the icon 63
Click 4 to "fast forward" the VTR 221 to the beginning of the unrecorded portion of the videotape. Next, if the icons 635 and 633 are sequentially clicked, additional recording will be started. Now, the bill created through the above processing is managed using the menu 640. The main functions of the menu 640 are described below. "New" is a function for creating a new bill. "Open" is a function to open an existing tag. "Save" is a function for saving the currently edited tag. “End” is a function of ending the bill hanging sheet 600. With the list display 650, not only the objects currently being edited but with tags attached can be displayed, and the objects to be attached with tags can be specified and changed. The link display 650 has an ordinary text editing function, and can input an object name, change an already displayed object name, and modify it by using a mouse, a keyboard, or the like. Next, the procedure for referring to the tag will be described below. First, the item "icon display" of the tag menu 531 is selected to display the tag icon (procedure 1).
Of course, if the bill icon is already displayed, this procedure is unnecessary. Next, the tag icon of the tag to be referred to is clicked twice consecutively (this is referred to as "double click") (step 2). When double-clicked, the billing sheet 600 opens and the data of the double-clicked bill is displayed. That is, the title, creator, and address of the bill are displayed in the areas 610, 611, and 612, respectively. In addition, the names of all the objects to which the tags are attached are displayed in the list 650. If the tag has graphic data, the graphic data is displayed in the graphic editing area 620. If the card has audio data, the item "audio" of the menu 630 is selected, and if it has video data, the item "video" is selected.

If it has both data, the item "video" is selected by default (of course,
The item “voice” may be selected in the default state). When the play icon 633 is clicked, the data selected in the menu 630 is played. In step 2 above, the tag icon is double-clicked to open the tag hanging sheet 600. However, even if you click the tag icon once and then select the item "Tag hanging sheet" in the tag menu 531, The seat 600 can be opened. Further, the above procedure 1 is not always necessary.
In other words, even if the icon is not displayed, even if the object on which the tag is hung is directly clicked with the mouse and then the item “Clip sheet” of the tag menu 531 is selected, the tag sheet 600 is opened. be able to. FIG. 4 shows an example of the screen when the tag icon is displayed in the above procedure 1. In this figure, the tag icons 700, 70
1, 702 and 703 are illustrated. The bill icon is composed of a combination of three symbols, that is, a symbol representing paper, a symbol representing a speaker, and a symbol representing a person (circle and triangle). Each symbol represents the type of data that the tag has. The paper symbol indicates that it has graphic data, the speaker symbol has voice data, and the person symbol has video data. That is, in FIG. 4, the tag icon 70
0, 701, 702, and 703 mean that the corresponding tags have only audio data, only graphic data, graphic data and video data, and graphic data and audio data. The tag can be searched based on the title, the creator's name, the destination, the date, the tagged object name, the text entered in the graphic editing area 620, and the like. Tags that match the search conditions are displayed along with the suspended objects. For example, when a tag is placed on the subject of the video image 520, the tag is searched, and at the same time, the image of the subject is displayed on the video image 520. FIG. 5 is a conceptual diagram of an implementation method in the case of tagging a subject of a video image. In FIG. 5, the camera 7
Video footage 750, video footage 7 taken at 51
50, a modeling object 761 having information about what is shown, a graphic object 760 defining which area of the video image 750 the modeling object corresponds to, and a modeling object 761
Shows a tag object 762 linked to. The modeling object 761 defines what is shown in the video image. The graphic object 760 defines where on the video image 750 the modeling object 761 appears. Video footage of operator 7
When a position on the position 50 is designated, the graphic object at the position is identified (referring to checking what the designated designated object is), and the modeling object to which the graphic object belongs, that is, the subject is identified. To be done. If the camera parameters such as the shooting direction, orientation, and angle of view of the camera are changed, the shape, position, etc. of the subject, the subject on the image, changes. Of course, if multiple cameras are installed, the subject will be different for each camera. Therefore,
Depending on the camera and the camera parameters of that camera,
A graphic object 760, a modeling object, etc. are defined. In the case of accumulated video images,
Since the scene changes for each reproduction section, a graphic object 760, a modeling object, etc. are defined corresponding to the reproduction section. The tag is an area on the video image 750,
Graphic object 760, modeling object 76
You can link to any of the above. In this figure, the tag object 762 is linked to the modeling object 761. FIG. 6 shows a detailed data structure example of this embodiment. First, live video objects 800 and 801 defined for each scene will be described. One live video object corresponds to one scene. In addition, the live video object has the attributes described below. The "active state" indicates, by a binary flag, whether or not interactive operation (operation between the operator and the workstation) is possible. When "false" (for example, "0"), the interactive operation is not accepted. When "true" (for example, "1"), the interactive operation is accepted. Here, enabling interactive operation is referred to as “activating”, and disabling interactive operation is referred to as “inactivating”.
Further, this information is included in all objects. “Area information” is an attribute that specifies an area on the screen of the workstation 100. The video of the camera will be displayed in the area on the screen specified by the area information. “Camera information” is an attribute configured to include a camera identifier for shooting and camera parameters. The camera identifier identifies one of a plurality of cameras, and the camera parameters include information such as the position, angle of view, shooting direction (horizontal angle and vertical angle), and attitude of the camera when shooting a certain scene. Consists of. That is, the scene is determined by the camera information. The "object list" stores a pointer to a graphic object defined for a certain scene or a pointer to a tag object hung on a certain scene. "Additional information" is information other than the above related to the scene, for example, the type of camera,
Information such as maintenance history is stored. Instead of the live video object, the accumulated video object 840 shown in FIG. 7 corresponds to the scene of the accumulated video image. The stored video object 840 has video information attributes instead of camera information. The video information attribute includes an identifier of a video device used for reproduction and information about a reproduction section of the scene. In the reproduction section, a reproduction start address and an end address are designated by a frame number or a time code. Other attributes are the same as the live video object. Also, here, the live video object and the accumulated video object are collectively referred to as a video object. Figure 6
Includes a graphic object 810 defined for each scene.
~ 813 are also shown. The graphic object is data corresponding to various graphic elements such as a circle, a rectangle, a polygon, and a line. The outline and position of the subject are defined by one or more graphic objects. Such a graphic object has the following attributes. The "active state" indicates, by a binary flag, whether or not interactive operation (operation between the operator and the workstation) is possible. When "false" (for example, "0"), the interactive operation is not accepted. "true"
When (for example, “1”), the interactive operation is accepted.

When the live video object is activated, all graphic objects registered in the object list are also activated. “Area information” defines the area occupied by this graphic object on the screen of the workstation 100. When the area defined by this area information is designated, this figure object is selected. “Part-of” is a pointer to a modeling object in which a graphic object represents a region. When a graphic object is selected on the screen, par
The modeling object in which the pointer is stored in the t-of information is also selected. Generally part-
A pointer to the parent object is stored in of.
Here, the parent object refers to an object composed of a plurality of other objects (referred to as child objects). The "additional information" refers to information relating to figures other than the above, for example, information such as the type of figure, color, and presence / absence of painting. Modeling object 820
6 to 823 are shown in FIG. The modeling object is a conceptual object corresponding to the subject and has information about the subject. The modeling object is composed of at least one graphic object or another modeling object. For example, the burner object 820 is composed of a graphic object 810, the fuel pipe object 821 is composed of graphic objects 811, 821, and the boiler object 823 is composed of a burner object 820 and a fuel pipe object 821. When the operator points to a point on the image, a graphic object including the point in the area is identified, and a modeling object linked to the graphic object is identified. As described above, the link between the graphic object and the modeling object is part-of the graphic object.
It can be realized by storing the pointer to the modeling object in the attribute.

The modeling object has the following attributes. The "active state" indicates, by a binary flag, whether or not interactive operation (operation between the operator and the workstation) is possible. When "false" (for example, "0"), the interactive operation is not accepted. When "true" (for example, "1"), the interactive operation is accepted.

When a graphic object which is a child object is activated, a modeling object which is a parent object is also activated. "Tag object"
Is a pointer to a bill object. When a plurality of bills are hung on one modeling object, the bills are connected in a list. “Part-of” is a pointer to a modeling object that is a parent. In the example shown in FIG. 6, the boiler object is a parent object of the burner object 820 and the fuel pipe object 821, and a pointer to the boiler object 823 is stored in each part-of attribute. In the "additional information", information regarding the subject other than the above, for example, the burner object 820 has design information regarding the burner, manual information, maintenance information, and the like as additional information. Furthermore, the tag objects 830 and 831
Is shown in FIG. The bill object is a bill hanging sheet 600
It holds information created by and has the following attributes. The "active state" indicates, by a binary flag, whether or not interactive operation (operation between the operator and the workstation) is possible. When "false" (for example, "0"), the interactive operation is not accepted. When "true" (for example, "1"), the interactive operation is accepted.

When the tag object is activated, a tag icon is displayed on the screen. The “area information” is information that specifies the display area of the tag icon corresponding to the on-screen tag object of the workstation 100. The "card object" is a pointer to the subsequent card object. When one modeling object has a plurality of cards, this attribute is used to connect the card objects in a list. When there is no subsequent bill, for example, "0" is set. “Bill information” is information about the contents of the bill created by the bill hanging sheet 600, and has a pointer to graphic data, a start address and an end address of audio data, a start address and an end data of recorded data, and the like. Consists of An "invalid address" is stored in the data that has not been created. Here, the “invalid address” is an address that is not used as an address for actual data. "Additional information" is
Information other than the above, for example, the creator of the bill, the creation date and time, the reference count, the referrer, the reference date and time, and the like are included. FIG. 8 shows an example of a processing procedure for displaying a scene. Less than,
Each processing procedure will be described in detail. First, the video object corresponding to the designated scene is activated (step 1100).

Next, an instruction to activate is sent to all the objects registered in the object list of activated video objects. When the object registered in the object list is a graphic object, the active state is set to "true" in response to the activation command. The activated object is part-o
An instruction to activate is sent to the parent object specified by the f attribute, and the parent object is also activated. When the pointer to the bill object is registered in the bill object attribute, the activated modeling object sends a command to activate the bill object to activate the bill object. The activated bill object activates the bill object if a subsequent bill object exists (that is, if a pointer is registered in the bill object attribute). When the tag object receives the command to activate it, it not only sets the active state to "true" but also displays the tag icon on the screen (step 11).
10).

Next, scene display processing is performed separately depending on whether the designated scene is a live image from a camera or an accumulated image by the VTR 221, LD 220 or the like.
For live video, follow steps 1130 through 1 below.
The processing of 140 is performed, and in the case of the stored image, the processing of the following steps 1150 to 1160 is performed (step 1120). In step 1130, the shared media server 140 is requested to connect the camera specified by the camera information attribute of the live video object and the video compositing hardware 110. In step 1140, the shared media server 140 is requested to set the camera parameters specified by the camera information attribute of the live video object in the camera. In step 1150, if the specified scene is an accumulated video, the video device (VTR22) specified by the video information attribute of the accumulated video object.
1 or LD 220) and the video compositing hardware 110 are requested to the shared media server 140 and connected. At step 1160, the shared media server 140 is requested to
Instructs the video device to repeat playback between the start address and end address specified by the video information attribute of the stored video object. Lastly, the area information of the video object is called, the video display hardware 110 is instructed via the local media driver 130 about the display position of the video, and the composite display of the video on the screen of the workstation 100 is started. The image is displayed in a color different from the display color of the bill icon by combining using the chroma key function (step 1170). Accordingly, the tag icon displayed in step 1110 is displayed on the image. In the processing procedure shown in FIG. 8, when displaying an image, a tag icon is also displayed (see step 1110). Of course, the tag icon is not displayed at the time of image display, and at the time of a request from the operator (for example, when the item “icon display” of the tag rack menu 531 is selected),
A bill icon may be displayed. Further, in the data structure shown in FIG. 6, the video object and the modeling object can be linked with the bill object. That is, the video object is linked to the bill object by setting the pointer to the bill object in the object list attribute. The modeling object is linked to the bill object by setting a pointer to the bill object in the bill object attribute. When the tag object is linked to the video object, the tag icon is displayed only in the scene designated by the video object. When the tag object is linked to the modeling object, the tag icon is displayed in all the scenes in which the subject corresponding to the modeling object is displayed. Of course, objects other than video objects and modeling objects (graphic objects and other objects not shown in FIG. 6, for example, user interface tools such as menus and icons, graphs, images, etc.) are provided with tag object attributes, It may be configured so that it can be linked to the bill object. As described above, according to the present invention,
The following effects can be obtained. First, the operator can add notes to the equipment on site and refer to the notes while in the operation room. By associating and adding a memo to the device shown in the video of the surveillance camera, you can work as if you were actually hanging on the site. Second
Moreover, the memo can be managed by a computer such as a workstation or a personal computer. As a result, there are advantages that the reliability of storage and the like can be improved and the search speed can be increased, as compared with the paper tagging. Thirdly, it is possible to make a bill using not only text but also media such as audio and video. As a result, the memo can be easily recorded and recorded. Fourth, memos can be attached not only to on-site equipment but also to man-machine components on the screen. As a result, it is possible to convey the precautions for screen operation. Fifth, communication between operators can be promoted. That is, when the operator changes, the matters to be sent to the next operator are created in correspondence with the displayed objects on the screen, so that it is possible to make an easy-to-understand and sure offer.

[0022]

[Effects of the Invention] Information can be stored and referred to by linking to a subject appearing in a video image taken by a video camera, and tagging to a remote device can be simulated on the screen, so that the driver can operate in the operation room. While on the go, you can add memos to on-site equipment and refer to memos. Further, there are advantages such as improvement in reliability of information storage and speedup of information retrieval.

[Brief description of drawings]

FIG. 1 is an example of an overall configuration diagram of a plant monitoring control system according to the present invention.

FIG. 2 is an explanatory diagram of a display example on a display of the plant monitoring control system according to the present invention.

FIG. 3 is an explanatory diagram of a display example of a bill hanging sheet.

FIG. 4 is an explanatory diagram of a display example of a bill icon.

FIG. 5 is an explanatory diagram showing a concept of implementing tagging on a video image.

FIG. 6 is an explanatory diagram of an example of a data structure for tagging a video image.

FIG. 7 is an explanatory diagram of an example of a data structure of a stored video object.

FIG. 8 is an explanatory diagram of a processing example when a scene is displayed.

[Explanation of symbols]

10 ... network, 100 ... workstation, 1
10 ... Video synthesis hardware, 111 ... Camera, 120 ... Audio processing hardware, 121 ... Speaker, 122 ... Speaker, 1
23 ... Microphone, 130 ... Local media driver, 1
40 ... Shared media driver, 200 ... Data switching matrix switcher, 210 ... Camera, 211
… Microphone, 220… Laser disk (LD), 221…
Video tape recorder (VTR), 250 ... Control switcher, 251, ... Camera controller, 252 ... Camera controller, 500 ... Display screen, 501 ... Pointer, 510 ... Plant configuration diagram, 520 ... Screen 500
Video image displayed on the screen, 530 ... Billboard icon, 5
31 ... Billing menu, 600 ... Billing sheet, 610
… Title title input and display area, 611… Title creator name input and display area, 612… Title address input and display area, 613… Date and time when bill is created , 620 ... (Figure) editing area, 621 ... Scroll bar, 622 ... Scroll bar, 625 ... Figure menu, 626 ... Menu for enlarging or reducing, 6
30 ... Menu for specifying recording / reproducing target, 631 ...
Icon, 632 ... Icon, 633 ... Icon, 63
4 ... Icon, 635 ... Icon, 640 ... Management menu, 650 ... Object name list display, 651 ... Scroll bar, 700 ... Tag icon, 701 ... Tag icon, 702 ... Tag icon, 703 ... Tag icon, 750
... video image, 751 ... camera, 760 ... graphic object, 761 ... modeling object (boiler object), 762 ... bill object, 800 ... live video object, 801 ... live video object, 810 ... graphic object, 811 ... graphic object, 812 ... graphic object, 813 ... graphic object, 820 ... modeling object, 821 ... modeling object, 822 ... modeling object, 823 ... modeling object, 830 ... bill object, 831 ... bill object, 840 ... accumulated video object

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Masayasu Futagawa 4026 Kuji Town, Hitachi City, Ibaraki Prefecture Hitate Manufacturing Co., Ltd.Hitachi Research Laboratory (72) Masato Hotta 4026 Kuji Town Hitachi City, Ibaraki Prefecture Nitate Manufacturing Co., Ltd. Hitachi Research Laboratory (72) Inventor Shinya Yato 4026 Kuji-machi, Hitachi City, Ibaraki Prefecture Hitachi Co., Ltd. Hitachi Research Laboratory (72) Inventor Atsuhiko Nishikawa 5-2-1 Omika-cho, Hitachi City, Ibaraki Hitachi, Ltd. Inside the Omika Plant (72) Inventor Atsuhiko Hirota 5-2-1 Omika-cho, Hitachi City, Ibaraki Prefecture Hitachi Ltd. Inside the Omika Plant (72) Inventor Harumi Uchigasaki 4026 Kuji Town, Hitachi City, Ibaraki Nitate Manufacturing Co., Ltd. Hitachi Research Laboratory

Claims (10)

[Claims] 1. When displaying a video image, necessary information is created in advance by at least one of a medium including a sound and an image, and the information is related to a predetermined portion in a video image of the subject. A screen tagging method characterized by storing and displaying graphics (tags) that clearly indicates that the previously created information is associated with the subject based on the stored information. 2. The screen according to claim 1, wherein when the displayed tag is instructed by the instructing means, the information produced by any one of the media created at least including audio and video is output. Billing method. 3. When displaying a video image of a subject photographed by the image pickup means, necessary information is created in advance by at least one of a medium including a sound and an image, and the information is stored in at least a plurality of image pickup means. It is stored that there is a relation with the image pickup unit setting parameter including the image pickup unit number and the setting position of the image pickup unit that indicates which one is used, and based on the stored information, the previously created information is stored in the subject. A screen tagging method characterized by creating and storing graphics (tags) that clearly indicate that they are associated. 4. A method according to claim 3, wherein when a certain image pickup means setting parameter is selected, a tag associated with the image pickup means setting parameter is displayed, and when the displayed tag is instructed by the instructing means, it is created in advance. A screen tagging method, characterized in that the information is output by any one of the recorded media including at least audio and video. 5. When displaying a video image of a subject photographed by the image pickup means, necessary information is created in advance by at least one of a medium including audio and video, and the information is designated on the display image. Remember that it is related to the area,
A screen tagging method characterized by creating and storing graphics (tags) that clearly shows that the previously created information is associated with the subject based on the stored information. 6. The apparatus according to claim 5, wherein when a designated area on a certain display image is selected, a tag associated with the designated area on the displayed image is displayed, and the displayed bill is designated. The screen tagging method is characterized in that, when instructed by, the information generated by any one of the previously created media including at least audio and video is output. 7. The screen tagging method according to claim 1 or 2, wherein a plurality of scenes of a subject are photographed, accumulated as video images, and a tag is applied in association with each scene. 8. When a subject is photographed by an imaging means and a video image of the subject is displayed, necessary information is created in advance by at least one medium including audio and video, and the information is used as a video image of the subject. A screen for displaying that graphics (tags) that clearly indicate that the subject is associated with the previously created information based on the stored information that is related to a predetermined portion therein is stored. A method of tagging, in which only a video image of a subject is first displayed, and then, only when a part of the subject is instructed, it is determined whether or not a tag is placed on the instructed part. A screen tagging method, wherein when a tag is hung, information related to the tag is output. 9. A screen tagging device comprising a display means for displaying a picked-up image and a tagging processing means, wherein the tagging processing means is an information creating means for creating necessary information. And a means for storing the captured information in association with a part of a predetermined captured image, and the display means for displaying the captured image is associated with the information created by the information creating means on the display screen. A screen tagging device characterized by having a function of clearly indicating a displayed image by a graphic (tag) when the image is present. 10. The device according to claim 7, further comprising means for instructing a part of the captured image, wherein the display means for displaying the captured image includes a graphic (tag) clearly indicated by the display means as a part of the captured image. The screen tagging device having a function of outputting the created information stored in association with the tag when instructed by the means for instructing.