JP2002165212A - Remote image operating system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a remote image operating system provided with camera operation helping functions by which an operator can easily recognize the actual position and size of an object shot by a camera, even delay time occur in image compression and expansion processings. SOLUTION: The position on a monitor of subject, in which the camera actually captures it and a range of shoot are anticipated by calculation by the delay time due to the compressing and expansion process of image data and the motion of an operating board. The anticipated position and scope is displayed on the monitor, so that an operator of the operating board can recognize it, and the actual position and size of the object shot by the camera can be recognized by superimposing the displayed image on the moving image actually shot by the camera.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement of a device combining a console of a video camera and a moving image data compression / expansion device.

[0002]

2. Description of the Related Art In recent years, image processing techniques, especially compression techniques for image information and audio information, have been remarkably developed, and high-quality moving images and audio can be transmitted even on a transmission line having a small capacity. became. For example, in applications where a camera and monitor are connected to a network to monitor the state of a subject at a remote location, high-quality moving images can be transmitted in combination with an image compression / decompression device even in a network with a small data transmission capacity. You can do it. FIG. 2 shows an example of a usage form in which this image compression / decompression device is connected to a network.
And will be described. The network 1 includes a place where a camera 31 for photographing a subject in a remote place is installed,
9 and camera 3 for monitoring the subject photographed by
This is a data transmission path connecting two or a plurality of remote locations to a place where an operation console 15 for operating the device 1 is installed. The network 1 has a purpose of transmitting data, and outputs transmission data to an external device connected to an arbitrary portion of the network 1 and inputs transmission data from the external device. The image compressor 32 and the camera controller 33 connected to the camera 31 are located at a remote place where the subject is photographed. The image compressor 32 and the camera controller 33 are connected to the network 1. The camera 31 captures an image of a remote place, and functions of the camera 31 such as zoom, pan, and tilt are controlled by camera control information 40 supplied from an external camera controller 33. Image data 39 captured by the camera 31 is output to the image compressor 32. The image compressor 32 outputs image data 39 input from the camera 31.
, Image compression processing is performed. The image data 39 is converted into compressed data 41 having a data amount that can be transmitted through the network 1 by the image compression processing. Compressed data 4
1 is output to the network 1.

A camera controller 33 analyzes the contents of camera control data 42 input from the network 1 and converts the contents into camera control information 40 for controlling the camera 31.
Output to the camera 31. The monitoring place for monitoring the state of the subject includes an image expander 4 connected to a monitor 9 and a camera control data generator 5 connected to a console 15. The image expander 4 and the camera control data generator 5 Are connected to the network 1. The image expander 4 is connected to the network 1
An image decompression process is performed on the input compressed data 2. The compressed data 2 is converted into image data 6 that can display a moving image on the monitor 9 by the image expansion process. The image data 6 is output to the monitor 9. The monitor 9 displays the image data 6 output from the image expander 4. The console 15 is for instructing the camera 31 to perform operations such as zooming, panning, and tilting. By moving a push button, an operation stick (joystick), or the like, the operation is output to the camera control data generator 5 as operation information 14 represented by an electric signal. The camera control data generator 5 receives the camera operation information 14 input from the console 15.
Is analyzed, converted into data suitable for transmission, and output to the network 1.

[0004] Next, the operation of a usage mode in which the image compression / decompression device is connected to a network will be described. The image captured by the camera 31 is output to the image compressor 32 as image data 39 of a moving image, and is subjected to image compression processing by the image compressor 32 to be converted into compressed data 41.
Output to Generally, image compression processing requires a large amount of processing, and it takes time from input of image data to output of compressed data. In particular, if a compression method that emphasizes the compression efficiency is adopted, the compression efficiency is increased by using the redundancy of a plurality of pieces of frame information of a moving image. appear. The delay time of the several frames is long enough to recognize the delay from the viewpoint of human time. The compressed data 41 is transmitted to a remote place via the network 1 and output to the image decompressor 4 as compressed data 2. The image expander 4 performs an image expansion process on the compressed data 2, converts the compressed data 2 into image data 6, and outputs the image data 6 to the monitor 9. In the image decompression process, although not as large as the image compression process, a delay time occurs from the input of the compressed data 2 to the output of the image data 6 due to the decompression process. As described above, the image photographed by the camera 31 is output to the monitor 9 at a remote place through compression and decompression processing. As described above, in the compression processing and the decompression processing, a delay time occurs from the input of data to the output of the data. Therefore, an image captured by the camera 31 is displayed on the monitor 9 with a delay time that can be recognized by a human. This delay time remains fixed at a fixed delay time once the image compressor 32 and the image decompressor 4 are operated.
2 and the specifications of the image intermediater 4 can be known in advance.

On the other hand, at the place where the monitor 9 is installed, there is a console 15 for controlling the camera 31. The console 15 has push buttons, operation sticks, and the like, and instructs execution of functions of the camera 31 such as zoom, pan, and tilt by operating these buttons. The information specified by the push button, the operation stick, or the like is converted into electrical information and output to the camera control data generator 5 as camera operation information 14. The camera control data generator 5 analyzes a content of the input camera operation information 14 and outputs a plurality of instruction information such as push buttons and operation sticks to the network 1. It is converted into control data 3 and output. This camera control data 3
At a remote place where the camera 31 is installed, it is input to the camera controller 33 as camera control data 42 via the network 1. The camera controller 33 analyzes the content of the input camera control data 42 and provides camera control information 4 for instructing individual functions of the camera 31 such as zooming, panning, and tilting.
It is converted to 0 and output to the camera 31. In the camera 31, operations such as zoom, pan, and tilt are performed according to the camera control information 40.

[0006]

Here, when the camera 31 is operated from a remote location, a moving image captured by the camera 31 is monitored on a monitor 9 at a remote location, and a subject to be monitored is displayed on a monitor screen. Is instructed from the console 15 so that the camera 31 can be displayed in a desired size and angle of view. However, since image compression / decompression processing is performed during transmission of image data, a delay time occurs before an image captured by the camera 31 is output to the monitor 9. Here, for example, it is assumed that the delay time is 0.5 seconds, and the subject that has been stationary at the center of the screen starts moving to the right. In this case, the subject projected on the monitor 9 starts moving to the right after 0.5 second, and by the time the subject started moving to the right is projected on the monitor 9, the actual subject moves to the right end of the screen. I have. Therefore, when the operator instructs the operation console 15 to pan the camera 31 rightward while looking at the monitor screen and capturing the subject that has started moving rightward with the camera 31, the actual subject is constantly ahead of the image on the monitor 9. Since the subject has moved to the right, the subject has already moved out of the monitor screen, and cannot be captured by the camera 31. It is also assumed that an object moving to the right suddenly stops. Also in this case, the subject displayed on the monitor 9 is
It moves to the right by a time corresponding to the delay time of the compression / expansion processing. Therefore, if the console 15 is operated to follow the moving right object projected on the monitor 9, the camera 31 is operated to pan right even though the actual object is still, The direction of the camera 31 goes too far to the right. This is the camera 31 such as zoom and tilt.
A similar problem occurs in all of the operations. As described above, due to the delay time caused by the compression / expansion processing, when operating the console 15 while viewing the image displayed on the monitor 9, it is very difficult to move the subject to the camera position intended by the operator. Work. The present invention eliminates these drawbacks and allows the operator to easily recognize the current position and size of the subject with respect to the subject even when a delay time occurs in the image compression / decompression processing. It is an object of the present invention to realize a remote image operation system equipped with a camera operation assisting function that can be used.

[0007]

SUMMARY OF THE INVENTION In order to achieve the above object, the present invention provides a remote image operation system for controlling a remote video camera with an operation console while viewing compressed image data from the video camera on a monitor screen. In the monitor screen, an indication frame indicating a position and a range on a monitor at which the video camera is expected to be currently photographing is displayed by superimposing the compressed image data on a playback image obtained by expanding the compressed image data. Things. Further, based on the delay time caused by the image compression / expansion processing and the operation amount of the console, the indication frame is predicted based on the moving amount of the video camera at that time and the range of the image being captured. Is generated. Further, taking into account the transmission delay time of the image data due to the image compression / expansion processing, the object on the monitor apparently moves in the opposite direction due to the movement of the video camera, but is automatically synchronized in synchronization with the movement. By moving the indication frame, the positional relationship between the subject captured by the video camera and the indication frame is maintained. Also, taking into account the transmission delay time of the image data due to the compression / expansion processing, the video camera zooms, so that the subject on the monitor appears apparently,
While the size changes, the size of the indication frame is automatically increased or decreased so as to synchronize with the increase or decrease in the size, so that the size of the area between the subject captured by the video camera and the indication frame Is to be maintained. That is,
Based on the delay time due to the image data compression / expansion processing and the operation amount of the console, the position on the monitor where the camera will actually capture the subject and the shooting range are predicted from calculation, and the predicted position and The range is displayed on the monitor so that the operator of the console can recognize it, and the displayed image is superimposed on the moving image actually shot by the camera, so that the camera can position and size the subject with respect to the current position. Now you can see if you are shooting.

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows a configuration of a remote image operation system equipped with a camera operation assisting function according to the present invention, which will be described in detail below. The network 1 and the image decompressor 4 are the same as those of the related art (FIG. 2). The compressed data 2 output from the network 1 is input to the image decompressor 4, and the image decompressor 4 performs image decompression processing. 6 is output. The image data 6 is input to the image synthesizer 7. The image synthesizer 7 superimposes the two images of the input image data 6 and the pointing frame data 10, synthesizes the image synthesis data 8, and outputs it to the monitor 9. The monitor 9 is the same as the conventional one, and displays the image synthesis data 8 output from the image synthesizer 7. The operation console 15 is the same as the conventional one, and includes a camera 31 with a push button or an operation stick (joystick).
Are instructed to perform operations such as zoom, tilt, and pan. The instructed information is converted into electrical information, and as camera operation information 14, the camera control data generator 5,
Output to the operation information analyzer 13. The camera control data generator 5 is the same as the conventional one, and can analyze the contents of the input camera operation information 14 and output a plurality of instruction information such as push buttons and operation sticks to the network 1.
The camera control data 3 is converted into a set of data format and output. The operation information analyzer 13 analyzes the contents of the input camera operation information 14. For example, when a zoom instruction is given, whether the instruction is large or small on the subject, and how fast the movement of the camera 31 by the zoom instruction is performed are analyzed. Also, when a van or tilt command is given, the direction of the command to the subject, up, down, left, or right, and the speed of the movement of the camera 31 by the pan or tilt command. It is analyzed whether it is performed in. For any analysis, how long
When an instruction for movement is given from the console 15, it is necessary to know in advance how much the camera 31 actually moves, and the magnitude of the movement can be known from the specifications of the camera 31.

In the analysis of the movement of the camera 31, information on the magnitude of the movement of the camera 31 is given in advance. The operation information analyzer 13 outputs such information of the direction and speed of the movement of the camera 31 to the instruction generator 11 as the movement information 12. In addition, after a delay time from when the console 15 gives a command to the camera 31 to move to the time when the image of the camera 31 is displayed on the monitor 9 elapses, the movement of the camera 31 output as the movement information 12 The information on the movement in the opposite direction is generated and added to the movement information 12 of the camera 31 input from the console 15 at this time. This delay time is also information given in advance. Indication generator 11
Indicates a region on the screen of the monitor 9 where the camera 31 is likely to be currently photographing, from the information indicating the direction and magnitude of the movement of the camera 31 using the input movement information 12. Create such image data. This image data is output to the image synthesizer 7 as the instruction frame data 10.

Next, the operation of the entire operation system will be described with reference to an image decompressor equipped with a camera operation assisting function shown in FIG. 1 as an example. Compressed data 2 input from network 1
Are subjected to decompression processing by the image decompressor 4 and output as image data 6. The operation up to this point is the same as the conventional operation. The image data 6 is input to the image synthesizer 7. The image synthesizer 7 receives the instruction frame data 10 from the other input.
Is entered. The image synthesizer 7 superimposes the two input images and outputs the superimposed image data to the monitor 9 as image synthesized data 8 which is one image data. The instruction frame data 10 will be described later. To change the position or the like of the subject projected on the monitor 9, the operator operates the console 15 while watching the moving image projected on the monitor 9, and instructs the functions of the camera 31 such as zoom, pan, and tilt to change the subject. I do. Information on the camera operation specified by the console 15 is input to the camera control data generator 5 as camera operation information 14.
The camera control data generator 5 analyzes the contents of the camera operation information 14 and outputs a plurality of pieces of instruction information such as push buttons and operation sticks to the network 1. And output. This operation is the same as the conventional one. The camera operation information 14 is also supplied to the operation information analyzer 13. The operation information analyzer 13 analyzes information such as the moving direction and the size of the camera 31 and determines the difference between the position of the image displayed on the monitor 9 and the position of the image currently captured by the camera 31. , The difference in the shooting range of the subject due to zooming or the like. Then, the camera control data generator 5 outputs the information on the calculated position and area difference to the pointer generator 11 as movement information 12.

In the indication generator 11, the person operating the console 15 while watching the monitor 9 can see the difference between the image displayed on the monitor 9 and the image actually captured by the camera 31 at that time. In order to do so, an image in which the area photographed by the camera 31 is indicated by a frame line is generated based on the image position reflected on the monitor 9. This frame line is not displayed when the frame is stationary at the center of the screen. In addition, the indication is displayed such that solid lines, dotted lines, or areas that do not obscure the information of the image displayed on the monitor 9 are painted in a rectangular shape. This image data is output to the image synthesizer 7 as the instruction frame data 10, and the image synthesizer 7 synthesizes the image data 6 and the instruction frame data 10 into image synthesis data 8 which is superimposed. Then, by projecting the image synthesis data 8 on the monitor 9, a person operating the camera 31 using the console 15 can use the camera 31 to change the camera 31
It is possible to know which position is being photographed and in what size by the indication frame. Thereby, even if there is a delay time due to the compression / decompression processing, the operator of the camera 31 can grasp the positional relationship between the subject and the screen actually captured by the camera 31.

Next, the operation information analyzer 13 described above is used.
Will be described in detail by taking the block configuration of FIG. 3 as an example. The operation information analyzer 13 includes a motion direction analyzer 45, a delay unit 46, a direction inverter 47, a zoom amount analyzer 48, a delay unit 49, a zoom amount inverter 50, adders 51 and 52, and a movement information generator 53. Become. The camera operation information 14 input from the console 15 is input to the motion direction analyzer 45 and the zoom amount analyzer 48. The movement direction analyzer 45 analyzes from the content of the camera operation information 14 whether the camera 31 has been instructed to move in the up, down, left, or right direction. The result of this analysis is
The moving direction and the distance of the movement are digitized and output to the adder 51 and the delay unit 46. Delay unit 46
Delays the input data by the same time as the delay time generated in the image compression / decompression processing, and outputs the delayed data to the direction inverter 47. The direction inverter 47 converts the input data indicating the moving direction of the camera 31 into data indicating the opposite moving direction, and outputs the data to the adder 51. Adder 51
Adds the data of the movement direction of the camera 31 output from the movement direction analyzer 45 and the data of the direction opposite to the movement direction of the camera 31 delayed by a certain time outputted from the direction inverter 47, and finalizes this. The data is output to the movement information generator 53 as data on the movement direction of the typical camera 31.

The zoom amount analyzer 48 stores the camera operation information 1
From the contents of 4, the camera 31 is analyzed as to how large the camera 31 is instructed to photograph the subject by the zoom operation. In the case of zooming in, the extent to which the imaging range has been narrowed is quantified, and this data is output to the adder 52 and the delay unit 49. The delay unit 49 delays the input data by the same time as the delay time generated in the image compression / expansion processing, and outputs the data to the zoom amount inverter 50. Zoom amount inverter 5
A value of 0 converts the data indicating the degree to which the area to be imaged by the camera 31 is zoomed in to be narrowed into data indicating the degree to which the area to be imaged in the opposite zoom-out direction is expanded, and outputs the data to the adder 52. The adder 52 adds the data indicating the degree of reduction of the shooting area output from the zoom amount analyzer 48 and the data output from the zoom amount inverter 50 and indicating the degree of expansion of the shooting area. The data is output to the movement information generator 53 as data indicating the size of the shooting area of the camera 31. In the case of zoom-out, the opposite is true. Data indicating the degree of expansion of the imaging region is output from the zoom analyzer 48, and data indicating the degree of reduction of the imaging region is output from the zoom amount inverter 50. The movement information generator 53 is data that the indication frame generator 11 can recognize from the output data of the adders 51 and 52 as to which direction the camera 31 is moving and how large an area is being photographed. It is output as certain movement information 12.

Next, according to the above configuration, the monitor 9 is actually
With reference to FIG. 4, the relationship between the subject and the indication frame shown in FIG. The square represented by the solid line is the range of the screen displayed on the monitor 9. For example, a triangle as a subject is projected on the monitor 9. Here, the camera 31
Is assumed to be operated so that this triangle is always centered on the screen. A square indicated by a dotted line is an instruction frame. An image 54, an image 56, an image 58, and an image 60 are diagrams showing the positional relationship between the monitor screen, the subject, and the pointing frame. Image 55, Image 57, Image 59, Image 6
0 represents the state of the screen when the respective images are actually projected on the monitor 9. Image 54 is
The camera 31 is stationary. The subject triangle is shown in the center of the screen. At this time, the instruction frame is not displayed on the screen. Therefore, the instruction frame is not displayed in the image 55 which is the monitor output. Image 56 shows camera 3 because the triangle of the subject has moved to the left.
It is a state where 1 was vanned to the left. Since the camera 31 was instructed to make a leftward van in accordance with the triangle that has moved to the left, an instructive image is shown on the left. In the image 57 which is a monitor output, a portion protruding to the left of the screen is cut off and output. An image 58 shows a state where the camera 31 is tilted upward because the triangle of the subject has moved upward.
Since the camera 31 has been instructed to tilt upward in accordance with the triangle that has moved upward, the instructional style is displayed above.
In the image 59, which is a monitor output, a portion protruding above the screen is cut and output. The image 60 is a state in which the camera 31 is zoomed in because the subject has moved far and apparently has become smaller. Since the zoom-in was instructed to the camera 31 in accordance with the reduced triangle of the subject, the pointing frame is shown small. Also in the image 61 which is a monitor output, a small triangle and a pointer are output. As described above, since the indication frame is displayed for the subject, the operator operating the console 15 while looking at the monitor 9 allows the camera 31 to adjust the indication frame to the subject to be photographed. Can be operated.

Next, the subject, which was stationary at the center of the screen,
FIG. 5 shows a temporal change of an image, taking as an example a case in which the camera 31 is swung to the left because the camera 31 is moved to the left, and the subject is captured in the center of the screen. In the image 62, the triangle of the subject is stationary at the center. In this case, no instruction frame is displayed. In the image 63, the subject has already started to move.
In the image 64, the subject moves further to the left and stands still. Here, the operator monitoring the monitor 9 notices the movement of the subject, and operates the console 15 to instruct the camera 31 to pan left. In the image 65, the instruction frame is displayed at the center of the screen, and starts moving to the left. However, the image itself has not moved. At this point, the camera 31 has started moving leftward in response to the panning left instruction. In the image 66, the camera 31 is instructed to pan further left, and the designated frame is adjusted to the subject. The image itself has not moved.
The camera 31 has finished moving to the left and has captured the subject in the center of the screen. An image 67 is a state in which the subject on the monitor screen apparently starts moving to the right because the image shot by the camera 31 is delayed by the compression / expansion processing and the camera 31 starts moving to the left. That is, by the operation of the delay unit 46 and the direction reversing unit 47 built in the operation information analyzer 13, when the image 65 is displayed on the monitor screen, the information instructing the camera 31 to pan left is changed in the opposite direction. The movement of the instruction frame is performed as the movement information, and the instruction frame also starts moving to the right as the subject apparently moves to the right. An image 68 is an image when the camera 31 captures the subject at the center of the screen at the time of the image 66. That is, the movement of the subject to the right stops, and the subject is displayed in the center of the screen.
At the same time, when the movement of the instruction frame reaches the center of the screen, the display of the instruction frame disappears. In this manner, the moved pointer appears to move at a speed corresponding to the speed at which the subject in the screen moves in the direction opposite to the movement of the camera 31, and disappears when it reaches the center of the screen. As described above, in a system that performs compression / expansion processing on image data captured by a camera and displays a moving image on a monitor, operating the camera from the console while monitoring the monitor requires a delay time associated with the compression / expansion processing. However, by displaying an instruction frame that immediately responds to an instruction from the console on the monitor, the position of the subject and the camera can be easily grasped.

[0016]

According to the structure of the present invention described above,
Remote images equipped with a camera operation assist function that allows the operator to easily recognize the current position and size of the camera with respect to the subject even if a delay time occurs in the image compression / expansion processing. An operation system can be realized.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of a remote image operation system according to the present invention.

FIG. 2 is a block diagram showing a configuration of an example of a conventional remote image operation system.

FIG. 3 is a block diagram showing the configuration of an embodiment for realizing the operation information analyzer of the present invention.

FIG. 4 is a diagram showing a relationship between a subject and an indication frame displayed on a monitor according to the present invention.

FIG. 5 is a diagram showing a state of a monitor screen on which an instruction frame according to the present invention is displayed by a temporal change.

[Explanation of symbols]

1: Network, 2: Compressed data, 3: Camera control data, 4: Image decompressor, 5: Camera control data generator,
6: image data, 7: image synthesizer, 8 image synthesized data,
9: monitor, 10: instruction frame data, 11: instruction generator, 12: movement information, 13: operation information analyzer, 14: camera operation information, 15: console, 31: camera, 32: image compressor, 33: Camera controller, 39: Image data, 4
0: Camera control information, 41: Complementary data, 42: Camera control data, 45: Motion direction analyzer, 46, 49: Delayer, 47: Direction inverter, 48: Zoom amount analyzer, 50:
Zoom amount inverters, 51 and 52: adders, 53: movement information generator.

Claims (4)

[Claims] 1. A remote image operation system for controlling a video camera at a distance by a console while watching compressed image data from the video camera on a monitor screen, wherein a reproduction is performed by expanding the compressed image data on the monitor screen. A remote image operation system, which displays an indication frame indicating a position and a range on a monitor on which the video camera is currently assumed to be taking a picture, superimposed on the image. 2. The remote image operation system according to claim 1, wherein the instruction frame is photographed based on a delay time due to image compression / expansion processing and an operation amount of the console, and a moving amount of the video camera at that time. A remote image operation system for estimating an image range that is present and generating an image of a corresponding instruction frame. 3. The remote image operation system according to claim 2, wherein the video camera moves, and a subject on the monitor appears to be opposite when the video camera moves in consideration of a transmission delay time of image data due to image compression / expansion processing. A remote image characterized by maintaining the positional relationship between the subject captured by the video camera and the indication frame by automatically moving the indication frame in synchronization with the movement. Operation system. 4. The remote image manipulation system according to claim 2, wherein the video camera zooms and the size of the object on the monitor is apparently reduced in consideration of the transmission delay time of the image data due to the compression / expansion processing. While it changes
By automatically increasing or decreasing the size of the pointing frame so as to synchronize with the increase or decrease in the size, the size of the area between the subject captured by the video camera and the pointing frame is maintained. Remote image operation system.