JP3792901B2 - Camera control system and control method thereof - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a camera control system and a control method therefor, and more particularly, to a video data communication processing system that remotely uses moving image data input from a camera and processed by a digital processing device, such as a TV conference system or a video surveillance system. And suitable.
[0002]
[Prior art]
Conventionally, in fields such as TV conferences and surveillance cameras, a system is configured by a camera and a user terminal capable of remotely controlling parameters thereof via a network, and the user controls the remote camera remotely to shoot. It was configured to receive and display the recorded video. As a camera, a camera with a pan head has been used in order to change the shooting posture.
[0003]
This camera with a pan head uses a dedicated controller to control the left and right pans and the top and bottom tilts with buttons and joysticks. In recent years, a dedicated controller is displayed on a computer display in a pseudo manner using a GUI or the like, and is controlled by a mouse or the like.
[0004]
Many surveillance cameras are configured to control a pan head with a predetermined movement and monitor a predetermined number of images even if the pan head is attached.
[0005]
[Problems to be solved by the invention]
However, in these conventional systems, the only video that was transmitted was the video that was actually captured by the camera with the pan head at a certain point in time, so the surrounding conditions could not be seen and the pan head was moved. I couldn't intuitively understand the video after that. Therefore, in order to photograph a desired position, it is necessary to correct the position of the pan head many times by looking at the transmitted video.
[0006]
By the way, when a cameraman takes a camera and shoots at the shooting site, he looks through the viewfinder of the camera with one eye, but the other is observing the surroundings, and then instantly places the camera at the desired angle. Can be directed. This is possible because the cameraman sees not only the image in the viewfinder that is actually photographed, but also other areas that may be photographed at the same time.
[0007]
In this way, when you want to interactively control the attitude of a remote camera with a pan head while watching the video in a TV conference or remote monitoring, the pan head while watching the video of all areas that the camera can shoot by moving. If the user is controlled, the user feels as if he / she is holding the camera like the above-mentioned photographer and is controlling the shooting range while being in that place, and can easily operate it.
[0008]
For this purpose, the conventional method is to first shoot all areas that can be photographed by moving the camera, and convert these images into one image by image processing. Then, the user controls the shooting posture and zoom of the camera while viewing the still image obtained thereby.
[0009]
However, the image of the entire area that can be generated by such image processing is a still image, not a real-time video. For this reason, it is most necessary to point the camera instantly at the angle at which the user wants to shoot, such as pointing the camera at a participant who is requesting a speech in a TV conference, or pointing the camera at a sudden intruder in remote monitoring. There was a problem that the case was not fully supported.
[0010]
The present invention has been made to solve such a problem, and it is possible to easily and instantaneously indicate a camera posture or the like desired by a user during a TV conference or remote monitoring. Objective.
[0011]
[Means for Solving the Problems]
The camera control system according to the present invention includes a wide-angle camera that captures an image with a wide angle of view, a first device that includes a controllable camera that can control a shooting state, and a network connection with the first device. A camera control system including a second device capable of remotely controlling a shooting state of a control camera, wherein the first device continuously transmits first frame images obtained by the wide-angle camera at a first interval. First output means for automatically generating and outputting, and second output means for continuously generating and outputting the second frame image obtained by the controllable camera at a second interval, wherein the second device comprises: Display means for displaying the first and second frame images output from the first and second output means and displaying an area indicating the imaging area of the controllable camera on the first frame image. When, Panning, tilting, and zooming of the controllable camera so as to capture an image corresponding to the area instructed by the instructing means, instructing means instructing a desired area in the first frame image displayed by the displaying means Control means for issuing at least one of the control commands, wherein the first output means is the first at a timing independent of the first interval at which the control of the controllable camera is completed according to the control command. A frame image is generated and output, and the display means updates and displays the first frame image in response to the timing.
In addition, the control method of the camera control system of the present invention includes a wide-angle camera that captures an image with a wide angle of view and a controllable camera that can control the shooting state, and the first frame image obtained by the wide-angle camera is a first frame image. A first device for continuously generating and outputting at one interval and generating and outputting a second frame image obtained by the controllable camera at a second interval; and a network connection with the first device And a control method of a camera control system configured by a second device capable of remotely controlling a shooting state of the controllable camera, wherein the second device outputs the first output from the first device. In the display step of displaying the first and second frame images and displaying the region indicating the imaging region of the controllable camera on the first frame image, An instruction step for indicating a desired area in the first frame image displayed by the display step, and the controllable camera so as to capture an image corresponding to the area indicated by the instruction step in the second apparatus. A control step of issuing at least one control command of panning, tilting, and zooming, and, in the first device, independent of the first interval at which control of the controllable camera is completed in response to the control command An output step of generating and outputting a first frame image at timing, and an update step of updating and displaying the first frame image in response to the timing in the first device are executed.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of a camera control system according to an embodiment of the present invention. In FIG. 1, 101 is a wide-angle camera capable of photographing a wide angle of view, 102 is an image digitizing circuit for digitizing video captured by the wide-angle camera 101, 103 is an image processing circuit for processing digitized video, and 104 is It is an image encoding circuit for encoding video.
[0013]
Reference numeral 105 denotes a communication control circuit including a CPU, a memory, a network adapter, and the like. 106 denotes a zoom camera included in the camera 110 with a pan head. 107 denotes a pan head included in the camera 110 with a pan head. An image digitizing circuit 109 for digitizing the recorded video is a second device placed remotely via a network. Note that the first device includes the above-described 101 to 108 with respect to the second device 109.
[0014]
Although the second device 109 will not be described in detail, the second device 109 is configured by an existing personal computer (PC), network PC, or workstation having a function of connecting to a network and displaying hypertext data. In this embodiment, the second device 109 will be described using a PC that can display a digital image using display software such as a JPEG image.
[0015]
FIG. 2 is a diagram showing a display example of video on the PC 109 according to the present embodiment. As shown here, the video is displayed on a multi-window system. In FIG. 2, 201 is a display, 202 is a detailed video display window for displaying video captured by the camera with pan head 110, and 203 is a panoramic video display window for displaying video captured by the wide-angle camera 101.
[0016]
Reference numeral 204 denotes an instruction frame for indicating where the detailed video corresponds in the entire video (panoramic video), and is displayed superimposed on the panoramic video display window 203. Reference numeral 205 denotes a GUI slider bar for performing zoom control of the zoom camera 106, and reference numeral 206 denotes a GUI button for acquiring control rights of the camera 110 with a pan head.
[0017]
FIG. 3 schematically shows a data flow according to the present embodiment in a time chart. In FIG. 3 (also in FIG. 1), 112 is a video signal photographed by the wide-angle camera 101, 114 is a video signal compression-coded by the image coding circuit 104, and 116 is a video photographed by the camera 110 with a pan head. A signal 118 is a control signal for controlling the zoom camera 106 and the pan head 107.
[0018]
Reference numeral 115 denotes a network signal that flows on the network, which includes a pan / tilt camera image request signal 115 (a), a pan / tilt camera encoded image signal 115 (b), a wide-angle camera image request signal 115 (c), a cloud, and the like. The table / zoom control signal 115 (d) and the wide-angle camera encoded video signal 115 (e) are divided into five. In FIG. 3, the horizontal axis represents time.
[0019]
FIG. 4 schematically shows an image captured by the wide-angle camera 101. In the present embodiment, a case where a camera having an orthographic fisheye lens is used as the wide-angle camera 101 will be described.
[0020]
FIG. 4A schematically shows light entering the fisheye lens in three dimensions, and A to F are pillars that are set up perpendicular to a plane passing through the optical axis. Column A is set at an angle of 0 ° with respect to the optical axis, and column B is set at an angle of 10 ° with respect to the optical axis. It stands at an angle of 30 °, 45 °, 60 ° and 90 °. The distances to the fisheye lenses of the pillars A to F are all the same.
[0021]
FIG. 4B shows an image obtained by photographing the pillars A to F shown in FIG. 4A with a fisheye lens. A ′ to F ′ in FIG. 4B are images of the pillars A to F in FIG.
FIG. 4C shows a panoramic image in which the pillars A to F shown in FIG. 4A are expressed so that the angle with respect to the optical axis of each image is constant. A ″ to F ″ in FIG. 4C are images of the pillars A to F shown in FIG.
[0022]
FIG. 5 is a view showing a panoramic image corresponding to an image captured by the wide-angle camera 101. In FIG. 5, 51 is a half-sky image taken by a fisheye lens, and 52 is a range of ± 90 ° in the horizontal direction and ± 30 ° in the vertical direction in the real world. 54 is a panoramic image actually used for display, and arrows 53 and 55 indicate the direction of image processing. The numerical value enclosed in parentheses () indicates the coordinate position on each video.
[0023]
The operation of this embodiment will be described below with reference to FIGS. First, the flow of video data will be described with reference to FIG.
The video signal 111 collected by the wide-angle camera 101 having a fisheye lens and converted into an electrical signal is converted into a digital video signal 112 by the image digitizing circuit 102 and further subjected to image processing to be described later by the image processing circuit 103. As a result, the panoramic video signal 113 is obtained. The panoramic video signal 113 generated in this way is compressed and encoded by the image encoding circuit 104 by an encoding method such as JPEG.
[0024]
On the other hand, the video signal 116 condensed by the zoom camera 106 of the camera with pan head 110 and converted into an electric signal is converted into a digital video signal 117 by the image digitizing circuit 108 and further compressed by the image encoding circuit 104. It becomes. The image encoding circuit 104 selects and encodes either the panoramic video signal 113 output from the image processing circuit 103 or the digital video signal 117 output from the image digitizing circuit 108.
[0025]
The video signal 114 encoded by the image encoding circuit 104 is sent to the communication control circuit 105, sent to the network, and sent to the PC 109. Then, the PC 109 performs decryption processing and the like, and displays them on the display. Here, the network is an Internet-compliant computer network using, for example, TCP / IP.
[0026]
Next, the flow of video and camera control will be described with reference to FIG.
First, the control program installed in the PC 109 sends a pan head camera image request signal 115 (a) (camera head image request A10). The camera platform image request A 10 from the PC 109 is received by the communication control circuit 105. The communication control circuit 105 instructs the image encoding circuit 104 to select the digital video signal 117 from the image digitizing circuit 108.
[0027]
At this time, one frame of the video captured by the image digitizing circuit 108 is a frame A11 of the camera video signal 116 with a pan head, which is encoded by the image encoding circuit 104 and becomes a frame A12 of the encoded video signal 114. Further, the communication control circuit 105 transmits the frame A13 of the camera platform encoded video signal 115 (b) to the network and displays it as the frame A14 in the detailed video display window 202 of FIG. This series of operations is repeated until the end is instructed by the user (the pan head camera image request signal 115 (a) is output at a constant interval of A20, A30,... Following A10).
[0028]
On the other hand, the control program installed in the PC 109 sends out a wide-angle camera video request signal 115 (c) (panoramic video request B10). The panorama video request B 10 from the PC 109 is received by the communication control circuit 105. The communication control circuit 105 instructs the image encoding circuit 104 to select the panoramic video signal 113 from the image processing circuit 103.
[0029]
At this time, one frame of the video captured by the image processing circuit 103 is a frame B11 of the wide-angle camera video signal 112, which is converted into a panoramic video signal by the image processing circuit 103 and then encoded by the image encoding circuit 104. This is the frame B12 of the encoded video signal 114. Further, the communication control circuit 105 superimposes the parameters of the camera platform 107 and the zoom camera 106 at that time, and transmits them to the network as a frame B13 of the wide-angle camera encoded video signal 115 (e).
[0030]
Then, it is displayed as a frame B14 by the PC 109 in the panoramic video display window 203 of FIG. At this time, the control program of the PC 109 designates a rectangle on the panoramic video of the frame B14 displayed in the panoramic video display window 203 based on the parameters of the pan head 107 and the zoom camera 106 superimposed on the frame B13. The frame 204 is superimposed and displayed.
[0031]
A user observes the display 201 at this point, and can easily make a decision such as where the camera 110 with a pan head is facing and where the camera 110 with a pan head is directed. Thus, the user creates a control command for the position of the pan head 107 and the zoom of the zoom camera 106 using a control program installed in the PC 109.
[0032]
The position of the pan head 107 is instructed by designating a desired position or region on the panoramic video display window 203 with a mouse, for example. The zoom of the zoom camera 106 is instructed using the slider bar 205 shown in FIG. A control command based on this instruction is transmitted from the PC 109 to the network as a control command C10 of the pan / zoom control signal 115 (d). This is received by the communication control circuit 105, and is transmitted to the camera with pan head 110 as a zoom / head operation command C 13 of the zoom / head control signal 118.
[0033]
The communication control circuit 105 monitors and controls the pan head 107 and the zoom camera 106, and selects the panoramic video signal 113 from the image processing circuit 103 to the image encoding circuit 104 when the operation based on the control command C10 is completed. Give instructions to do. As a result, the image encoding circuit 104 encodes the panoramic video B21 generated from the wide-angle camera video signal 112 to generate a frame B22 of the encoded video signal 114.
[0034]
Further, the communication control circuit 105 superimposes the parameters of the camera platform 107 and the zoom camera 106 on the frame B22 to form the frame B23 of the wide-angle camera encoded video signal 115 (e), which is transmitted to the PC 109. Is done. This video is displayed as a frame B24 in the panoramic video display window 203 of the PC 109. Also at this time, the control program of the PC 109 superimposes the rectangular instruction frame 204 on the video of the frame B24 of the panoramic video display window 203 based on the parameters of the pan head 107 and the zoom camera 106 superimposed on the frame B23. And display.
[0035]
In the display frame of the display 201 shown in FIG. 3, the frame with the symbol A is a detailed image captured by the camera 110 with the pan head, and the frame with the symbol B is captured by the wide-angle camera 101. Panoramic video. As can be seen, the detailed video and the panoramic video are transmitted to the PC 109 in a timely manner and displayed on the display 201. In addition to the detailed video, the entire range that can be photographed by the camera 110 with the pan head is simultaneously provided as a panoramic video. It has become so.
[0036]
Therefore, by outputting the wide-angle camera image request signal 115 (c) at an appropriate interval, not only the detailed image but also the surrounding image can be viewed in almost real time, and the change in the surrounding situation can be accurately observed. I can grasp it. Thus, by remotely controlling the parameters of the camera with pan head 110 while looking at the display 201, for example, the camera is directed to a participant who is requesting speech in a TV conference, or the camera is directed to a sudden intruder in remote monitoring. And so on, you can instantly point the camera to the angle you want to shoot.
[0037]
Since the detailed video is more important than the panoramic video for the user of the PC 109, in this embodiment, the rate of transmitting the frame of the detailed video is set higher than the rate of transmitting the frame of the panoramic video. The changes are made smoother.
[0038]
Next, a panoramic video generation method according to the present embodiment will be described in detail with reference to FIGS. 4 and 5.
The panoramic image used in the present embodiment is an image including the maximum range that can be captured by changing the pan / tilt / zoom values with the camera with pan head 110 used for acquiring the detailed image.
[0039]
The camera with pan head 110 has, for example, a pan of ± 50 ° and a tilt of ± 20 °, a zoom magnification of 8 times, a viewing angle of 6.14 ° on the telephoto side, and a viewing angle on the widest side. Is 45.84 °. The above pan / tilt values are relative to the optical axis. When the range to be imaged is shown, pan is ± 80.5 ° and tilt is ± 42.9 °.
[0040]
In the present embodiment, an image with a constant viewing angle is used as the panoramic image. For example, when the viewing angle of 1 ° corresponds to 1 pixel, in the case of the above-described camera, the panoramic image is an image of 161 × 86 pixels.
In the following description, for the sake of simplicity, a camera with a pan field of view of ± 90 ° and a tilt of ± 30 ° is assumed.
[0041]
The digital video signal 112 photographed and digitized by the wide-angle camera 101 has a wide range of images captured by the fisheye lens, but the orthographic projection type fisheye lens used in this embodiment is the panoramic image described above. In contrast, the angle in the normal direction at the periphery is small, and the angle in the normal direction at the center is greatly projected. This is illustrated in FIG. 4 (b). In other words, the angle between A ′ and C ′ and the angle between E ′ and F ′ are both 30 ° originally, but E ′ and F ′ in the peripheral part rather than the distance between A ′ and C ′ in the central part. The interval of is shorter.
[0042]
In the following, the relationship between the real world, a fisheye camera image, and a panoramic image will be described.
The images of the pillars A to F shown in FIG. 4A are projected onto the image sensor by the fisheye lens. FIG. 4B schematically shows how the images of the pillars A to F are projected onto the image sensor.
[0043]
That is, in the orthographic projection type fisheye lens, the displacement y from the optical axis center of the image, the focal length ξ, and the incident angle φ are between
y = ξ · sin (φ)
There is a relationship. Therefore, the six pillars A to F in FIG. 4A are imaged on the image sensor as A ′ to F ′ in FIG.
[0044]
The angle between the column A and the column C (= 30 °) and the angle between the column E and the column F (= 30 °) are as follows on the image sensor.
{Sin (30 °) −sin (0 °)}: {sin (90 °) −sin (60 °)} = 1: 0.27
The difference comes out. A panoramic image obtained by correcting this by image processing is shown in FIG. That is, 30 ° between the columns A and C and 30 ° between the columns E and F are mapped 1: 1 on the corrected image (between A ″ and C ″ and E ″, F). Process the images so that the distance between them is the same.
[0045]
In the present embodiment, the wide-angle camera 101 provides a panoramic image that is displayed by simultaneously imaging the entire range that can be imaged by the camera 110 with a pan head. As a display method, the angle of the image with respect to the lens is on the image plane. Use a projection method that projects to the same size. Therefore, the X coordinate represents the pan angle, and the Y coordinate represents the tilt angle.
[0046]
In order to convert a video shot with a fisheye lens into a panoramic video, the position on the panoramic video is (θ, φ), the corresponding position on the video shot with the fisheye lens is (η, ζ),
η = ξ · cos (φ) sin (θ) (1)
ζ = a · ξ · φ / π (2)
Correct the relationship. Here, a is a constant, ξ is the radius of the circle of the image portion on the image taken by the fisheye lens, and ξ = 100 is used in FIG.
[0047]
What is used as a panoramic image in this embodiment is
−90 ≦ θ ≦ + 90, −30 ≦ φ ≦ + 30
Is within the range. The image processing circuit 103 obtains corresponding pixels of the image captured by the fisheye lens for all panoramic image pixels within this range. A rectangle indicated by 54 in FIG. 5B is the range of the panoramic image, and the value of each pixel is obtained by raster scanning according to the arrow 55. A video 51 in FIG. 5A is a half-sky video generated by an orthographic projection fisheye lens having an angle of view of 180 °, and a video in a range 52 in the video corresponds to the panoramic video 54 in FIG. .
[0048]
The image processing circuit 103 is provided with an image storage means (not shown), and each of the 181 × 61 pixels of the panoramic video stored therein is handled by using the above equations (1) and (2). The pixel position of the fisheye lens image to be calculated is calculated. At this time, since the pixel position is generally not an integer, the pixel value is determined by using a nearest pixel method for rounding down or a weighted average method using four neighboring points.
[0049]
Next, a method for superimposing the instruction frame 204 will be described. The position of the rectangular instruction frame 204 superimposed on the panoramic video display window 203 is calculated from the pan / tilt / zoom values of the camera 110 with a pan head. That is, the pan value is θ (horizontal angle, unit is degree), the tilt value is φ (vertical angle, unit is degree), the zoom value is Z (angle formed by image diagonal, unit is degree), and the zoom value Z is Assume that it is not very large. Since the NTSC image has a 4: 3 aspect ratio, the field angle in the horizontal direction is (4Z / 5) and the field angle in the vertical direction is (3Z / 5).
[0050]
Therefore, the instruction frame 204 is
Point (θ− (4Z / 5), φ− (3Z / 5)),
Point (θ− (4Z / 5), φ + (3Z / 5)),
Point (θ + (4Z / 5), φ + (3Z / 5)),
Point (θ + (4Z / 5), φ− (3Z / 5))
It becomes a rectangle connecting. In the above description, the rectangular instruction frame 204 is superimposed on the PC 109, but may be superimposed on the panoramic video prior to image encoding.
[0051]
As described above in detail, in the present embodiment, the detailed image captured by the camera with pan head 110 and the panoramic image captured by the wide-angle camera 101 over the entire range that can be captured by the camera with pan head 110 are transmitted to the PC 109. Since each of them is displayed on a different window, the user of the PC 109 can view not only the detailed video that he / she desires but also the video around it in almost real time. Thus, by remotely controlling the parameters of the camera with pan head 110 while looking at these windows, the display of the detailed video can be instantaneously changed according to the surrounding situation change or the like.
[0052]
In addition, since the instruction frame 204 corresponding to the field of view of the detailed image captured by the camera 110 with the pan head is superimposed on the panoramic image captured by the wide-angle camera 101, the camera 110 with the pan head is displayed. Therefore, it is possible to easily confirm the current shooting posture and the surrounding image at that time, and to prevent operational errors such as posture control.
[0053]
(Second Embodiment)
FIG. 6 is a block diagram showing a second embodiment according to the present invention. In the first embodiment shown in FIG. 1, only one wide-angle camera 101 is used. However, in the second embodiment, a plurality of cameras are used, and images captured by these are combined and sent out by image processing. . In FIG. 6, the same elements as those shown in FIG.
[0054]
In the second embodiment, the two wide-angle cameras 101 and 601 are each fixed at an angle of 180 ° as shown in FIG. In FIG. 7, 71 and 72 are wide-angle cameras, and 73 is a zoom camera with a pan head. The pan-tilt zoom camera 73 may be configured to be separable from the main body.
[0055]
In FIG. 6, a video signal 611 photographed by the wide-angle camera 601 is converted into a digital video signal 612 by an image digitizing circuit 602 and given to an image processing circuit 603. The image processing circuit 603 processes the digital video signal 112 from the image digitizing circuit 102 and the digital video signal 612 from the image digitizing circuit 602 into panoramic images without distortion, and further synthesizes and outputs them. To do. On the contrary, the image may be processed into a panoramic image without distortion after being synthesized first.
[0056]
As described above, according to the second embodiment, as described in the first embodiment, panoramic images of 180 ° in the pan direction are respectively obtained by the two wide-angle cameras 101 and 601 (71 and 72 in FIG. 7). Obtainable. In the second embodiment, these panoramic images are connected to create a 360 ° image, which is used to control the camera 110 with a pan head. Therefore, it is possible to change the display of the detailed video instantaneously according to a change in the surrounding situation while viewing a wider range.
[0057]
(Third embodiment)
FIG. 8 is a block diagram showing a configuration of a camera control system according to the third embodiment of the present invention. In FIG. 8, 801 is a wide-angle lens, 802 is an optical sensor and an A / D converter (hereinafter simply referred to as an A / D converter) for an image formed by the wide-angle lens 801, 803 is an image encoder, Reference numeral 804 denotes a communication control device including a CPU, a memory, a network adapter, etc., 805 denotes a zoom lens of a camera with a pan head, 806 denotes an optical sensor of the camera with a pan head, and an A / D converter (hereinafter simply referred to as an A / D converter). 807 is a pan head.
[0058]
FIG. 9 is a diagram showing a display example on the PC 109 according to the present embodiment, and the present embodiment is also configured with a multi-window system. In FIG. 9, reference numeral 901 denotes a display, 902 denotes a video display window of a camera with a pan head picked up by a zoom lens 805 and an A / D converter 806, and 903 picks up an image by a wide-angle lens 801 and an A / D converter 802. A wide-angle video display window 904 is an instruction frame superimposed on the wide-angle video display window 903.
[0059]
FIG. 10 schematically shows a data flow according to the present embodiment in a time chart. In FIG. 10 (also in FIG. 8), 811 is a wide-angle camera video signal, 812 is an encoded video signal, 813 is a camera video signal with a pan head, 814 is a pan head control signal, and 815 is a zoom control signal.
[0060]
Reference numeral 816 denotes a network signal that flows on the network, which includes a pan / tilt camera video request signal 816 (a), a pan / tilt camera encoded video signal 816 (b), a wide-angle camera video request signal 816 (c), a cloud, and the like. The table / zoom control signal 816 (d) and the wide-angle camera encoded video signal 816 (e) are divided into five. In FIG. 10, the horizontal axis represents time.
FIG. 11 is an external view showing an implementation example of the camera according to the present embodiment.
[0061]
The operation of this embodiment will be described below with reference to FIGS. First, the flow of video data will be described with reference to FIG.
The video signal 811 collected by the wide-angle lens 801 and converted into an electrical signal by the A / D converter 802 is compressed and encoded by an image encoder 803 using an encoding method such as JPEG.
[0062]
On the other hand, the video signal 813 collected by the zoom lens 805 and converted into an electrical signal by the A / D converter 806 is similarly compression encoded by the image encoder 803. The image encoder 803 selects either the wide-angle camera image signal 811 output from the A / D converter 802 or the camera image signal 813 with a pan head output from the A / D converter 806 for encoding. Turn into.
[0063]
The video signal 812 encoded by the image encoder 803 is sent to the communication control device 804, sent to the network, and sent to the PC 109. Then, the PC 109 performs decryption processing and the like, and displays them on the display. Here, the network is an Internet-compliant computer network using, for example, TCP / IP.
[0064]
Next, the flow of video and camera control will be described with reference to FIG.
First, the control program installed in the PC 109 sends a pan head camera image request signal 816 (a) (head camera image request A10). The camera platform video request A10 from the PC 109 is received by the communication control device 804. The communication control device 804 instructs the image encoder 803 to select the camera image signal 813 with a pan head from the A / D converter 806.
[0065]
At this time, one frame of the video captured by the image encoder 803 is a frame A11 of the camera video signal 813 with a pan head, and this is encoded by the image encoder 803 to become a frame A12 of the encoded video signal 812. Further, the communication control device 804 transmits a frame A13 of the camera platform encoded video signal 816 (b) to the network, and the PC 109 displays the frame A14 in the camera platform video display window 902 of FIG. This series of operations is repeated until the end is instructed by the user.
[0066]
On the other hand, the control program installed in the PC 109 sends out a wide-angle camera image request signal 816 (c) (wide-angle camera image request B10). The wide-angle camera image request B10 from the PC 109 is received by the communication control device 804. The communication control device 804 instructs the image encoder 803 to select the wide-angle camera video signal 811 from the A / D converter 802.
[0067]
At this time, one frame of the video captured by the image encoder 803 is a frame B11 of the wide-angle camera video signal 811, which is encoded by the image encoder 803 to become a frame B12 of the encoded video signal 812. Further, the communication control device 804 superimposes the parameters of the camera platform 807 and the zoom lens 805 at that time, and transmits the frame B13 of the wide-angle camera encoded video signal 816 (e) to the network.
[0068]
Then, it is displayed as a frame B14 by the PC 109 in the wide-angle video display window 903 of FIG. At this time, the control program of the PC 109 uses a rectangular instruction frame 904 on the video of the frame B14 displayed on the wide-angle video display window 903 based on the parameters of the camera platform 807 and the zoom lens 805 superimposed on the frame B13. Is superimposed and displayed.
[0069]
The user can observe the display 901 at this point, and can easily make a decision such as where the camera with the pan head is facing and where the camera with the pan head is directed. Accordingly, the user creates a control command for the position of the camera platform 807 and the zoom of the zoom lens 805 using a control program installed in the PC 109.
[0070]
This is transmitted from the PC 109 to the network as a control command C10 of the pan / zoom control signal 816 (d). This is received by the communication control device 804 and transmitted to the camera platform 807 as the camera platform operation command C13 of the camera platform control signal 814. Similarly, the zoom operation command C12 of the zoom control signal 815 is transmitted to the zoom lens 805.
[0071]
The communication control device 804 monitors and controls the camera platform 807 and the zoom lens 805, and when the operation based on the control command C10 is completed, the wide-angle camera video signal from the A / D converter 802 is sent to the image encoder 803. An instruction to select 811 is issued. Accordingly, the image encoder 803 encodes the frame B21 of the wide-angle camera video signal 811 to generate the frame B22 of the encoded video signal 812.
[0072]
Further, the frame B22 is superimposed on the parameters of the camera platform 807 and the zoom lens 805 by the communication control device 804 to become a frame B23 of the wide-angle camera encoded video signal 816 (e), which is transmitted to the PC 109. Is done. This video is displayed as a frame B24 in the wide-angle video display window 903 of the PC 109. Also at this time, the control program of the PC 109 superimposes the rectangular instruction frame 904 on the image of the frame B24 of the wide-angle image display window 903 based on the parameters of the camera platform 807 and the zoom lens 805 superimposed on the frame B23. And display.
[0073]
As described above in detail, according to the third embodiment, as in the first embodiment, the display of the detailed video can be instantaneously changed according to the surrounding situation change or the like. In the third embodiment, the wide-angle lens 801 is not a fish-eye lens, and no image processing means for correcting a distorted image is provided, so that the cost of the entire apparatus can be kept low and the processing speed can be increased. .
[0074]
(Fourth embodiment)
FIG. 12 is a block diagram showing a configuration of a camera control system according to the fourth embodiment of the present invention.
In the third embodiment, the process of superimposing the rectangular instruction frame 904 on the video of the wide-angle video display window 903 is performed in the PC 109. However, in the fourth embodiment described below, the processing is performed on the wide-angle camera video signal 811. The instruction frame 904 is superimposed and sent out in the form of an image.
[0075]
In FIG. 12, the same elements as those shown in FIG. 8 are denoted by the same reference numerals, and redundant description is omitted. In the fourth embodiment, the wide-angle camera image request B10 from the PC 109 shown in FIG. 10 is received by the communication control device 804, and the wide-angle camera image from the A / D converter 802 is sent to the image encoder 803. An instruction to select signal 811 is made.
[0076]
At this time, one frame of the video output from the A / D converter 802 is a frame B11 of the wide-angle camera video signal 811 and is temporarily stored in the image storage device 1201. The communication control device 804 draws a rectangular instruction frame 904 on the stored frame B11 through the memory address data signal 1212 (obtained from the parameters of the camera platform 807 and the zoom lens 805 at that time). The result is given to the image encoder 803.
[0077]
The frame B11 of the wide-angle camera video signal 1211 in which the instruction frame 904 is drawn is encoded by the image encoder 803 to become a frame B12 of the encoded video signal 812. Further, the communication control device 804 transmits the frame B13 of the wide-angle camera encoded video signal 816 (e) to the network, and the PC 109 displays the frame B14 in the wide-angle video display window 903 of FIG.
[0078]
Since other operations are the same as those in the third embodiment, a duplicate description is omitted.
In the fourth embodiment, the frame image is temporarily stored using the image storage device 1201, but it is obvious that this may be replaced with a line buffer or the like.
[0079]
(Fifth embodiment)
FIG. 13 is a block diagram showing a configuration of a camera control system according to the fifth embodiment of the present invention. In the fifth embodiment, as in the second embodiment described above, a plurality of wide-angle cameras are used, and wide-angle images captured by these cameras are combined and sent out by image processing.
[0080]
That is, in FIG. 13, video signals photographed by the wide-angle cameras 801 and 1301 are digitized by the A / D converters 802 and 1302, respectively, and are supplied to the image synthesizer 1303. The image synthesizer 1303 synthesizes and outputs the two input digital video signals. Other configurations and operations are the same as those described with reference to FIG. 8, and thus detailed description thereof is omitted here.
[0081]
(Other embodiments of the present invention)
In order to operate various devices in order to realize the functions of the above-described embodiments, a program code of software for realizing the functions of the above-described embodiments is provided to an apparatus connected to the various devices or a computer in the system. What is implemented by operating the various devices according to a program supplied and stored in a computer (CPU or MPU) of the system or apparatus is also included in the scope of the present invention.
[0082]
In this case, the program code of the software itself realizes the functions of the above-described embodiments, and the program code itself and means for supplying the program code to the computer, for example, the program code is stored. The recording medium constitutes the present invention. As a recording medium for storing the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
[0083]
Further, by executing the program code supplied by the computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) or other application software in which the program code is running on the computer, etc. It goes without saying that the program code is also included in the embodiment of the present invention even when the functions of the above-described embodiment are realized jointly.
[0084]
Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, the CPU provided in the function expansion board or function expansion unit based on the instruction of the program code Needless to say, the present invention includes a case where the functions of the above-described embodiment are realized by performing part or all of the actual processing.
[0085]
【The invention's effect】
As described above, according to the present invention, an image captured by a controllable camera and an image captured by a wide-angle camera over the entire range that can be captured by the controllable camera are transmitted from the first device to the second device, and each is displayed. As a result, the user of the second apparatus can view not only the detailed image desired by the user but also the surrounding image in a form almost in real time. Thus, by remotely controlling the shooting state of the controllable camera while viewing these displays, it is possible to instantaneously change the display of the detailed image in accordance with the surrounding situation change or the like. In addition, since the first frame image is generated at the timing according to the control command, the latest wide-angle image can be viewed, the surrounding situation change can be accurately grasped, and accurate camera control can be performed.
Therefore, for example, when it is desired to interactively control the shooting state of a remote controllable camera while watching a video such as a TV conference or remote monitoring, while watching the video of all areas that the controllable camera can move and shoot The photographing state can be controlled and can be easily operated.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a camera control system according to a first embodiment of the present invention.
FIG. 2 is a diagram showing an image display example according to the first embodiment of the present invention.
FIG. 3 is a data flow diagram for explaining the operation of the first exemplary embodiment of the present invention.
FIG. 4 is a diagram showing a relationship between a video image captured by the fish-eye camera used in the first embodiment and a panoramic video image.
FIG. 5 is a diagram for explaining image processing according to the first embodiment of the present invention;
FIG. 6 is a block diagram illustrating a configuration of a camera control system according to a second embodiment of the present invention.
FIG. 7 is an external configuration diagram of a first apparatus for explaining a lens position according to a second embodiment of the present invention.
FIG. 8 is a block diagram showing a configuration of a camera control system according to a third embodiment of the present invention.
FIG. 9 is a diagram illustrating an image display example according to a third embodiment of the present invention.
FIG. 10 is a data flow diagram for explaining the operation of the third exemplary embodiment of the present invention.
FIG. 11 is an external configuration diagram showing an implementation example of the first device according to the third embodiment of the present invention.
FIG. 12 is a block diagram showing a configuration of a camera control system according to a fourth embodiment of the present invention.
FIG. 13 is a block diagram showing a configuration of a camera control system according to a fifth embodiment of the present invention.
[Explanation of symbols]
101 Wide-angle camera
103 Image processing circuit
105 Communication control circuit
106 Zoom camera
107 pan head
109 PC (second device)
110 Camera with pan head
202 Detailed video display window
203 Panorama image display window
204 Instruction frame
601 Wide angle camera
603 Image processing circuit

Claims (9)

A first apparatus having a wide-angle camera that captures an image with a wide angle of view, and a controllable camera capable of controlling the shooting state, and the first apparatus connected to the network, and remotely controlling the shooting state of the controllable camera a camera control system which is constituted by the second device and capable,
The first device includes:
First output means for continuously generating and outputting a first frame image obtained by the wide-angle camera at a first interval;
Second output means for continuously generating and outputting second frame images obtained by the controllable camera at a second interval;
The second device is
Display means for displaying the first and second frame images output from the first and second output means and displaying an area indicating the photographing area of the controllable camera on the first frame image; ,
Instruction means for indicating a desired region in the first frame image displayed by the display means;
Control means for issuing at least one control command of panning, tilting and zooming of the controllable camera so as to capture an image corresponding to the area designated by the instruction means;
The first output means generates and outputs a first frame image at a timing independent of the first interval at which the control of the controllable camera is completed according to the control command,
The camera control system according to claim 1, wherein the display means updates and displays the first frame image in response to the timing . The first frame image captured by the wide-angle camera is a panoramic image, and the aspect ratio of the panoramic image is the same as the aspect ratio of the entire area that can be captured by the controllable camera. The camera control system described. The wide-angle camera includes a super-wide-angle lens that causes distortion in a captured image,
3. The image processing apparatus according to claim 1, further comprising an image processing unit configured to generate a panoramic image as the first frame image by performing image processing on the image captured by the super wide-angle lens and eliminating the distortion. The camera control system described. The camera control system according to any one of claims 1 to 3, wherein the display means displays the area as a frame in a superimposed manner. The first apparatus generates a first frame image by performing image processing on a plurality of wide-angle cameras and a plurality of images captured by the plurality of wide-angle cameras before transmission and combining them into one image . the camera control system according to any one of claims 1-4, characterized in that it comprises a second image processing means. A wide-angle camera that captures an image with a wide angle of view, and a controllable camera that can control the shooting state, and continuously generate and output the first frame image obtained by the wide-angle camera at a first interval. A first device that continuously generates and outputs a second frame image obtained by a controllable camera at a second interval, and is connected to the first device via a network so that the photographing state of the controllable camera can be remotely controlled. a method of controlling a camera control system which is constituted by a such a second device,
In the second apparatus, the first and second frame images output from the first apparatus are displayed, and an area indicating the imaging area of the controllable camera is displayed on the first frame image. Display process;
In the second apparatus, an instruction step for indicating a desired region in the first frame image displayed by the display step;
A control step of issuing at least one control command of panning, tilting and zooming of the controllable camera so as to capture an image corresponding to the region instructed by the instructing step in the second device;
An output step of generating and outputting a first frame image at a timing independent of the first interval at which the control of the controllable camera is completed in response to the control command in the first device;
A control method for a camera control system, characterized in that, in the first device, an update step of updating and displaying the first frame image in response to the timing is executed . Same as the aspect ratio of the entire area that can be imaged by the controllable camera by applying image processing to the image captured by the wide-angle camera equipped with a super-wide-angle lens that causes distortion in the captured image. 7. The camera control system according to claim 6 , wherein a panorama image as the first frame image having a ratio is generated, and the generated panorama image is transmitted from the first device to the second device. Control method . The image captured by the wide-angle camera, according to claim 6 or 7, characterized in that so as to display the superimposed information of the frame which corresponds to the visual field of the image being picked up by the controllable camera Control method of camera control system . 9. The first frame image according to claim 6, wherein a plurality of images picked up by a plurality of wide-angle cameras are subjected to image processing prior to transmission and combined into one image to generate the first frame image. A control method for the camera control system according to claim 1 .

Images