JP2001036898A - Camera system for generating panoramic video - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid generating of unnatural distortion due to the difference of photographing directions in a panoramic picture to be generated by providing a photographing camera and a photographing parameter calculating means for calculating the photographing parameter of a camera based on a photographing picture photographed by means of the photographing camera. SOLUTION: When the input part 163 of a panorama video display part 160 receives input from a user, an overlapping area detecting part 187 obtains an area where videos photographed by a first camera part 100 and a second camera part 130 are overlapped. At the time of dividing a featured block, namely video, within the frame of video 1 photographed by means of the first camera part 100 and the second camera part 130 into blocks consisting of 16 pictures ×16 pixels, a feature point extracting part 166 obtains a block maximizing its distributed value. Then, through the use of this feature point, a photographic parameter arithmetic part 169 compares the size and the perspective drawing of an object in video to calculate the photographic parameter of the part 130 from the comparing result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a camera system for generating a panoramic image and a method of using the same.

[0002]

2. Description of the Related Art In a conventional panoramic image generation system,
The photographing direction of the camera is not considered at all, and an image of an arbitrary photographing direction and a zoom ratio photographed by the camera is corrected to generate a panoramic image. For example, QuickTimeV
In R, etc., a panoramic image was generated by distorting the camera image so that the captured image and the image seemed to be connected seamlessly.

[0003]

However, when a panoramic image is generated using the above-described conventional technology, (1) if the photographing directions of the photographed images are largely different, the images seem to be connected seamlessly. Therefore, the image must be greatly distorted, and only an unnatural panoramic image can be obtained.

(2) If the zoom of the captured image is different, the degree of detail between the image and the image is large, and the quality of the generated panoramic image is partially detailed or coarse. Panoramic images could not be seen continuously.

(3) Alternatively, when the resolution of the photographed image is different, in order to make the image and the image have the same level of detail,
It is necessary to match the level of detail of one of the videos with the video of a lower level of detail, and the quality of the generated panoramic image has deteriorated.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and camera parameters are set such that respective camera images are photographed at the same resolution and images are photographed in the same photographing direction. The purpose is to seek.

[0007]

SUMMARY OF THE INVENTION An object of the present invention is to detect a degree of image distortion based on an image captured by a camera and calculate an optimal camera imaging direction and zoom ratio for a panoramic image based on the distortion information. This is achieved by providing parameter calculation means.

With such a configuration, according to the present invention, unnatural distortion due to a difference in photographing direction does not occur in the generated panoramic image.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment to a fourth embodiment of the present invention will be described with reference to FIGS.

First, a first embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a system configuration diagram of the first embodiment. In the first embodiment, in a system for real-time panoramic synthesis processing of live video from two TV cameras that zoom, tilt, and pan a wide area such as a river and monitor the panoramic video,
By making the image quality of the two cameras substantially the same, such as the size and perspective (degree of distortion) of an object such as a building, a tree, or a person in the camera image, the image is enlarged by the panorama synthesis process. Calculate the shooting parameters (pan angle, tilt angle, zoom ratio) of the TV camera to prevent the difference between the roughness of the two images and the unnatural distortion of the image due to the perspective correction processing, and store the shooting parameters in a table. This is an example of a camera system for generating a panoramic video, in which a user designates the content of a table, controls the TV camera with parameters of the content, and shoots an image free from the above-mentioned problem with the TV camera.

This system includes a first camera unit 100 that allows a user to control the pan, tilt, and zoom rates of a TV camera.
A second camera unit 130 in which the pan angle, the tilt angle, and the zoom ratio of the TV camera are controlled according to the perspective and size of an object captured in the video captured by the first camera unit 100; The video of the second camera unit 130 is compared, and the pan, tilt,
A panorama image display unit 160 that calculates a zoom, performs a panorama synthesis process on a live image captured by the first camera unit 100 and the second camera unit 130, and displays the panorama image; a first camera unit 100 and a second camera unit 130 Panoramic video display section 16
0 and a network 190 for communicating video data and photographing parameters.

The first camera unit 100 includes a TV camera 103 capable of controlling a zoom ratio, a tilt angle, and a pan angle for photographing an image of a site, and instructs the TV camera 103 on a pan angle, a tilt angle, and a zoom ratio. A camera control unit 106 for receiving pan and tilt angles and a zoom ratio measured by the TV camera 103, a video capture unit 109 for receiving an analog video signal from the TV camera 103 and capturing the video, and a panoramic video display unit The camera receives a shooting parameter (pan angle, tilt angle, zoom rate) designated by the user transmitted from the camera control unit 160, transmits the measured shooting parameter received by the camera control unit 106 to the panorama video display unit 160, Video capture unit 1
The camera communication unit 112 transmits the video data captured in step 09 to the panoramic video display unit 160, and the camera central control unit 115 controls the overall flow of the first camera unit 100.

The second camera unit 130 is provided with the first camera unit 10.
0, a TV camera unit 133, a camera control unit 136, a video capture unit 139, and a camera communication unit 14.
2 and a camera central control unit 145. The function of each unit of the second camera unit is the same as the function of each unit of the first camera unit.

The panoramic image display unit 160 obtains an input unit (mouse) 163 for receiving an input from the user, and an area where the respective images captured by the first camera unit 100 and the second camera unit 130 overlap each other. The overlap area detection unit 187 and the characteristic blocks in one frame of each image captured by the first camera unit 100 and the second camera unit 130, that is, the image is divided into blocks of 16 pixels × 16 pixels. A feature point extracting unit 166 for finding a block having the largest variance value,
Using the feature points extracted by the feature point extraction unit 166, the size and perspective of the object in the video are compared, and the second
A photographing parameter calculator 169 for calculating photographing parameters of the camera unit 130, a photographing parameter table 172 holding the photographing parameters calculated by the photographing parameter calculator 169, the first camera unit 100 and the second camera unit 13
Panorama video generation unit 175 that performs real-time panorama video synthesis processing on video data sent from 0, a video display unit (monitor) 178 that displays a panorama video generated by the panorama video generation unit 175, and a first camera unit A panorama video display communication unit 181 that receives video data transmitted from the second camera unit 100 and the second camera unit 130 and transmits shooting parameters of a TV camera to the first camera unit 100 and the second camera unit 130; Panoramic image display central control unit 1 for controlling the overall flow of the image display unit
84.

The TV camera 103 of the first camera unit 100 and the TV camera 133 of the second camera unit 130 are arranged side by side.

FIGS. 2 to 4 are flowcharts of the system according to the present embodiment. FIGS. 2 and 3 are flowcharts of the panoramic image display unit 160 of the system of the present embodiment.
FIG. 4 is a flowchart of the first camera unit 100 and the second camera unit 130 of the system according to the present embodiment.

First, a panoramic video display communication unit 18
In step 1, in step 200, a network communication connection is made between the first camera unit 100 and the second camera unit 130 as an initial setting. Then, in step 400, the first camera unit 100 and the second camera unit 130 make a network communication connection with the panoramic video display unit 160 as an initial setting.

Next, at step 205, the first camera unit 103 registered in the photographing parameter table 172 in advance.
TV camera 103 and TV camera of the second camera unit 130
The panorama image display communication unit 181 transmits the shooting parameters at the time of the initial posture to the first camera unit 100 and the second camera unit 130.

First, in the first camera section 100, step 4
At 05, the camera communication unit 112 receives the shooting parameters. Next, at step 410, the TV camera 103
Camera posture, pan, tilt, zoom
06 is controlled. Then, in step 415, the camera control unit 106 receives the imaging parameters measured by the TV camera 103 after the control, that is, the pan angle, the tilt angle, and the zoom ratio, and transmits them from the camera communication unit 112 to the panoramic video display unit 160.

The second camera unit 130 also uses the first camera unit 10
0, the camera communication unit 142 receives the shooting parameters in step 405, and in step 410, the posture of the TV camera 133, that is, pan, tilt,
The zoom is controlled by the camera control unit 136, and step 415 is performed.
In, the camera control unit 136 receives the imaging parameters measured by the TV camera 133 after the control, that is, the pan angle, the tilt angle, and the zoom ratio, and transmits the parameters from the camera communication unit 142 to the panorama video display unit 160.

Next, in the panoramic video display section 160, the panoramic video display communication section 181 receives the controlled shooting parameters transmitted from the first camera section 100 and the second camera section 130 in step 210. And
In step 215, the shooting parameters of the first camera unit 100 and the second camera unit 130 received in step 210 are temporarily stored in the shooting parameter table.

Next, at step 220, the panoramic video display communication section 181 transmits a video transmission request to the first camera section 100. Then, in step 420, a video transmission request is received by the camera communication unit 112, and in step 425, one video frame of the TV camera 103 is captured by the video capture unit 109. Then, in step 430, one frame of the video captured in step 425 is transmitted from the camera communication unit 112 to the panorama video display communication unit 181. Then, in step 225,
One frame of video transmitted from the first camera unit 100 is received by the panoramic video display communication unit 181.

Next, at step 230, the panoramic video display communication section 181 transmits a video transmission request to the second camera section 130. Then, in step 420, a video transmission request is received by the camera communication unit 142, and in step 425, one video frame of the TV camera 133 is captured by the video capture unit 139. Then, in step 430, one frame of the video captured in step 425 is transmitted from the camera communication unit 142 to the panorama video display communication unit 181. Then, in step 235,
One frame of the video transmitted from the second camera unit 130 is received by the panoramic video display communication unit 181.

Next, at step 250, step 225
And the first camera unit 100 received in step 235
The overlap area detection unit 187 detects an overlapping portion from one frame of the image of the second camera unit and one frame of the image of the second camera unit.

Here, a process for detecting an overlap area from two camera images will be described. FIG. 5 is a flowchart for detecting an overlap area.

In the overlap area detection processing of the present embodiment, two camera images are mosaic-processed in order to speed up the processing, and then the two images mosaiced are shifted little by little and superimposed to obtain the maximum correlation. Follow the process.

First, in step 500, initialization for performing the overlap area detection processing is performed. Here, the initial value 1000 of the correlation value in the overlapping part of the image of the first camera unit 100 and the image of the second camera unit 130 handled in steps 520 and 525 is set.

In step 505, the image of the first camera unit 100 and the image of the second camera unit 130 are subjected to mosaic processing.

FIG. 6 shows an image 600 captured by the first camera unit 100 and an image 620 captured by the second camera unit 130. Here, the first camera unit image 600 and the second camera unit image 620 are necessarily located on the right side (overlap area 610) of the first camera unit image 600 and on the left side (overlap area) of the second camera unit image 620 due to camera arrangement conditions. Overlap at the overlap region 630). The resolutions of the first camera unit image 600 and the second camera unit image 620 are 320 pixels horizontally and 240 pixels vertically.

In the mosaic processing, each image is divided into blocks each composed of 16 pixels × 16 pixels, the average luminance of each block is determined, and the luminance of each block is determined as the average luminance.

Next, at step 510, the image obtained by performing the mosaic processing on the first camera section image 600 and the second camera image 60 are displayed.
An initial position for correlating 0 with a mosaic-processed image is set. FIG. 7 is a diagram for explaining how to obtain a correlation between the image 700 obtained by performing the mosaic processing on the first camera unit image 600 and the image 710 obtained by performing the mosaic processing on the second camera image 720. In step 510, a position where the lower one block of the first camera unit mosaic image 700 and the upper one block of the second camera unit mosaic image 720 overlap and where each image overlaps in the horizontal direction without shifting is set as an initial position.

Subsequently, in step 515, the first camera unit mosaic image 700 and the second camera unit mosaic image 71
The correlation with the overlapping portion 720 with 0 is obtained. Here, the value of the correlation is the reciprocal of a value obtained by dividing the sum of the difference between the luminance of the block of the image 700 and the luminance of the block of the image 710 in the overlapping portion 720 by the total number of blocks in the overlapping portion 720,
First camera unit mosaic image 7 at overlapping portion 720
The more similar the 00 block and the block of the second camera unit mosaic image 710 are, the larger the correlation value is. If not, the value of the supervisor is small.

Next, at step 520, step 515
In order to obtain the minimum value of the correlation value obtained in the above, the current minimum value of the correlation is compared with the correlation value obtained in step 515. If the correlation value obtained in step 515 is smaller, the minimum value of the correlation is used as the correlation value obtained in step 515 in step 525, and the image obtained when the correlation value is obtained in step 515 in step 530. The overlapping position between 700 and the image 720 is stored. On the other hand, if the correlation value obtained in step 515 is larger or equal, the process proceeds to step 535.

In step 535, the second camera unit mosaic image 710 is set at a position where the second camera unit mosaic image 710 is slid one block higher than the first camera unit mosaic image 700. In step 540, it is determined whether or not the mosaic image overlap portion 720 exists. If there is an overlapping portion 720 of the mosaic video, the process returns to step 515. If there is no overlapping portion 720 of the mosaic video, the process proceeds to step 545, and the first
An image is set at a position where the lower one block of the camera unit mosaic image 700 and the upper one block of the second camera unit mosaic image 720 overlap, and in step 550, the second camera unit mosaic image 720 slides one block to the right. Set to the position that was made. In step 555, it is determined whether or not the overlapping portion 720 of the mosaic image exists,
If there is an overlapping portion 720 of the mosaic video, the process returns to step 515. If there is no overlapping portion 720 of the mosaic video, the process proceeds to step 550 to end the process.

The above is the overlap area detection processing.

Next, at step 245, the feature point extracting unit 1
At 66, a characteristic block is detected from the overlap area 610 of the first camera unit image 600, and the second camera unit image 620 corresponding to the characteristic block is further detected.
The block in the overlap area 630 is detected. Here, a characteristic block of the overlap area 610 is called a feature block, a block corresponding to the feature block of the overlap area 630 is called a corresponding block, and the size of the feature block and the corresponding block is 16 pixels × 16 pixels. I do.

FIGS. 8 and 9 are flowcharts of the feature point extracting process.

Here, the overlap area 610 of the first camera unit image 600 is divided into blocks of 16 pixels × 16 pixels, and the variance of each block is calculated. Excluded ones are set as feature block candidates. Then, among the block candidates, the four blocks at the upper left, lower left, upper right, and lower right are set as characteristic blocks. Then, using the feature block as a template, the second camera unit image 620 is scanned, and a portion having the largest correlation is detected as a corresponding block. Further, a rectangle connecting the four feature blocks and a rectangle connecting the four corresponding blocks are created to calculate the photographing parameters.

First, at step 800, feature point extraction processing is started, and the routine proceeds to step 805. Step 8
05 is the overlap area of the first camera section image 600
610 is divided into blocks of 16 pixels x 16 pixels,
The variance value of each block is calculated, and the process proceeds to step 810.
In step 810, a candidate flag = 1 indicating a candidate for a feature block is initially set for all blocks in the overlap region 610 of the first camera unit image 600, and the process proceeds to step 815. Here, the block with the candidate flag = 0 means that the block is outside the feature block candidate.

In steps 815 to 850, it is determined whether or not horizontally adjacent blocks have the same pattern, and blocks having the same pattern are excluded from the feature block candidates.

In step 815, the overlap area 610 is used as a block (hereinafter referred to as a reference block) for determining whether or not the variance values of adjacent blocks are similar.
Is set as the initial position.
FIG. 10 is a diagram for explaining scanning of a feature block from the overlap area 610 of the first camera section image 600. In step 810, an upper left block 1040 of the overlap area 610 is set as a reference block.

Next, in step 820, it is determined whether or not the reference block is the rightmost block 1050 of the overlap area 610. If not, the process proceeds to step 835. If the reference block is the rightmost block, the process proceeds to step 835. Proceed to step 825.

In step 825, the overlap area 6
It is determined whether or not 10 is the lower block 1060,
If it is the block at the lower end, the process proceeds to step 855,
If it is not the bottom block, go to step 830,
Move the reference block to the leftmost block one step down. For example, if it is the current reference block 1070, the process moves to block 1080.

Next, at step 835, the variance value of the reference block and the variance value of the block immediately to the right of the reference block are determined in order to determine whether the reference block and the block to the right of the reference block have similar patterns. Calculate the absolute value of the difference. For example, in the case of the reference block 1000, the absolute value of the difference between the variance of the reference block 1000 and the variance of the block 1010 on the right is calculated.

Next, in step 840, step 835
It is determined whether or not the absolute value of the difference between the variance values calculated in step (1) is less than a certain value 20. That is, when the absolute value of the difference between the variances is less than 20, it is determined whether or not the patterns of adjacent blocks are similar (not a characteristic point). If the absolute value of the difference between the variances is less than 20, the process proceeds to step 850; otherwise, the process proceeds to step 845, and the feature flag is set to 0 for the reference block and the block on the right. And then step 850
Proceed to.

In step 850, the reference block is moved one block to the right. For example, reference block 10
If it is 00, the reference block is slid to block 1010.

Then, the process returns to step 820.

In steps 855 to 940, it is determined whether or not vertically adjacent blocks have the same pattern, and blocks having the same pattern are excluded from the feature block candidates.

In step 855, the reference block position is set to the block 1 located at the upper left of the overlap area 610.
030 is set as the initial position, and the routine proceeds to step 900.

At step 900, it is determined whether or not the reference block is the lower end block 1060. If the reference block is the lower end block 1060, the flow proceeds to step 935; otherwise, the flow proceeds to step 905.

In step 905, the variance value of the reference block and the variance value of the block one stage below the reference block are determined in order to determine whether the reference block and the block one stage below the reference block have similar patterns. Calculate the absolute value of the difference of. For example, in the case of the reference block 1000, the absolute value of the difference between the variance value of the reference block 1000 and the variance value of the block 1020 one stage below is calculated.

Next, in step 910, step 905
It is determined whether or not the absolute value of the difference between the variance values calculated in step (1) is less than a certain value 20. If the absolute value of the difference between the variances is less than 20, the process proceeds to step 920; otherwise, the process proceeds to step 815, and the feature flag = 0 is set for the reference block and the block on the right thereof. Then, the process proceeds to step 820.

In step 920, it is determined whether or not the reference block is the rightmost block 1050 of the overlap area 610. If the reference block is the rightmost block, the flow advances to step 925 to move the reference block to the block one step below the leftmost block. And returns to step 900. If the block is not the rightmost block, the process proceeds to step 930, moves to the block on the right, and returns to step 900.

At step 935, points closest to the four corners of the overlap area 610 (upper left point, lower left point, upper right point, lower right point) are detected from the blocks having the candidate flag of 1, respectively. Let them be feature blocks.

Next, in step 940, step 93
Each of the four feature blocks detected in step 5 is used as a template, and the location that best matches the template is slid one pixel at a time in both the vertical and horizontal directions, and detected from the overlap area 630 of the second camera image 620. The four blocks most matching the four feature blocks are defined as corresponding blocks.

Next, in step 945, a rectangle is formed by connecting the four feature blocks with lines, and the rectangle is defined as a feature rectangle. Next, in step 950, a rectangle is formed by connecting four corresponding blocks with lines, and the rectangle is set as a corresponding rectangle.
Then, the process ends in step 955.

The above is the feature point extraction processing.

Next, in step 250, the photographing parameter calculation unit 169 calculates the characteristic rectangle of the first camera unit image and the second rectangle.
By comparing the corresponding rectangle of the camera unit image with the corresponding rectangle, the zoom ratio at which the size of the object photographed in each image is the same, and the pan angle and the tilt angle at which the perspective state of each image is the same are obtained.

Here, the photographing parameter calculation processing will be described.

FIG. 13 is a diagram showing a perspective (distortion degree) of a feature rectangle and a corresponding rectangle, and a method of obtaining a pan angle, a tilt angle, and a zoom rate from the size.

First, a method of correcting a pan angle will be described. Here, it is assumed that feature rectangle 1300 is a square. The pan angle of the TV camera 133 of the second camera unit 130 is controlled so that the corresponding area 1310 having a trapezoidal shape in which the left side and the right side are parallel to each other due to the perspective becomes the same shape (square) as the characteristic rectangle. That is, the ratio (perspective (degree of distortion)) of the length of the right side to the length of the left side of the characteristic rectangle 1300 of the first camera unit 100 and the ratio of the length of the right side to the length of the left side of the corresponding rectangle 1310 of the second camera unit 130 The pan angle is controlled so that (perspective (degree of distortion)) becomes the same.

Next, a method of correcting the tilt angle will be described. The tilt angle of the TV camera 133 of the second camera unit 130 is controlled so that the trapezoidal corresponding region 1320 whose upper side and lower side are parallel to each other due to the perspective makes the same shape (square) as the characteristic rectangle. That is, the ratio (perspective (degree of distortion)) of the length of the upper side to the lower side of the characteristic rectangle 1300 of the first camera unit 100 and the corresponding rectangle 132 of the second camera unit 130
The tilt angle is controlled so that the ratio of the length of the upper side to the length of the lower side of 0 (perspective (degree of distortion)) is the same.

Next, the correction of the zoom ratio will be described. T
The zoom ratio of the TV camera 133 of the second camera unit 130 is controlled to make the corresponding rectangle 1330 having a different size from the feature rectangle because the distance between the V camera and the shooting target is different to the same size as the feature rectangle. . That is, the zoom ratio is controlled so that the length around the characteristic rectangle 1300 of the first camera unit 100 and the length around the corresponding rectangle 1320 of the second camera unit 130 are the same.

FIGS. 11 and 12 are flowcharts of the photographing parameter calculation process.

First, in step 1100, a photographing parameter calculation process is started, and the flow advances to step 1105. In step 1105, a calculation flag = 0 for determining whether or not the calculation of the shooting parameters has been completed is set. The calculation flag increments the correction flag by one each time the calculation of the pan angle, the tilt angle, and the zoom ratio is completed, and ends the photographing parameter calculation process when the calculation flag becomes three.

Next, steps 1100 to 113
Until 0, the process of calculating the pan angle of the second camera unit 130 is performed.

In step 1100, to determine whether the parse (degree of distortion) of the feature rectangle and the corresponding rectangle is equal, the (length of left side / length of right side) of the feature rectangle and the (length of left side of the corresponding rectangle) are determined. It is determined whether the absolute value of the difference of (/ length of the right side) is less than a certain amount of 0.1, and if it is less than 0.1, the process proceeds to step 1130, where the value of the calculation flag is set to 1
After the addition, the process proceeds to step 1135. Otherwise, go to step 1115.

In step 1115, it is determined whether or not the characteristic rectangle (left side length / right side length) is larger than the corresponding rectangle (left side length / right side length). Then, the correction pan angle of the second camera unit 130 is set to 1 degree in the left direction, and the process proceeds to step 1135. If it is smaller, the process proceeds to step 1125 and the second camera unit 130
Is set to 1 degree in the right direction, and the flow advances to step 1135.

In step 1135, to determine whether the parse (degree of distortion) of the feature rectangle and the corresponding rectangle is equal, the (length of the upper side / length of the lower side) of the feature rectangle and the (length of the upper side) of the corresponding rectangle are determined. It is determined whether or not the absolute value of the difference of (/ length of the lower side) is less than a certain amount of 0.1, and if it is less than 0.1, the process proceeds to step 1155 and the value of the calculation flag is set to 1
After the addition, the process proceeds to step 1200. Otherwise, go to step 1140.

In step 1135, it is determined whether or not (length of the upper side / length of the lower side) of the characteristic rectangle is larger than (length of the upper side / length of the lower side) of the corresponding rectangle. Then, the correction tilt angle of the second camera unit 130 is set to 1 degree in the upward direction, and the process proceeds to step 1200. If it is smaller, the process proceeds to step 1150, where the second camera unit 13
At step 1200, the correction tilt angle of 0 is set to 1 degree downward.
Proceed to.

In step 1100, to determine whether the size of the feature rectangle and the corresponding rectangle are equal, the length around the feature rectangle (the length of the upper side + the length of the lower side + the length of the left side +
Whether the absolute value of the difference between the length of the right side) and the perimeter of the corresponding rectangle (top side length + bottom side length + left side length + right side length) is less than a certain amount, 1 Is determined, and if it is less than 1, the process proceeds to step 1220, where the value of the calculation flag is incremented by 1, and then the process proceeds to step 1225. Otherwise, the process proceeds to step 1205.

In step 1205, the length around the characteristic rectangle (length of the upper side + length of the lower side + length of the left side + length of the right side) is calculated as the length of the circumference of the corresponding rectangle (length of the upper side + length of the lower side). Length + left side length + right side length), and if it is larger, the process proceeds to step 1210, where the correction zoom amount of the second camera unit 130 is set to -10% (wide angle), and Proceed to 1225. If smaller, the process proceeds to step 1215, in which the correction zoom amount of the second camera unit 130 is increased by + 10%.
(Telephoto) and the process proceeds to step 1225.

At step 1225, it is determined whether or not the calculation flag = 3 to determine whether or not the photographing parameter calculation processing has been completed. If the calculation flag is 3, the process proceeds to step 1230, where the calculation process end flag is set to 1, and
Proceed to step 1240. Otherwise, the process proceeds to step 1235, where the calculation process end flag is set to 0, and the process proceeds to step 1240. In step 1240, the process proceeds to the next process.

The above is the photographing parameter calculation processing.

Next, in step 255, the panorama image display central control unit 184 executes the photographing parameter calculation process to execute the image of the first camera unit 100 and the second camera unit 1.
It is determined whether the perspective (distortion degree) of the 30 images and the size of the shooting target have become the same. Here, the determination is made based on the calculation end processing flag obtained in step 250, and when the calculation end processing flag = 0, that is, when the imaging parameter calculation processing is not completed, the process proceeds to step 260,
When the calculation end processing flag = 1, that is, when the photographing parameter calculation processing ends, the process proceeds to step 300.

Next, in step 260, step 250
The photographing parameters of the second camera unit 130 calculated in the above are transmitted from the panoramic video display communication unit 181 to the second camera unit 130.
And the process proceeds to step 265.

In the second camera section 130, in step 405, the camera communication section 112 receives the photographing parameters,
In step 410, the pan, tilt, and zoom of the TV camera 103 are controlled by the camera control unit 106, and
In step 15, the camera communication unit 112 transmits the panorama image display unit 160 with the shooting parameters measured by the TV camera 103 after control.

In step 265, the photographing parameters sent from the second camera section 130 are transmitted to the panorama image display communication section 1.
It receives at 81 and proceeds to step 270.

In step 270, the contents of the shooting parameters of the second camera unit 130 temporarily stored in the shooting parameter table 172 are updated to the shooting parameters received in step 265. Then, the process returns to step 230.

In step 300, the panoramic video display communication section 181 transmits a video transmission request to the first camera section 100 and the second camera section 130.

In the first camera section 100, step 420
When a video transmission request is received at step 425, the process proceeds to step 425, and the video capture unit 109 captures one frame of the video of the TV camera 103. Then, in step 430, one frame of the captured video is
Is transmitted to the panoramic video display unit 160.

Similarly, when the second camera unit 130 receives a video transmission request in step 420, the process proceeds to step 425, and the video capture unit 139 sets the TV.
Capture one frame of the image of camera 133, step 4
In step 30, one frame of the captured video is transmitted from the camera communication unit 142 to the panoramic video display unit 160.

In the panoramic image display section 160, in step 305, the first camera section 100 and the second camera section 13
The panorama image display communication unit 181 receives one frame of each image sent from 0.

Next, at step 310, the panorama video generation section 175 performs a panorama video generation process on the video received from the two cameras and synthesizes the video as one video.

Next, at step 315, the panoramic video synthesized by the panoramic video generator 175 is displayed on the video display 178, and the flow advances to step 320.

At step 320, it is determined whether or not a termination request from the user has been received from the input unit 163. If there is a termination request, the process proceeds to step 325. If there is no termination request, the process proceeds to step 330.

At step 325, the panorama image display communication unit 181 transmits an end command to the first camera unit 100 and the second camera unit 130, and proceeds to step 375. In step 375, the panorama image display central control unit 1
At 84, the end processing of the panoramic video display unit 160 is performed.

In the first camera section 100, step 435 is executed.
In step S440, when the camera communication unit 112 receives the end command, the process proceeds to step S440, and the camera central control unit 115 performs the end process of the first camera unit 100. Similarly, in the second camera unit 130, when the camera communication unit 142 receives an end command in step 435, the process proceeds to step 440.
To the second camera unit 1 in the camera central control unit 145.
Then, the end processing of 30 is performed. In step 330, the input unit 1
At 63, it is determined whether or not there is a photographing parameter registration request from the user. If there is a photographing parameter registration request, the process proceeds to step 335, where the photographing parameters temporarily stored in the photographing parameter table are registered in the photographing parameter table. Proceeding to step 340, if there is no photographing parameter registration request, proceeding to step 340.

In step 340, it is determined whether or not the user has selected a desired photographing parameter from a menu describing the contents of the photographing parameter table in the input unit 163. If so, the process proceeds to step 345; Goes to step 355.

In step 345, the first camera unit 100 and the second camera unit 13 in the panorama image display communication unit 181
In step 350, the panorama image display communication unit 181 transmits the shooting parameters to the first camera unit 1.
00 and the photographing parameters returned from the second camera unit 130, the end of the camera control of the two camera units is confirmed, and the process returns to step 300.

Next, in step 355, the user inputs a pan to the first camera unit 100 using the input unit 163.
It is determined whether a tilt or zoom operation has been performed. If no operation has been performed, the process returns to step 300, and if a user operation has been performed on the camera, the process proceeds to step 360.

In step 360, the panorama image display communication unit 181 sends the first camera unit 100 a step 355.
In step, the pan angle, tilt angle, and zoom ratio when the user operates the camera are transmitted. In step 365, the photographing parameters after camera control of the first camera unit 100 returned from the first camera unit 100 are transmitted in a panoramic video display communication. Part 18
1 to receive.

Next, in step 370, the s shooting parameters of the first camera unit 100 received in step 365 are temporarily stored in the shooting parameter table 172. Then, the process returns to step 220.

As described above, in the first embodiment, the photographing parameters of the two TV cameras that minimize the distortion due to the image composition are automatically calculated, the photographing parameters are stored in the table, and the user can instruct. By controlling the camera with the contents of the shooting direction table (shooting parameters) that the user wants to monitor, the difference in the roughness of the two images due to the image enlargement processing in the panorama synthesis processing and the unnaturalness of the images due to the perspective correction processing Distortion can be prevented.

Next, a second embodiment of the present invention will be described with reference to FIGS.

FIG. 14 is an arrangement diagram of a TV camera according to the second embodiment. In the second embodiment, TV cameras are arranged side by side with respect to a long shooting target such as a stadium, and when the user operates the center camera 1503, the left and right TV cameras (left sub Camera 152
3 and the right sub camera 1543) are automatically controlled to a shooting direction and a zoom ratio that can generate a panoramic video without distortion. In the system according to the present embodiment, the panoramic video is obtained by performing live panorama synthesis processing of live video from three cameras in real time and displaying a panoramic video of a moving image. In the system of the present embodiment, a plurality of TV cameras (TV camera 1410, TV camera)
Camera 1523, TV camera 1503, TV camera 15
43, TV camera 1420), the user can arbitrarily select the TV camera that the user wants to operate, and the system sets the left and right TV cameras selected by the user to the optimal shooting direction and zoom ratio for panoramic video generation. Control automatically. However, the TV camera installed at the end is T
V camera 1410 and TV camera 1420 cannot be selected. In this embodiment, for example, when the user shoots a soccer player who is playing at the stadium, the TV camera closest to the player is selected. Then, when the position of the player who wants to photograph is greatly changed by passing the ball, the user selects a TV camera closest to the player who wants to newly photograph. All the TV cameras of this embodiment can control the pan, tilt, rotation around the photographing direction axis, front / rear, left / right and up / down directions and the zoom ratio.

FIG. 15 is a system configuration diagram of the present embodiment.

This system comprises a main camera unit 1500 which allows the user to control the pan and tilt of the TV camera, rotation around the photographing direction axis, front / rear, left / right and up / down directions.
The shooting parameters of the TV camera (pan angle, tilt angle, rotation angle around the shooting direction axis, front / rear left / right up / down movement distance, and zoom ratio) are controlled according to the perspective, tilt, and size of the object in the video shot at 1500. The left and right sub camera sections (left sub camera section 1520 and right sub camera section 1540) are compared with the images of the main camera section 1500 and the left and right sub camera sections to calculate shooting parameters of the left and right sub camera sections. Section 1500 and left sub camera section 1520
And a panoramic video display unit 1560 that performs panorama synthesis processing on live video captured by the right camera unit 1540 and displays the processed live video.
The main camera unit 1500, the left sub-camera unit 1520, the right sub-camera unit 1540, and the panoramic video display unit 1560 include a network 1590 for communicating video data and shooting parameters.

A main camera section 1500 is a TV camera 1503 capable of controlling pan, tilt, rotation around a shooting direction axis, front / rear, left / right and up / down directions for shooting an image of a site.
And commands for the pan angle, tilt angle, and zoom ratio to the TV camera 1503, and the photographing parameters (pan angle, tilt angle, rotation angle around the photographing direction axis, front / rear left / right and up / down moving distances) measured by the TV camera 103. Camera control unit 1506 for receiving a zoom ratio), a video capture unit 1509 for receiving an analog video signal from the TV camera 1503 and capturing the video, and a panoramic video display unit 160
From the user (pan angle, tilt angle, rotation angle around the shooting direction axis, front / rear left / right up / down movement distance and zoom rate) transmitted by the user, or measured by the camera control unit 1506 A camera communication unit 1512 for transmitting a shooting parameter to the panoramic video display unit 160 and transmitting video data captured by the video capture unit 1509 to the panoramic video display unit 1560.
And a camera central control unit 1515 that controls the overall flow of the main camera unit 1500.

The left sub camera unit 1520 has the same configuration as the main camera unit 1500, and includes a TV camera unit 1523, a camera control unit 1526, a video capture unit 1529,
It comprises a camera communication unit 1532 and a camera central control unit 1535. The function of each unit of the left sub camera unit 1520 is the same as the function of each unit of the main camera unit 1500. The right sub camera unit 1540 has the same configuration as the main camera unit 1500, and includes a TV camera unit 1543 and a camera control unit 154.
6, a video capture unit 1549, and a camera communication unit 15
52, and a camera central control unit 1555. The function of each unit of the right sub camera unit 1540 is
00 has the same function as each section. Panoramic video display 1
Reference numeral 560 denotes an input unit (mouse) 1563 that receives an input from a user, a main camera unit 1500, and left and right sub camera units (a left sub camera unit 1520 and a right sub camera unit 15).
An overlap area detection unit 1572 that obtains an area where the respective images captured in 40) overlap each other.
And a characteristic block in one frame of each image captured by the main camera unit 1500 and the left and right sub camera units (the left sub camera unit 1520 and the right sub camera unit 1540), that is, the image is 16 pixels × 16 pixels A feature point extracting unit 1566 for obtaining a block having the largest variance when divided into blocks each including
Using the feature points extracted in step 1 to compare the size and perspective of the object in the video, and calculate the shooting parameters of the left and right sub-camera units (left sub-camera unit 1520, right sub-camera unit 1540) from the comparison result Calculation unit 1569
A panorama video generation unit 1575 that performs real-time panorama video synthesis processing on video data sent from the main camera unit 1500 and the left and right sub camera units (the left sub camera unit 1520 and the right sub camera unit 1540); An image display unit (monitor) 1578 that displays a panoramic image generated by the 1575, a main camera unit 1500, and left and right sub camera units (left sub camera unit 152)
0, the right sub-camera unit 1540), the main camera unit 1500 and the left and right sub-camera units (the left sub-camera unit 1520, the right sub-camera unit 1).
540), a panorama video display communication unit 1581 for transmitting the shooting parameters of the TV camera, and a panorama video display central control unit 1584 for controlling the overall flow of the panorama video display unit.

FIGS. 16 to 19 are flowcharts of the second embodiment. 17 to 19 are flowcharts on the side of the panoramic video display unit 1560, and FIG. 19 is a flowchart on the side of the main camera unit 1500, the left sub camera unit 1520, and the right sub camera unit 1540.

First, a panoramic image display unit 1560
On the side, the process starts at step 1600 and proceeds to step 160
Go to 5. In step 1605, the user has many TVs
The user selects a TV camera to operate from the cameras. When the user selects a TV camera, the monitor 1
The desired T from the camera layout diagram displayed at 578
Click the V camera. The current position and shooting direction of the TV camera are displayed on the camera layout.

When the user selects a camera, step 16
In 10, the camera designated by the user is registered as the main camera in the panorama image display central control unit 1584,
A camera located on the left side of the camera designated by the user is registered as a left sub camera, and a camera located on the right side of the camera designated by the user is registered as a right sub camera. In a default state (an initial state in which there is no designation from the user), the main camera is the camera located at the center, and the cameras located on the left and right thereof are the left and right sub cameras.

Next, in step 1615, the panorama image display / communication section 1581 gives the main camera section 1500 shooting parameters of the initial posture (pan angle, tilt angle, rotation angle around the shooting direction axis, front / rear left / right up / down moving distance, zoom ratio). Send

On the main camera unit 1500 side, the photographing parameters are received by the camera communication unit 1512 in step 1905, and then the camera control unit 1512 in step 1910.
06 and the TV camera 15
03 and TV camera 1 in step 1915
The camera parameters after control measured in step 503 are transmitted from the camera communication unit 1512 to the panorama image display unit 1560. Similarly, in the left sub camera unit 1520, step 1905
In step 1910, the camera parameters are received in the camera communication unit 1532, then in step 1910, the TV camera 1523 is controlled by the shooting parameters received in the camera control unit 1526, and in step 1915, the controlled shooting parameters measured by the TV camera 1523 are used. Camera communication unit 153
2 to the panorama video display unit 1560. Similarly, the right sub-camera unit 1540 receives the shooting parameters in the camera communication unit 1532 in step 1905, and then controls the TV camera 1543 with the shooting parameters received in the camera control unit 1546 in step 1910.
Then, in step 1915, the camera parameters after control measured by the TV camera 1543 are transmitted from the camera communication unit 1552 to the panoramic video display unit 1560.

In step 1620, the panorama image display unit 1560 receives the shooting parameters sent from the main camera unit 1500, the left sub camera unit 1520, and the right sub camera unit 1540 in the panorama image display communication unit 1581, and proceeds to step 1625. . In step 1625, a left camera shooting parameter calculation end flag indicating whether or not the shooting process of the left sub-camera unit 1520 has been completed is set to 0, and whether or not the shooting process of the right sub-camera unit 1540 has been completed is determined. Is set to the right camera shooting parameter calculation end flag = 0.

Next, at step 1630, the panoramic video display communication section 1581 transmits a video transmission request to the main camera section 1500. In the main camera section 1500, in step 1920, the panoramic image display communication section 15
The camera communication unit 1512 receives the video transmission request from 81. Then, in step 1925, the video capture unit 1509 captures one frame of the video of the TV camera 1503, and in step 1930, the camera communication unit 15
12 transmits one frame of video.

Next, in step 1635, the panorama image display communication section 1581 receives one frame of image transmitted from the main camera section 1500 in the panorama image display section 1560.

Next, steps 1640 to 168
Up to 5, correction processing of the shooting direction of the left sub-camera unit 1520 is performed.

At step 1640, panoramic image display central control section 1584 determines whether or not the left camera shooting parameter calculation end flag = 1 indicating that the correction processing of the shooting direction of left sub camera section 1520 has been completed. When the photographing parameter calculation end flag = 0, it indicates that the correction processing has not been completed. If the left camera shooting parameter calculation end flag = 1, proceed to step 1700; if the left camera shooting parameter calculation end flag = 0, go to step 1700.
Proceed to 645.

Next, in step 1645, the panoramic image display communication section 1581 transmits a video transmission request to the left sub camera section 1520, and in step 1650, receives one frame of video transmitted from the left sub camera section 1520.

Next, at step 1655, the main camera section 1500 is detected by the overlap area detecting section 1572.
And an overlap area between the video transmitted from the sub-camera unit 1520 and the video transmitted from the left sub-camera unit 1520 is detected.
The method of detecting the overlap area is as described in the first embodiment.

Next, at step 1660, the feature point extracting unit 1
At 566, a feature rectangle is detected from the overlap area of the video sent from the main camera unit 1500,
A corresponding rectangle is extracted from the video overlap area sent from the left sub camera unit 1520. The method of extracting the feature rectangle and the corresponding rectangle is as described in the first embodiment.

Next, at step 1665, a photographing parameter of the left sub camera unit 1520 is calculated by the photographing parameter calculating unit 1569. Here, the calculation procedure of the imaging parameters will be described. FIG. 20 to FIG. 22 are flowcharts showing the calculation processing of the photographing parameters.

First, in step 2000, photographing parameter processing is started. And step 2005
In, the movement correction flag for determining whether or not the movement control of the TV camera in the left-right direction and the up-down direction is completed is set to 0. When the movement correction flag is 3, it indicates that the movement control in the left-right direction and the up-down direction has been completed, and when the value is any other value, it indicates that the movement control has not been completed.

In steps 2010 to 2060, the optimal TV is set based on the position and size of the overlap area.
This is a process for calculating the movement position of the camera in the horizontal direction and the vertical direction. Here, the process of calculating the moving position of the TV camera will be described. FIG. 23 shows a state in which an image 2300 captured by the main camera unit 1500 and an image 2310 captured by the right sub camera unit 1540 overlap. As shown in FIG.
When the position is above the right sub camera image 2310, the camera is moved upward, and when it is below, the camera is moved downward. Also, overlap area 2
When the ratio of the horizontal length 2340 of the right sub camera unit 1540 to the horizontal length 2330 of 330 is large (the overlapping portion is large), the camera is slid in the direction away from the main camera, and the ratio is large ( (The overlapping part is small.) Slide the camera toward the main camera. The same applies to the image of the left sub camera unit 1520.

In step 2010, it is determined whether or not the overlap in the horizontal direction of the overlap area between the main camera section image and the left sub camera section image is less than 30%. The process proceeds to step 2025 in which the amount of horizontal movement correction of the left sub-camera unit 1520 is set to 50 cm in the direction approaching the main camera, and if not, the movement correction flag is set to 1 in step 2020.
Then, the process proceeds to step 2025.

In step 2025, it is determined whether or not the overlap in the horizontal direction of the overlap area between the main camera section video and the left sub camera section video is 50% or more. If the overlap is 50% or more, step 2030 is performed. The horizontal movement correction amount of the left sub camera unit 1520 is set to 50 cm in a direction away from the main camera, and the process proceeds to step 2040.
Otherwise, at step 2035, the movement correction flag is incremented by 1, and the routine proceeds to step 2040.

In step 2040, it is determined whether or not the vertical overlapping portion of the overlap area is 90% or more. If it is 90% or more, the flow advances to step 2050 to increment the movement correction flag by one. If it is less than 90%, the process proceeds to step 2045. In step 2045, the position of the overlap area is determined. If the overlap area is below the right camera image, the process proceeds to step 2055 and the left sub camera unit 1520
The upward / downward correction movement amount is set to 50 cm in the upward direction, and the process proceeds to step 2100.
The process proceeds to 0, and the vertical movement of the left sub-camera unit 1520 is set to 50 cm in the downward direction.

In step 2100, the movement correction flag is set to 3
Is determined, and when the movement correction flag is 3 (when the movement correction control of the camera is completed), step 22 is performed.
The process proceeds to step 50, and if the movement correction flag is less than 3 (if the camera is under the movement correction control), the process proceeds to step 2105, the shooting direction correction flag is set to 0, and the process proceeds to step 2110. The shooting direction correction flag is a flag for determining whether or not the correction processing of the camera's pan angle, tilt angle, rotation angle around the shooting direction axis, and zoom ratio has been completed. Indicates that the correction processing of the shooting direction has been completed. Step 2110 to step 214
Steps up to 5 are processes for calculating the pan angle, the tilt angle, the rotation angle around the shooting direction axis, and the zoom ratio of the TV camera from the perspective, size, and inclination of the feature rectangle and the corresponding rectangle.

Here, the photographing parameter calculation processing will be described.

FIG. 24 is a diagram showing a perspective (distortion degree) of a feature rectangle and a corresponding rectangle, and a method of obtaining a pan angle, a tilt angle, and a zoom ratio from the size.

First, a method of correcting the pan angle will be described. Here, it is assumed that feature rectangle 1300 is a square. The pan angles of the TV cameras of the left and right sub-camera units are controlled so that the trapezoidal corresponding region 1310 whose left side and right side are parallel to each other due to the perspective makes the same shape (square) as the characteristic rectangle. In other words, the ratio (perspective (degree of distortion)) of the length of the right side and the length of the left side of the characteristic rectangle 1300 of the left and right sub camera unit and the ratio of the length of the right side and the length of the left side of the corresponding rectangle 1310 of the left and right sub camera unit (parse (Distortion degree)) is controlled to be the same as the pan angle.

Next, a method of correcting the tilt angle will be described. The tilt angles of the TV cameras of the left and right sub-camera units are controlled so that the trapezoidal corresponding area 1320 whose upper side and lower side are parallel due to the perspective is made the same shape (square) as the characteristic rectangle. That is, the characteristic rectangle 1300 of the main camera unit
Ratio of the length of the upper side to the length of the lower side (Perspective (degree of distortion))
The tilt angle is controlled such that the ratio of the length of the upper side and the length of the lower side (perspective (degree of distortion)) of the corresponding rectangle 1320 of the left and right sub camera units is the same.

Next, the correction of the zoom ratio will be described. T
The zoom ratio of the TV cameras in the left and right sub-camera units is controlled to make the corresponding rectangle 1330 having a different size from the feature rectangle because the distance between the V camera and the shooting target is different from the feature rectangle. That is, the zoom rate is controlled so that the length around the characteristic rectangle 1300 of the main camera unit and the length around the corresponding rectangle 1320 of the left and right sub camera units are the same.

Next, the correction of the rotation angle of the camera about the photographing direction axis will be described. If the TV camera installation positions of the main camera and the left and right sub cameras are shifted, the inclination of the characteristic rectangle 1300 and the corresponding rectangle 1340 will be different. Therefore, in order to equalize the inclination of the feature rectangle 1300 and the corresponding rectangle 1340, the rotation angles of the left and right sub camera units around the shooting direction axis of the TV camera are controlled. That is, the characteristic rectangle 13 of the main camera unit
The rotation angle around the photographing direction axis is controlled so that the inclination of 00 and the inclination of the corresponding rectangle 1320 of the left and right sub camera units are the same.

In step 2110, to determine whether the parse (degree of distortion) of the feature rectangle and the corresponding rectangle are equal, the (length of left side / length of right side) of the feature rectangle and the (length of left side of the corresponding rectangle) are determined. It is determined whether or not the absolute value of the difference of (/ length of the right side) is less than a certain amount of 0.1, and if it is less than 0.1, the process proceeds to step 2210, and the value of the shooting direction correction flag is set to 1 After the addition, the process proceeds to step 2135. Otherwise, go to step 2115.

In step 2115, it is determined whether or not the characteristic rectangle (left side length / right side length) is larger than the corresponding rectangle (left side length / right side length). Then, the correction pan angle of the left sub camera unit 1520 is set to 1 degree in a direction approaching the main camera, and the process proceeds to step 2135. If it is smaller, the process proceeds to step 2130, the correction pan angle of the left sub-camera unit 1520 is set to 1 degree in a direction away from the main camera, and the process proceeds to step 2135.

In step 2135, in order to determine whether the parse (degree of distortion) of the feature rectangle and the corresponding rectangle is equal, the (length of the upper side / length of the lower side) of the feature rectangle and the (length of the upper side) of the corresponding rectangle are determined. It is determined whether or not the absolute value of the difference of (/ length of the lower side) is less than a certain amount of 0.1, and if it is less than 0.1, the process proceeds to step 2145 to set the value of the shooting direction correction flag to 1 After the addition, the process proceeds to step 2200. Otherwise, go to step 2140.

In step 2140, it is determined whether or not (length of the upper side / length of the lower side) of the characteristic rectangle is larger than (length of the upper side / length of the lower side) of the corresponding rectangle. Then, the correction tilt angle of the left sub-camera unit 1520 is set to 1 degree in the upward direction, and the flow proceeds to step 2200.
If it is smaller, the process proceeds to step 2155, in which the correction tilt angle of the left sub-camera unit 1520 is set to 1 degree downward, and the process proceeds to step 2200.

In step 2200, to determine whether the size of the feature rectangle is equal to the size of the corresponding rectangle, the length around the feature rectangle (the length of the upper side + the length of the lower side + the length of the left side +
Determines whether the absolute value of the difference between the length of the corresponding rectangle and the length of the periphery of the corresponding rectangle (length of the upper side + length of the lower side + length of the left side + length of the right side) is smaller than a certain amount 1 If it is less than 1, the process proceeds to step 2210, and the value of the shooting direction correction flag is set to 1
After the addition, the process proceeds to step 2225. Otherwise, go to step 2205.

In step 2205, the perimeter of the feature rectangle (the length of the upper side + the length of the lower side + the length of the left side + the length of the right side) is determined by the perimeter of the corresponding rectangle (the length of the upper side + the length of the lower side). Length + left side length + right side length). If it is larger, the process proceeds to step 2215, where the correction zoom amount of the left sub camera unit 1520 is set to -10% (wide angle), and Proceed to 2225. If smaller, step 222
0, and increases the correction zoom amount of the left sub camera unit 1520 by +
It is set to 10% (telephoto) and the process proceeds to step 2225.

In step 2225, it is determined whether or not the inclination of the characteristic rectangle is equal to the inclination of the corresponding rectangle. Therefore, when the absolute value of the difference between the inclination of the left side of the characteristic rectangle and the inclination of the left side of the corresponding rectangle is smaller than a certain amount of less than 1 degree. It is determined whether or not there is, and if it is less than 1 degree, the process proceeds to step 2235, and the value of the shooting direction correction flag is added by 1, and then the process proceeds to step 2250. Otherwise, go to step 2230.

In step 2230, if the left side of the corresponding rectangle is inclined clockwise with respect to the left side of the feature rectangle, the flow advances to step 2240 to correct the amount of correction for the rotation angle of the left sub-camera unit 1520 about the shooting direction axis counterclockwise. 1 in direction
In step 2250, if the camera is tilted counterclockwise, the flow proceeds to step 2245, and the left sub camera unit 1
The correction amount related to the rotation angle about the shooting direction axis at 520 is set to 1 degree in the clockwise direction, and the process proceeds to step 2250.

In step 2250, in order to determine whether or not the photographing parameter calculation processing has been completed, it is determined whether or not the movement correction flag = 3 and the photographing direction correction flag = 4. If the flag is 4, the flow advances to step 2255 to set the correction end flag = 1, and if not, the flow advances to step 2260 to set the correction end flag = 0. Then, in step 2265, the process proceeds to the next process.

The above is the photographing parameter calculation processing of the left sub camera unit 1520.

Next, in step 1670, it is determined whether or not the calculation end flag is 1 in order to determine whether or not the panorama photographing parameter calculation processing has been completed. Step 1700: Set camera photographing parameter calculation end flag to 1
If not 1, the process proceeds to step 1675, where the photographing parameters calculated in step 1665 are transmitted to the left sub-camera unit 1520, and the photographing parameters from the left sub-camera unit 1520 are received in step 1680.

Next, steps 1700 to 174
Up to 5, correction processing of the photographing direction of the right sub camera unit 1540 is performed.

In step 1700, the panoramic image display central control unit 1584 determines whether or not the right camera shooting parameter calculation end flag = 1 indicating that the correction processing of the right sub camera unit 1540 in the shooting direction has ended. When the right camera shooting parameter calculation end flag = 0, it indicates that the correction process has not been completed. When the right camera shooting parameter calculation end flag = 1, the process proceeds to step 1750, and when the left camera shooting parameter calculation end flag = 0, the process proceeds to step 1705.

Next, in step 1705, the panoramic image display communication section 1581 transmits a video transmission request to the right sub camera section 1540, and in step 1710, receives one frame of video transmitted from the right sub camera section 1540.

Next, at step 1715, the overlap area detection section 1572 causes the main camera section 1500 to operate.
And an overlap area between the image transmitted from the sub camera unit 1540 and the image transmitted from the right sub camera unit 1540 is detected.
The method of detecting the overlap area is as described in the first embodiment.

Next, at step 1720, the feature point extracting unit 1
At 566, a feature rectangle is detected from the overlap area of the video sent from the main camera unit 1500,
A corresponding rectangle is extracted from the video overlap area sent from the right sub camera unit 1540. The method of extracting the feature rectangle and the corresponding rectangle is as described in the first embodiment.

Next, at step 1665, a photographing parameter of the right sub-camera unit 1540 is calculated by the photographing parameter calculating unit 1569. The shooting parameter calculation processing is as described in step 1665.

Next, at step 1730, it is determined whether or not the calculation end flag is 1 in order to determine whether or not the panorama shooting parameter calculation processing has been completed. Step 1750 sets the camera photographing parameter calculation end flag to 1
If not 1, the process proceeds to step 1740 to transmit the photographing parameters calculated in step 1725 to the right sub camera unit 1540, and receives the photographing parameters from the right sub camera unit 1540 in step 1745.

Next, in step 1750, in order to determine whether or not the correction of the cameras of the left and right sub camera units has been completed,
The panorama video display central control unit 1584 determines whether the left camera shooting parameter calculation end = 1 and the right camera shooting parameter calculation end = 1.

If the left camera shooting parameter calculation end = 1 and the right camera shooting parameter calculation end = 1, proceed to step 1800; otherwise, go to step 16
Returning to step 40, the photographing parameters of the left and right sub camera units are calculated.

Next, from step 1800 to step 181
Up to 5, the main camera section 1500 and the left sub camera section 15
A panoramic video display process is performed to perform panoramic synthesis of the video from the sub-camera 20 and the right sub-camera unit 1540 for display.

At step 1800, a video transmission request is transmitted from panoramic video display communication section 1581 to main camera section 1500, left sub camera section 1520, and right sub camera section 1540. Then, in step 1805, the panorama image display communication unit 1581 receives the images transmitted from the left sub camera unit 1520 and the right sub camera unit 1540. Next, in step 1810, the three images received by the panorama image generation unit 1575 are subjected to panorama synthesis processing to make one panorama image. And step 1
In step 815, the panorama-composed image is displayed on the monitor 1578.
To be displayed.

Next, at step 1820, the monitor 157
It is determined whether or not the user has pressed the end button displayed in Step 8 with the mouse 1563. If the end button has been pressed here, that is, if there is an end request, Step 18 is executed.
Proceed to step 30; if not pressed, proceed to step 1825. In step 1830, an end command is transmitted to main camera 1500, left sub camera 1520, and right sub camera 1540. And step 1 on the camera side
At 935, an end command is received, and at step 1940, end processing is performed in the camera central control unit.

At step 1825, it is determined whether or not the main camera has been designated by clicking the mouse 1563 from the camera layout displayed on the monitor 1578.

If the main camera is designated, the flow returns to step 1610; otherwise, the flow proceeds to step 1835.

Subsequently, at step 1835, the monitor 15
It is determined whether or not the camera operation buttons for panning, tilting, moving up and down, right and left, and zooming the camera displayed at 78 have been operated by the mouse 1563. If an operation has been performed, the process proceeds to step 1840; Return to step 1800.

In step 1840, the photographing parameters input by the camera operation buttons are
In step 1845, the control unit 500 receives the controlled shooting parameters transmitted from the main camera unit 1500.

Then, the process returns to step 1610.

As described above, in the second embodiment, when the user specifies the camera to be operated and the user operates the shooting direction, the shooting parameters that minimize the distortion due to the image synthesis with the left and right cameras are automatically set. By controlling the left and right cameras, it is possible to prevent the difference in the roughness of the synthesized image due to the image enlargement processing in the panorama synthesis processing and the unnatural distortion of the image due to the perspective correction processing from occurring. .

Next, a third embodiment of the present invention will be described with reference to FIGS.

FIG. 25 shows a digital camera according to the third embodiment. In the third embodiment, a digital camera that can capture an image in an optimal shooting direction in order to create a panoramic image without distortion will be described. The digital camera according to the present embodiment can capture a still image, and the user can take a picture according to the guidance of the optimal shooting direction displayed on a viewfinder or a monitor with a built-in digital camera to generate a panoramic image without distortion. Can be obtained.

FIG. 25 is a diagram of the digital camera of this embodiment. LCD monitor 2 on the back of the digital camera 2500
510 is built in, and the user can use the LCD monitor 2510
Shoot while watching the preview video displayed in. Then, a graphic 2520 (hereinafter referred to as a shooting direction guidance) for guiding the optimum shooting direction is displayed on the liquid crystal monitor 2510, and the user pans or tilts the camera by hand in accordance with the shooting direction guidance 2520 according to the shooting direction guidance 2520. Turn the shutter around and release the shutter when the optimal shooting direction is reached. The shooting direction guidance is displayed when the shutter is half-pressed.

Here, the guidance method of the shooting guidance 2520 will be described. FIG. 26 shows the shooting guidance 25.
It is a figure showing 20 guidance methods. The shooting guidance 2520 includes an arrow 2605 urging a left panning operation.
An arrow 2610 urging the user to perform a downward tilt operation, an arrow 2615 urging the user to perform a downward tilt operation, an arrow 2620 urging an upward tilt operation, and an arrow 2625 urging a clockwise rotation operation of the shooting direction axis. And an arrow 2630 that prompts the user to rotate the shooting direction axis in a counterclockwise direction. Then, an arrow related to a necessary operation is turned on, and when the operation becomes unnecessary (in a state where the positioning is completed), the arrow is turned off.
Then, when the positioning can be performed in all directions, the shooting direction guidance 2 in a state where the entire graphic is turned on.
650 is displayed to prompt the user to perform a shutter operation. If there is no overlap between the image taken immediately before and the image to be taken, the shooting direction guidance flashes and an alarm is issued to the user.

FIG. 27 is a system configuration diagram of the third embodiment.

The present system comprises a photographing unit 2705 for photographing a photographing target as a digital image and a first image memory 27 for temporarily storing one preview image photographed by the photographing unit 2705.
10, a second image memory 2715 for temporarily storing one image captured immediately before the latest image, an image storage unit 2720 for storing images captured by the imaging unit 2705, a first image memory 2710, An overlap area detection unit 2725 for detecting an overlap area from the image of the two-image memory 2715, and 16 blocks characteristic of the overlap area of the image of the first image memory 2710, that is, 16
When divided into blocks of pixels × 16 pixels, a feature point extracting unit 2730 for obtaining a block having the largest variance value and a corresponding block in the image of the second image memory 2715 corresponding to the block, A photographing parameter calculating unit 2735 to be calculated, a display unit 2740 for displaying a preview image or a photographing direction guidance, a shutter button 2745 for taking in a video from the photographing unit 2705, and controls the flow of the entire processing. And a central control unit 2750.

FIGS. 28 and 29 are flowcharts of the third embodiment.

First, in step 2800, the processing is started, and the flow advances to step 2805. Step 2805
Then, a preview of the image captured by the image capturing unit 2705 is displayed on the display unit 2740. Next, in step 2810, the user presses the shutter button 2 in the central control unit 2750.
It is determined whether or not 745 has been pressed, and if it has not been pressed, the process returns to step 2805 to continuously display a preview.
Step 2 when the shutter button 2745 is pressed
Proceed to 815. In step 2815, the image photographed by the photographing unit 2705 is transferred to the second image memory 2715 and is temporarily stored.
The image temporarily stored in No. 15 is copied and stored in the image storage unit 2720, and the process proceeds to Step 2830.

Next, at step 2830, the display unit 274
At 0, a preview of the image captured by the image capturing unit 2705 is displayed. Then, in step 2835, the central control unit 275
At 0, it is determined whether or not the shutter button 2745 is half-pressed. If the shutter button 2745 is half-pressed, the process proceeds to step 2840; otherwise, the process returns to step 2830 and the captured image is displayed on the display unit 2740. Display a preview. In step 2850, the first image memory 271
Then, the preview image data acquired by the photographing unit 2705 is transferred to and temporarily stored.

By the above processing, the first image memory 271
At 0, the preview image data is temporarily stored, and at the second image memory 2715, the image data obtained when the shutter is last released is temporarily stored.

Next, at step 2855, the overlap area detection unit 2725 detects the overlap area between the image temporarily stored in the first image memory 2710 and the image temporarily stored in the second image memory. The method of detecting the overlap area is as described in the first embodiment. However, in the present embodiment, the image in the first image memory 2710 and the second image memory 2
If the value of the correlation with the image 715 is extremely small, it is determined that no overlap area exists. Next, in step 2860, the central control unit 2750 determines whether or not there is an overlap area based on the detection result of the overlap area detection unit 2725. If there is an overlap area, the process proceeds to step 2865, where the overlap area is determined. If does not exist, the flow advances to step 2870 to blink the shooting direction guidance graphic displayed on the display unit 2740 to warn the user that there is no overlap area. And step 2830
Return to

In step 2865, the feature point extracting unit 273
At 0, the feature rectangle of the image temporarily stored in the first image memory 2710 and the corresponding rectangle of the image temporarily stored in the second image memory 2715 are extracted. The feature point extraction processing in the feature point extraction unit 2730 is as described in the first embodiment.

Next, at step 2900, a photographing parameter calculation unit 2735 calculates an optimum photographing direction of the camera. The calculation method of the shooting direction is as shown in the shooting parameter calculation processing of the second embodiment. However, here, only the direction to be corrected by the camera regarding the pan, tilt, and rotation of the camera around the shooting direction axis is calculated.

Next, at step 2905, the display unit 2740
Is displayed using the shooting direction guidance, and the user is instructed on the shooting direction to be corrected by the camera (pan, tilt, direction of rotation around the shooting direction axis).

Next, at step 2910, the central control unit 2
At 750, it is determined whether or not the correction of the shooting direction of the camera has been completed based on the calculation result of the shooting parameter calculation unit 2735 at step 2900, and if the correction has been completed, the process proceeds to step 2915; Return to 2830.

At step 2915, the graphic of the shooting direction guidance indicating that the shooting direction of the camera has ended is illuminated on the display unit 2740.

Next, at step 2920, the central control unit 2
At 750, it is determined whether or not the shutter button 2745 has been pressed. If the shutter button 2745 has been pressed, the process proceeds to step 2925, and if not, the process returns to step 2830.

At step 2925, the image data photographed by the photographing unit 2705 is transferred to the second image memory 2715 and is temporarily stored. Then, in step 2930, the second
The image temporarily stored in the image memory 2715 is copied and stored in the image storage unit 2720.

Next, a fourth embodiment of the present invention will be described with reference to FIGS.

FIG. 30 shows a state of shooting by the aerial photographing system of the fourth embodiment. In the fourth embodiment, when a plurality of aerial photographs are joined to create a panoramic aerial photograph, an aerial photographing camera system that prevents distortion of the panoramic aerial photograph by equalizing the photographing directions of the cameras at the time of image capturing. It is.

In order to take an aerial photograph, the airplane 300
A TV camera 3100 capable of controlling pan, tilt, zoom, and rotation around an axis in a shooting direction is mounted below 0. The TV camera 3100 mounted on the airplane 3000 captures an image of the capturing range 3020. And the photographed area 30
20 and a range 3030 to be photographed next are photographed so as to have an optimal overlap area 3040 (an area of about 35% of the entire area). The photographing is automatically performed by the camera system of this embodiment.

FIG. 31 is a system configuration diagram of the aerial photographing camera system of this embodiment. The camera system for aerial photography according to the present embodiment includes a TV camera 3100 that can control pan, tilt, zoom, and rotation around a photographing direction axis for photographing the ground, and a video capture unit 3105 that captures an image of the TV camera 3100. , TV camera 310
0, a camera control unit 3110 that controls a shooting direction, an input unit 3115 that receives a shooting start request or a shooting end request from a user, and a video capture unit 3105.
A first image memory 3120 for temporarily storing one frame of the preview video captured in step 1, and a second image memory 3125 for temporarily storing one image taken immediately before the latest image
And an image storage unit 3130 for storing still images captured by the video capture unit 3105, a display unit 3135 for displaying a preview image, and detecting an overlapping area from the images in the first image memory 3120 and the second image memory 3125. An overlap area detection unit 3140 that performs
The characteristic block in the overlap area of the image in the first image memory 3120, that is, the block having the largest variance value when the video is divided into blocks of 16 × 16 pixels, and the block corresponding to the block It comprises a feature point extraction unit 3145 for finding a corresponding block in the image of the two-image memory 3125, a shooting parameter calculation unit 3150 for calculating shooting parameters, and a central control unit 3155 for controlling the flow of the entire processing. FIG. 32 is a flowchart of the fourth embodiment.

First, in step 3200, the processing is started, and the flow advances to step 3205. Step 3205
In, the image captured by the TV camera 3100 is displayed as a preview on the display unit 3135 via the video capture unit 3105. Next, in step 2810, it is determined whether or not the user has pressed the shooting start button in the input unit 3115,
If the shooting start button has been pressed, the process proceeds to step 3215, and if the shooting start button has not been pressed, the process proceeds to step 315.
Return to 205. By pressing a shooting start button, continuous shooting of an image source for a panoramic image is started.

In step 3215, the video capture unit 3
At 105, one frame of the video of the TV camera 3100 is captured. Next, in step 3220, the video capture unit
1 frame captured in 3105 is stored in the first image memory 31
20 and store it temporarily. Next, in step 3225, the image data temporarily stored in the first image memory 3120 is copied and stored in the image storage unit 3130. Next, in step 3230, the video capture unit 3105 sets the TV
One frame of the image of the camera 3100 is captured. Then, in step 3235, one frame of the video captured in step 3230 is transferred to second image memory 3125. Through the above processing, the image previously captured for image storage is temporarily stored in the first image memory 3120, and the current TV camera 31 is stored in the second image memory.
00 is temporarily stored.

Next, at step 3240, the overlap area detecting section 3140 detects an overlap area between the image temporarily stored in the first image memory 3120 and the image temporarily stored in the second image memory 3125. The method of detecting the overlap area is as described in the overlap detection processing of the first embodiment.

Next, at step 3245, the feature point extracting unit 3
At 145, the feature rectangle of the image temporarily stored in the first image memory 3120 and the corresponding rectangle of the image temporarily stored in the second image memory 3125 are extracted. The feature point extraction processing is as described in the feature point extraction processing of the first embodiment.

Next, at step 3250, the photographing parameter calculation unit 3150 calculates the pan angle, the tilt angle, the rotation angle around the photographing direction axis, and the correction amount of the zoom ratio of the TV camera 3100. Then, it is determined whether or not the size of the overlap area is 30% or less of the whole. The above-described shooting parameter calculation processing is as described in the shooting parameter calculation processing of the second embodiment.

Next, at step 3255, the camera control unit 3
In step 110, the TV camera 3100 is controlled based on the pan angle, the tilt angle, the rotation angle around the shooting direction axis, and the correction amount of the zoom ratio calculated in step 3225.

Next, at step 3260, the input unit 3115
In step 327, it is determined whether the user has pressed the shooting end button.
0, the central control unit 3155 performs an end process, and if the imaging end button is not pressed, the process proceeds to step 3
Proceed to 265.

At the step 3265, the central control unit 3155
In step 3250, the photographing parameter calculation unit 3
The size of the overlap area calculated in 150 is image 1
It is determined whether the size is 35% or less of the size of the sheet.
%, The process proceeds to step 3275, where 35
If the overlapping portion is larger than%, the process returns to step 3230.

In step 3275, the second image memory 31
The image data temporarily stored in the storage unit 25 is stored in the image storage unit 31.
30 and copy and save. In step 3280, the image data temporarily stored in the second image memory 3125 is transferred to the first image memory
Save temporarily at 120. Then, the process returns to step 3230.

[0187]

According to the present invention, unnatural distortion due to a difference in a photographing direction does not occur in a generated panoramic image.
The quality of the panoramic image becomes uniform over the entire image, and the gap between the images becomes inconspicuous.

[Brief description of the drawings]

FIG. 1 is a system configuration diagram of a first embodiment.

FIG. 2 is a flowchart of the system according to the first embodiment.

FIG. 3 is a flowchart of the system according to the first embodiment;

FIG. 4 is a flowchart of the system according to the first embodiment.

FIG. 5 is a flowchart of an overlap area detection process.

FIG. 6 is an explanatory diagram of an overlap area detection process.

FIG. 7 is an explanatory diagram of an overlap area detection process.

FIG. 8 is a flowchart of a feature point extraction process.

FIG. 9 is a flowchart of a feature point extraction process.

FIG. 10 is an explanatory diagram of a feature point extraction process.

FIG. 11 is a flowchart of a shooting parameter calculation process.

FIG. 12 is a flowchart of a shooting parameter calculation process.

FIG. 13 is an explanatory diagram of a shooting parameter calculation process.

FIG. 14 is a layout diagram of a TV camera according to the second embodiment.

FIG. 15 is a system configuration diagram of a second embodiment.

FIG. 16 is a flowchart of the second embodiment.

FIG. 17 is a flowchart of the second embodiment.

FIG. 18 is a flowchart of the second embodiment.

FIG. 19 is a flowchart of the second embodiment.

FIG. 20 is a flowchart of a photographing parameter calculation process.

FIG. 21 is a flowchart of a calculation process of photographing parameters.

FIG. 22 is a flowchart of a calculation process of shooting parameters.

FIG. 23 is an explanatory diagram of a photographing parameter calculation process.

FIG. 24 is an explanatory diagram of a photographing parameter calculation process.

FIG. 25 is a digital camera according to a third embodiment.

FIG. 26 is an explanatory diagram of shooting guidance.

FIG. 27 is a system configuration diagram of a third embodiment.

FIG. 28 is a flowchart of the third embodiment.

FIG. 29 is a flowchart of the third embodiment.

FIG. 30 shows an aerial photographing camera system according to a fourth embodiment.

FIG. 31 is a system configuration diagram of the camera system for aerial photography according to the present embodiment.

FIG. 32 is a flowchart of the fourth embodiment.

[Explanation of symbols]

100: first camera unit, 103, 3100: TV camera, 106: camera control unit, 109: video capture unit, 112: camera communication unit, 115: camera central control unit, 130: second camera unit, 160: panoramic image Display unit, 163 input unit, 166, 2730 feature point extraction unit, 169, 2735 photography parameter calculation unit, 172
.., A photographing parameter table, 175, a panoramic image generation unit, 178, 2740, a display unit, 181, a panorama image display communication unit, 184, a panorama image display central control unit, 187
... overlap area detection unit, 190 ... network,
1500: main camera unit, 1520: left sub camera unit, 1540: right sub camera unit, 1560: panoramic image display unit, 1590: network, 2500: digital camera, 2510: liquid crystal display, 2705: photographing unit, 2710: first Image memory 2715 Second image memory 2720 Image storage unit 2725 Overlap area detection unit 2745 Shutter button 275
0: Central control unit, 3000: Airplane.

Claims (17)

[Claims] 1. A panoramic image generation camera system for synthesizing two or more images, comprising: a photographing camera; and photographing parameter calculating means for calculating photographing parameters of the camera based on a photographed image photographed by the photographing camera. Camera system for generating panoramic images. 2. The photographing parameter includes a pan angle of a camera,
2. The panoramic image generation camera system according to claim 1, wherein the camera system is at least one of a tilt angle and a rotation angle around a shooting direction axis. 3. The panoramic image generation camera system according to claim 1, wherein the photographing parameter is at least one of a vertical direction, a front-back direction, and a left-right direction of the camera. 4. The camera system for generating a panoramic image according to claim 1, wherein said photographing parameter is a zoom ratio of a camera. 5. The panoramic image generation camera system according to claim 1, wherein said photographing parameter calculating means calculates a photographing parameter based on a feature point or a region image of each image photographed by a camera for panoramic composition. 6. The photographing parameter calculating means, based on a distance between feature points and a size of an overlap area image.
The camera system for generating a panoramic image according to claim 5, wherein at least one of a zoom ratio of the camera, a front-rear direction of the camera, a left-right direction, or a vertical direction is calculated. 7. A camera pan angle or a camera pan angle based on a perspective (distortion degree) of an area constituted by straight lines connecting characteristic points or a perspective (distortion degree) of an overlap area image. 6. The panoramic image generation camera system according to claim 5, wherein at least one of the tilt angles is calculated. 8. The panoramic image generating apparatus according to claim 5, wherein said photographing parameter calculating means calculates a rotation angle of the camera around a photographing direction axis based on a slope of a straight line connecting the characteristic points or a slope of the overlap area image. Camera system. 9. The panoramic image according to claim 5, wherein said photographing parameter calculating means calculates a position in the front-rear direction, a position in the left-right direction, or a position in the up-down direction of the camera from the overlapping position of the overlap area between the images. Camera system for generation. 10. The panoramic image generating camera system according to claim 5, wherein said photographing parameter calculating means calculates a timing of releasing a photographing shutter from a size of an overlap area between images. 11. The panoramic image generation camera system according to claim 1, wherein said photographing parameter calculating means compares the images of a plurality of frames photographed by one camera and calculates photographing parameters. 12. The panoramic image generation camera system according to claim 1, wherein said photographing parameter calculation means compares images of two or more cameras and calculates photographing parameters. 13. The panoramic image generation camera system according to claim 12, further comprising camera control means for controlling the camera based on the photographing parameters calculated by the photographing parameter calculating section. 14. The panoramic video camera system according to claim 1, further comprising a designating means for designating a photographing direction of the camera and a zoom of an image by a user. 15. The camera system for generating a panoramic image according to claim 1, further comprising a photographing parameter teaching means for teaching a photographing parameter calculated by the photographing parameter calculating means to a user. 16. A camera system for capturing an original image for generating a panoramic image that combines two or more images to make it look like a single image. A camera system for generating panoramic images having holding means. 17. A panoramic image generation camera system according to claim 16, further comprising a photographing parameter designating means for allowing a user to designate a desired parameter from the photographing parameter holding means.