JPH1165004A - Panoramic image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a panoramaic image pickup device capable of forming a panoramic image with a wide dynamic range. SOLUTION: An image of a certain panoramic range is picked up by changing the image pickup direction of a video camera unit 3 rotating a universal head 30 by the control of a control unit 4 and a panoramaic image is formed by linking these images together by a recording reproducing device 2. The exposure are then changed and a panoramaic image of an identical panorama range is prepared. One sheet of a composition panorama image including a wide dynamic range is prepared by changing the light intensity of the picture elements of the respective panorama image to a value corresponding to the exposure and composing a plurality of panoramaic images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a panoramic image pickup system for picking up a panoramic image requiring a wide dynamic range (wavelength band of light).

[0002]

FIG. 9 is a block diagram showing a conventional panoramic image pickup system. As shown in FIG. 9, this panoramic imaging system includes a CC mounted on a rotating head 101.
The configuration includes a home video camera 100 with a built-in D element and a recording / reproducing device 110. This allows
While rotating the rotating camera platform 101, each part of the wide scene is captured by the video camera 100, and the video of each part is temporarily stored in the recording unit 111 of the recording and reproducing apparatus 110. Then, the saved images of each unit are connected as one panoramic image by the image connecting unit 112, and this image can be displayed on the display unit 113 at a time.

[0003]

However, the above-mentioned conventional system for capturing a panoramic image has the following problems. The CCD element used in the home video camera 100 has a narrow dynamic range of light that can be converted into an electric signal, and thus cannot display the entire dynamic range required for imaging. That is, when the video camera 100 is imaged in the backlight direction with a large amount of light, the amount of charge accumulated in the CCD element is saturated. For this reason, a bright portion (a portion where the light wavelength is short) of the image is displayed as pure white, and a so-called white saturation phenomenon occurs. Conversely, when the video camera 100 is imaged in the forward light direction where the amount of light is insufficient, the amount of charge accumulated in the CCD element is insufficient, and dark portions of the captured image are displayed as black. That is, a so-called black sun phenomenon occurs.

[0004] The present invention has been made to solve the above-described problem, and has as its object to provide a panoramic imaging system capable of creating a panoramic image having a wide dynamic range.

[0005]

In order to solve the above-mentioned problems, a panoramic image pickup system according to the present invention comprises an image pickup means for picking up an object and outputting the image, and a rotation for changing the image pickup direction of the image pickup means. Means for controlling an exposure condition and an image pickup operation of the image pickup means, and first control means for controlling the rotation means so that the image pickup means changes the image pickup direction at a predetermined angle, and a predetermined number of images connected to each other. An image processing unit having a first function of generating an image, and a second function of generating a predetermined combined image by combining a plurality of images having different exposure conditions according to the exposure condition of each image; Second control means for controlling the first and second functions of the processing means is provided. With this configuration, the first
In the control means, the exposure condition and the imaging operation of the imaging means are controlled, and the rotating means is controlled so that the imaging means changes the imaging direction at a predetermined angle. A first function of the image processing means connects a predetermined number of images to generate a panoramic image, and a second function combines a plurality of images having different exposure conditions according to the exposure conditions of each image. Thus, a predetermined composite image is generated.

[0006]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of a panoramic imaging system according to an embodiment of the present invention. As shown in FIG. 1, this panoramic imaging system includes:
An imaging device 1 and a recording / reproducing device 2 are provided.

The image pickup apparatus 1 is an apparatus for picking up an image of a subject. The image pickup apparatus 1 includes a video camera unit 3 as an image pickup unit mounted on a rotary head 30 as a rotation unit, and a video camera unit 3 and the rotary head 30. It comprises a control section 4 as first control means for controlling.

The video camera unit 3 includes a lens block 31
And a CCD element 32, which is a solid-state image sensor, and a signal separation and automatic gain adjustment circuit (hereinafter, referred to as "SH / AGC") 33
And analog / digital converter (hereinafter “A / D converter”)
), A signal processing circuit 35, and a monitor 36. As a result, the imaging light from the subject passes through the lens group or the aperture of the lens block 31 and the CCD element 32.
To receive light. Then, photoelectric conversion is performed in the CCD element 32, and the video signal is input to the SH / AGC 33. Then, in the SH / AGC 33, the video signal is sampled and held, and the gain is controlled so as to have a predetermined gain, and is output to the A / D converter 34. Then, the video signal obtained by digitizing the video signal by the A / D converter 34 is input to a signal processing circuit 35, which converts the video signal into a luminance signal, a color difference signal, and a video signal. The image is output to the recording / reproducing device 2 as an image.

On the other hand, the control unit 4 comprises a camera controller 40 for controlling the video camera unit 3 and a rotating head 30.
And a mode controller 42 for controlling these controllers 40 and 41
And Thereby, the camera controller 40
Outputs predetermined control signals to the lens block 31, the SH / AGC 33, the A / D converter 34, and the signal processing circuit 35 to control the lens block 31 (focus and zoom) and exposure control (aperture, Gain and electronic shutter speed), white balance control, and image quality control. For example, in response to a zoom variable lens drive request sent from the mode controller 42,
The camera controller 40 operates the stepping motor 31a of the lens block 31 to move the variable power lens 31b right and left to adjust the angle of view to be imaged and to change the current position of the variable power lens 31b to the mode controller 42. Is always sent to Also,
The pan / tilt controller 41 controls the rotary camera platform 30 having two degrees of freedom in the rotation direction. That is, the pan / tilt controller 41 drives the pan stepping motor 30a and the tilt stepping motor 30b in response to the pan and tilt drive requests sent from the mode controller 42, and sets the rotary head 30 in the mode. By rotating the video camera unit 3 in the direction required by the controller 42 to change the direction of the video camera unit 3, the position of the rotary camera platform 30 is constantly transmitted to the mode controller 42. As described above, in the control unit 4,
In response to a drive request from the mode controller 42, the camera controller 40 and the pan / tilt controller 41 control the wide angle and the imaging direction of the video camera unit 3, and indicate the position of the zoom lens 31b indicating the wide angle state and the imaging direction state. The rotation position of the rotary head 30 is constantly transmitted to the mode controller 42. The mode controller 42, which is the main control part of the control unit 4,
And a communication port 45 such as RS232C.
Each data is transmitted and received to and from the recording / reproducing apparatus 2 via the communication port 45.

The recording / reproducing apparatus 2 is a personal computer, and includes a control circuit 20 as a second control means, a video capture board 21, a hard disk 22,
An image processing circuit 23 as an image processing means is provided, and various controls are performed on the imaging apparatus 1 by operating the keyboard 24. A monitor 5 is connected to the recording / reproducing apparatus 2 so that an image currently being captured is displayed in a current image window 50 and a panoramic image is displayed in a panoramic image window 51. It has become.

The recording / reproducing apparatus 2 functions according to a predetermined algorithm. FIG. 2 is a flowchart illustrating a control algorithm of the recording / reproducing apparatus 2. Hereinafter, the algorithm of the recording / reproducing apparatus 2 will be described with reference to FIG. First, when the algorithm is started by turning on the power or the like, initialization is performed by the keyboard 24,
Opening of the communication port 45 and display setting of the current image window 50 and the panoramic image window 51 on the monitor 5 are performed (step S1 in FIG. 2). Then, the control circuit 20 performs periodic transmission and reception with the mode controller 42 of the control unit 4 via the communication port 45, and analyzes and stores the data when there is data received from the keyboard 24. (Step S2 in FIG. 2).
That is, in the keyboard 24, the exposure level L (1
To n), an imaging range W and a zoom magnification Z are set, and when these data are received by the control circuit 20, the panorama data D1 to Dn corresponding to each exposure level L are set.
Is generated. FIG. 3 is a plan view showing an imaging range and a rotation position of the video camera unit 3, and FIG. 4 is a diagram showing a configuration of panorama data. For example, as shown in FIG. 3, the exposure level L is set to “1” and the zoom magnification Z is set to p times,
After rotating the video camera unit 3 by a predetermined angle to capture an image of the panoramic range of the angle θ, the exposure level L is set to “2”,
When the setting is changed to “3”,..., “N” and the panorama range is imaged n times, the exposure level L is set to “1 to n”, the imaging range W is set to “θ”, and the zoom magnification is set. Set Z to “p”. Then, when this data is received by the control circuit 20, panorama data D1 to Dn corresponding to the exposure level L are generated as shown in FIG. At this time, the position d of the video camera unit 3 is calculated based on the imaging range W and the zoom magnification Z, and is added as “d1, d2, to dm”.

The control circuit 20 reads the panorama data D1 from the hard disk 22 among the panorama data D1 to Dn generated as described above and stored in the hard disk 22 (step S3 in FIG. 2), and sets the exposure level L to "1". "
And zoom control to set the zoom magnification Z to "p" (steps S4 and S5 in FIG. 2). That is, such control is performed by the control unit 4 via the communication port 45.
To the mode controller 42. As a result, the mode controller 42 issues a drive command to the camera controller 40, and the camera controller 40
1b is moved to a desired position, and the gain of the video signal from the aperture lens block 31c and the CCD element 32 and the electronic shutter are controlled to control the zoom magnification Z of the video camera unit 3.
Is set to “p” and the exposure level L is set to “1”.
These pieces of information are transmitted from the camera controller 40 via the mode controller 42 to the control circuit 20 of the recording / reproducing apparatus 2.
Is transmitted to the control circuit 20, the control circuit 20 monitors the received information, and controls the position of the zoom lens 31b and the lens block 31c.
Are checked to see if they have the desired values. Thereafter, the pan / tilt control for setting the position d of the video camera unit 3 to "d1" is performed (step S6 in FIG. 2). That is, the control is transmitted to the mode controller 42 of the control unit 4 via the communication port 45. As a result, the mode controller 42 issues a drive command to the pan / tilt controller 41 to drive the rotary head 30 and control the position d of the video camera unit 3 to be "d1". This information also
From the pan / tilt controller 41 to the mode controller 4
2 to the control circuit 20, and the control circuit 20
By monitoring the received information, it is confirmed whether or not the position d of the video camera unit 3 has become "d1". The control circuit 20
After confirming that the position d of the video camera unit 3 has become "d1", an imaging command is transmitted to the mode controller 42 (step S7 in FIG. 2). When the video signal captured by the video camera unit 3 at the position d1 is received by the video capture board 21, the video signal is displayed on the current image window 50 of the monitor 5 and stored on the hard disk 22 in a format such as bitmap or JPEG ( Step S in FIG.
8).

Next, the position d of the panorama data D1 is checked, and it is determined whether or not the current position d ("d1") is the final position dm. If the current position d is not "dm", the video camera unit 3 is moved to the next position. A control signal for moving to the position d ("d2") is transmitted to the mode controller 42 of the control unit 4 (NO in step S9, S10, S6 in FIG. 2). As a result, the pan / tilt controller 41 rotates the rotary head 30 to control the position d of the video camera unit 3 to be "d2". Thereafter, the same processing is repeated until the position d matches “dm” (N in step S9 in FIG. 2).
O, S10, S6 to S9), when the position d matches the final position dm, that is, when it is confirmed that the imaging of the entire imaging range W has been completed, it is determined whether or not a panorama creation request has been issued from the keyboard 24 (S10). (YES in step S9 in FIG. 2, S11). When a panorama creation request is issued, the control circuit 20 determines the positions d1 to dm stored in the hard disk 22.
Is sent to the image processing circuit 23. Then, the image processing circuit 23 performs the first function and connects the images at the positions d1 to dm side by side to form a panoramic image window 51 of the monitor 5.
Are stored in the hard disk 22 via the control circuit 20 (YES in step S11 in FIG. 2, S1).
2). That is, based on the panorama data D1, the panorama image P1 which is a linked image of the images captured at the positions d = d1 to dm under the condition of the exposure level L = 1 is displayed and stored in the panorama image window 51 and the hard disk 22. .

The control circuit 20 controls the hard disk 2
2 and confirms whether the panorama data D1 for processing is the final panorama data Dn (step S13 in FIG. 2), and determines the final panorama data Dn.
If not, the panorama data D2 is read from the hard disk 22 and sent to the image processing circuit 23, and the image processing circuit 23 repeats the same processing as in the case of the panorama data D1 (NO, S14, S3 to S3 to S13 in FIG. 2). S
12). Then, after displaying and storing the panoramic image Pn for the final panoramic data Dn, the control circuit 20
Determines whether there is a panorama image synthesis request from the keyboard 24, and if so, reads the panorama images P1 to Pn in the hard disk 22 and sends them to the image processing circuit 23, and the image processing circuit 23 The second function is performed to perform a panoramic image combining process (YES in step S13, YES in S15, and S16 in FIG. 2). FIG.
FIG. 9 is an explanatory diagram of a panoramic image synthesizing process. As shown in FIG. 5, the image processing circuit 23 transmits light intensity eij (k) (k) of each pixel (i, j) of the transmitted panoramic images P1 to Pn.
= 1 to n) and a weighting function fk (L = k, eij (k)). Here, the weight function fk (L = k, ei
j (k)) is the exposure level L of the k-th panoramic image Pk
And ij as variables. For example, according to the weighting function fk (L = k, eij (k)), the exposure level L at which white saturation occurs and the light intensity eij (k) is equal to or higher than the first reference value, and the exposure level at which black sunken occurs L and light intensity e
When ij (k) is equal to or less than a second reference value (<< first reference value), the respective values become substantially zero. In other cases, a numerical value corresponding to the exposure level L or the light intensity eij (k). A function such that Then, after calculating the sum of the light intensities ij (k) 'obtained by the above integration processing, the light intensity is divided by the sum of the weighting functions fk (L = k, eij (k)) to obtain the pixels of the combined panoramic image P. The intensity Eij of the light of (i, j) is determined, the combined panoramic image P is displayed in the panoramic image window 51, and the hard disk 2
Save to 2. That is, a composite panoramic image P composed of an image having the light intensity Eij determined by the following equation (1)
And display it.

(Equation 1) Further, after creating the composite panoramic image P, if the exposure and white balance of the composite panoramic image P are not appropriate, the image processing circuit 23 performs these adjustments (Step S17 in FIG. 2). That is, regarding the exposure, for example, a function g (Yma, Ym) is calculated for all pixels from the difference between the average luminance Yma of the panoramic image of the same scene captured by automatic exposure and the average luminance Ym of the composite panoramic image P. Correct by multiplying the value. For the white balance, for example, a diagonal matrix M such that Rm = Bm = Gm is obtained from the average values Rm, Gm, and Bm of R, G, and B of pixels having a luminance equal to or more than a certain value from the combined panoramic image P Determine and multiply all pixels by M. Finally, the synthesized panoramic image P after such adjustment is displayed in the panoramic image window 51, and the process ends (step S18 in FIG. 2).

Next, the operation of the panoramic imaging system according to this embodiment will be described. Here, in order to facilitate understanding, an image pickup range W of “180 °” is set at an exposure level L (this is set to level “1”) determined with emphasis on a dark part so that black sun does not occur. A case will be described below where an image is captured again at an exposure level L (this is referred to as a level “2”) determined with emphasis on a bright portion so that white saturation does not occur after the image is captured. However, at the time of imaging, the zoom magnification Z is set to “1 ×” of the widest angle.

FIG. 6 is a plan view showing the imaging range W and the rotational position of the video camera unit 3 in this case. FIG. 7 is a diagram showing the configuration of the panorama data D1, D2 created by the control circuit 20. . In FIG. 1, when the exposure level L, the imaging range W, and the zoom magnification Z are input by operating the keyboard 24, these data are received by the control circuit 20, and the respective exposure levels L (“1”, “2”) are received. )) Is generated. As shown in FIG. 6, assuming that the wide angle is 90 ° when the zoom magnification Z is “1”, the video camera unit 3 needs to be set at 45 ° in order to capture an image capturing range W of 180 °.位置 position d1 and 135
It is necessary to perform positioning with the position d2 of ゜. These positions d1 and d2 are calculated in the control circuit 20 and added to the panorama data D1 and D2, as shown in FIG. 7, and these panorama data D1 and D2 are stored in the hard disk 22.

Then, the panorama data D1 is read, and the exposure level L, the zoom magnification Z, and the position d1 are transmitted from the control circuit 20 to the mode controller 42. Thus, the diaphragm 31c of the lens block 31 and the conversion lens 3
1b and the pan head 30 are controlled by the camera controller 40 and the pan / tilt controller 41, and as shown by the solid line in FIG.
1, the zoom lens is positioned at the position d1 with the zoom magnification Z = 1. The image Q1 of the subject operated by the video camera unit 3 at this position is stored on the hard disk 22 via the video capture board 21 and displayed on the monitor 5. Thereafter, the data at the position d2 is transmitted to the mode controller 42. Similarly, the image Q2 captured by the video camera unit 3 at the position d2 is stored in the hard disk 22 and displayed on the monitor 5. FIG. 8 is a front view showing a panoramic image. FIG. 8A shows a panoramic image when the aperture is determined with emphasis on a dark part, and FIG. 8B shows an emphasis on a bright part. 8C shows a panoramic image when the aperture is determined, and FIG. 8C shows the combined panoramic image of FIGS. 8A and 8B. Hard disk 2
2, images Q1 and Q2 shown in FIG.
In the current image window 50 of the monitor 5, images Q1, Q2
Are displayed over time. Here, when a panorama creation request is made by operating the keyboard 24, the images Q 1 and Q 2 are read by the control circuit 20 and sent to the image processing circuit 23. As a result, the images Q1 and Q2 are connected side by side,
A panoramic image P1 as shown in FIG. 8A is created, stored in the hard disk 22, and displayed in the panoramic image window 51 of the monitor 5.

When the process of creating the panoramic image P1 is completed, the panoramic data D2 is read out from the hard disk 22, and substantially the same process as in the case of the panoramic data D1 is executed. As shown in FIG. Level L = 2
Are stored on the hard disk 22 and displayed in the current image window 50 with time. Then, in response to a panorama creation request, these images Q
1 and Q2 are linked side by side to create a panoramic image P2 as shown in FIG. 8B, which is stored in the hard disk 22 and displayed in the panoramic image window 51.

Here, when a panorama image synthesis request is made by operating the keyboard 24, the panorama images P1, P2 are read from the hard disk 22 and sent to the image processing circuit 23 via the control circuit 20. Then, the calculation of the above equation (1) is performed on the panoramic images P1 and P2. Specifically, in the panoramic image P1, as shown in FIG. 8A, white saturation occurs in the central portion, so that the light intensity of each pixel in that portion is made substantially zero. On the other hand, in the panoramic image P2, as shown in FIG. 8 (b), since black sunken portions occur at the lower left and right sides, each pixel in that portion is set to substantially zero. Then, after such an operation is performed, a combined panoramic image P having a wide dynamic range is combined and stored in the hard disk 22 and displayed in the panoramic image window 51 as shown in FIG. Is done.

The present invention is not limited to the above embodiment, and various modifications and changes can be made within the scope of the invention. For example, in the above embodiment,
After creating the panoramic images P1 to Pn, these panoramic images P1 to Pn are combined to create a combined panoramic image P. However, without creating the panoramic images P1 to Pn, different exposure levels L at each position d are set. May be combined first, and the combined images at the respective positions d may be connected side by side to create a single combined panoramic image P. For example, in FIG. 3, in a state where the video camera unit 3 is positioned at the position d1, an image is taken at an exposure level L = 1 and an exposure level L = 2, and then a combined image M1 of the two images Q1 and Q1 is combined. Thereafter, an image is taken with the video camera unit 3 positioned at the position d2, and a composite image M of the two images Q2 and Q2 is taken.
2 is synthesized. Then, the composite panorama image P is created by connecting these composite images M1 and M2 side by side. In the case where white saturation or black sunken is slight, a plurality of images obtained by electrically changing the gain may be synthesized.

[0021]

As described above in detail, according to the panoramic imaging system of the present invention, a predetermined number of images are connected to generate a panoramic image, and a plurality of images having different exposure conditions are set as the exposure conditions of each image. Since the configuration is such that a composite image is generated by performing composition in accordance with this, there is an excellent effect that a composite panoramic image with a wide dynamic range can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram of a panoramic imaging system according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a control algorithm of the recording / reproducing apparatus.

FIG. 3 is a plan view showing an imaging range and a rotation position of a video camera unit.

FIG. 4 is a configuration diagram of panorama data.

FIG. 5 is an explanatory diagram of a panoramic image synthesizing process.

FIG. 6 is a plan view illustrating an example of an imaging range and a rotation position of a video camera unit.

FIG. 7 is a configuration diagram of panorama data created by a control circuit.

8A and 8B are front views showing a panoramic image. FIG. 8A shows a panoramic image when an aperture is determined with emphasis on a dark part, and FIG. 8B shows an emphasis on a bright part. 8C shows a panoramic image when the aperture is determined, and FIG. 8C shows the combined panoramic image of FIGS. 8A and 8B.

FIG. 9 is a block diagram showing a conventional panoramic imaging system.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Image pick-up device, 2 ... Recording / reproducing device, 3 ... Video camera part, 4 ... Control part, 5 ... Monitor, 20 ...
Control circuit 23 image processing circuit 30 rotating head
31: lens block, 32: CCD element, 40
... camera controller, 41 ... pan-tilt controller, 42 ... mode controller.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Naoyasu Hosunuma 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Ken Tamayama 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo No. Sony Corporation

Claims (5)

[Claims] An imaging unit configured to image a subject and output the image; a rotation unit configured to change an imaging direction of the imaging unit; an exposure condition and an imaging operation of the imaging unit; First control means for controlling the rotation means such that the means changes the imaging direction at a predetermined angle, a first function of generating a panoramic image by connecting a predetermined number of images, and a plurality of images having different exposure conditions Image processing means having a second function of generating a predetermined composite image by synthesizing according to the exposure condition of each image, and a second function of controlling the first and second functions of the image processing means.
A panoramic imaging system comprising: 2. The panoramic imaging system according to claim 1, wherein the first control unit changes the exposure condition after the imaging unit sequentially captures one imaging range while changing an imaging direction. Controlling the imaging unit and the rotation unit so as to repeat the same imaging operation for the one imaging range; and the second control unit corresponds to each imaging direction of the imaging unit in the one imaging range. The first function of the image processing means is controlled so as to create a panoramic image of the one imaging range by concatenating the plurality of images thus obtained, and each of the pixels of the plurality of panoramic images having different exposure conditions. Controlling the second function of the image processing means so as to combine the plurality of panoramic images to create one combined panoramic image after performing weighting corresponding to the exposure condition. Lama imaging system. 3. The panoramic imaging system according to claim 1, wherein the first control unit changes the exposure condition after the imaging unit sequentially captures one imaging range while changing an imaging direction. The second control means controls the imaging means and the rotation means so as to repeat the same imaging operation for the one imaging range. The second control means controls the pixels in a plurality of images in the same imaging direction having different exposure conditions. After weighting corresponding to each exposure condition, the second function of the image processing means is controlled so as to create a composite image of the plurality of images, and the plurality of composite images having different imaging directions are connected. Controlling a first function of the image processing means so as to create one composite panoramic image. 4. The panoramic imaging system according to claim 2, wherein said image processing means calculates an optimal exposure and white balance from a luminance and a color difference of said composite panoramic image, and calculates an exposure and white balance of said composite panoramic image. A panoramic imaging system having a third function of processing so that the balance becomes the calculated exposure and white balance. 5. The panoramic imaging system according to claim 2, wherein said image processing means calculates an optimal exposure and white balance from a luminance and a color difference of said composite panoramic image, and calculates an exposure and white balance of said composite panoramic image. A panoramic imaging system having a third function of processing so that the balance becomes the calculated exposure and white balance.