JPH09322040A - Image generator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the image generator in which panorama images with high image quality are simply obtained at all times. SOLUTION: A detection means 190 detects image pickup conditions used for picking up an image of an object by an image pickup means in a way that the object is picked up as a plurality of images where part of adjacent images is overlapped. A storage means 130 stores a series of images obtained by the image pickup means 110 and stores the image pickup conditions obtained by the detection means 190 in cross reference with each image. An image synthesis means 172 selects adaptively a plurality of synthesis means based on the image pickup condition corresponding to each image to generate a synthesized image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image generating apparatus suitable for use in an electronic camera system or the like having a panoramic image capturing mode, and in particular, a panoramic image is generated by joining a plurality of images which partially overlap each other. The present invention relates to an image generating device that does.

[0002]

2. Description of the Related Art Conventionally, as an image generating apparatus for generating a panoramic image (hereinafter, also referred to as a combined image) by combining a plurality of images obtained by photographing so that a part of the field of view overlaps, for example, There is an image composition processing device disclosed in Japanese Patent Laid-Open No. 5-122606. This image composition processing device
A plurality of images to be joined are photographed so as to partially overlap each other, and the luminance difference or chromaticity difference of the end regions of the images to be joined is obtained. The luminance difference or chromaticity difference is set to "0" or the minimum value. This is a device that synthesizes images by connecting overlapping areas so that they overlap. As a result, the image composition processing device
It is simple because there is no need to precisely move and position the imaging device or the imaging target for joining multiple images, and the connection position is determined using the brightness difference or chromaticity difference between the images, so that the connection is accurate. Images can be combined without loss of sex.

Further, in the image generating apparatus as described above, when a panoramic image having a wide angle of view is generated from a plurality of images obtained by photographing so that part of the field of view overlaps, 2
A process of connecting two images on a plane by performing geometric transformation such as affine transformation so that the same points in the overlapping regions of the two images coincide with each other is performed. More specifically, first, when the camera is panned in the horizontal direction to capture two images, as shown in FIG. 35, the principal point O on the subject side of the camera lens is made to substantially match during the panning operation. And shoot. Here, in FIG. 35, I1 and I2 are the image pickup surfaces before and after the panning operation, h1 is the horizontal visual field of the image taken before the panning operation, and h2 is the horizontal direction of the image taken after the panning operation. The field of view. Therefore, assuming that the angle of the horizontal field of view of the camera lens is “θ”, an image with a field of view of “θ” to the left and right is obtained from the imaging planes I1 and I2. Therefore, when the overlapping angle of the two images is “α”, the horizontal field of view of the images obtained from the two images is (2θ−
α). At this time, when there is a plane with a rectangular frame in front of the camera at substantially equal angles with respect to the two imaging planes I1 and I2, the images of the two imaging planes I1 and I2 are as shown in FIG.
Images a and b as shown in FIG. These two images a,
A muffin transformation process is performed on b, and for example, two images a, a are obtained only by a translation without enlargement and rotation in the image plane.
When b is combined, a combined image ab as shown in FIG. 37 is obtained.

However, as shown in FIG. 37, the composite image ab obtained as described above has double frame lines in the overlapping portion P _ab , resulting in an unnatural image. It was This is because the two imaging planes I1,
Since I2 is not spatially in one plane, translation in the image plane, magnification, and rotation in the image plane result in
This is because the images cannot be combined accurately.

Therefore, in order to eliminate the unnaturalness of the image in the overlapping portion P _ab as described above, Japanese Unexamined Patent Publication No. 5-14751.
Discloses a panoramic image capturing device that synthesizes images by projecting a plurality of images obtained by shooting so that a part of the images are overlapped on a cylindrical surface and performing geometric conversion. According to this panoramic image capturing device, the images a and b of the two image pickup surfaces I1 and I2 are once projected and converted onto a common cylindrical surface, so that a frame line is doubled in the overlapping portion Pab. In addition, a synthetic image without unnaturalness can be obtained.

[0006]

However, a conventional image generating apparatus that joins the images by joining the regions in which the luminance difference or the chromaticity difference of the overlapping portions of the images is "0" or the minimum value is combined with each other. , The area where the luminance difference or the chromaticity difference is “0” or the minimum value is simply joined, so that the image to be joined is a rotated image,
When a magnification ratio is generated between the image to be joined and the image to be joined, the image quality of the composite image is significantly deteriorated.

Further, a conventional image generating apparatus for projecting each image on a cylindrical surface and geometrically transforming each image to synthesize each image is obtained by, for example, panning in both horizontal and vertical directions and photographing. When a plurality of images are combined, as in the case of the composite image ab in FIG. 37 obtained by performing only parallel movement, an unnatural image in which a frame line is doubled in the overlapping portion of the composite images is obtained. It was dead. For example, when a rectangular frame is photographed by framing four times, FIG.
Four images c1 to c4 as shown in are obtained. Therefore, if the images c1 to c4 are combined only by the parallel movement without projecting the image on the cylindrical surface, as shown in FIG. 39, an unnatural combination such that a frame line is doubled in the overlapping portion Pc. When the image becomes the image c, and the image is once projected on the cylindrical surface and is moved in parallel to synthesize the images c1 to c4, FIG.
As shown in 0, an unnatural composite image c ′ in which a frame line is doubled in the overlapping portion Pc ′ is generated. In addition, this means that even when two images obtained by panning and shooting are combined in the horizontal direction, if a slight camera tilt occurs during shooting, a frame line will appear in the overlapping portion of the combined images. This means that an unnatural image will appear that will occur twice.

Therefore, the present invention has been made in order to eliminate the above-mentioned drawbacks, and an object of the present invention is to provide an image generating apparatus which can always easily obtain a high quality panoramic image.

[0009]

An image generating apparatus according to the present invention divides an image of a subject into a plurality of screens so that a part of the screens overlaps each other, and photographs the images. An image generating apparatus for synthesizing images to generate a synthetic image, comprising: detecting means for detecting photographing conditions at the time of photographing; and a plurality of images obtained by the photographing means together with the detecting means corresponding to each image. The image forming apparatus includes a storage unit that stores the detected shooting conditions, an image combining unit that combines a series of images stored in the storage unit to generate a combined image, and a control unit that controls the image combining unit. , The image synthesizing means comprises a plurality of synthesizing means corresponding to the photographing conditions,
The control means selectively switches the plurality of synthesizing means based on shooting conditions corresponding to each image, and the synthesizing means selectively switched by the control means is configured to synthesize a series of a plurality of images. Characterize. Further, in the image generating apparatus according to the present invention, the image synthesizing means includes corresponding point detecting means for detecting corresponding points in overlapping portions of the images, and coordinate transformation for subjecting each image to coordinate transformation processing to generate a synthesized image. Means and parameter generating means for generating a photographing parameter based on the corresponding point detected by the corresponding point detecting means, and the coordinate converting means uses the photographing parameter generated by the parameter generating means for the coordinates. It is characterized in that conversion processing is performed. Further, in the image generating apparatus according to the present invention, the detecting unit detects the focus position information at the time of photographing as the photographing condition, and the image synthesizing unit synthesizes a series of a plurality of images obtained by the close-range photographing. The control means includes a short-distance synthesizing unit and a long-distance synthesizing unit that synthesizes a series of a plurality of images obtained by long-distance shooting, and the control unit includes a series of a plurality of images that are synthesized based on focal position information corresponding to each image Is determined by close-up photography or long-distance photography, and selectively switches between the short-distance synthesis means and the long-distance synthesis means based on the determination result. Is characterized by. Further, the image generating apparatus according to the present invention is characterized in that the image synthesizing means is provided with a converting means for converting a pixel value of an overlapping portion of each image based on a photographing condition corresponding to each image. Further, in the image generating apparatus according to the present invention, the detecting unit detects the exposure information at the time of shooting as the shooting condition, and the converting unit determines the density of the overlapping portion of each image based on the exposure information corresponding to each image. It is characterized by correcting the level. Also,
In the image generating apparatus according to the present invention, the image synthesizing means is provided with a spherical projection converting means for projecting and converting each image onto a spherical surface based on a photographing condition corresponding to each image to generate a spherical projection image, It is characterized in that a plurality of spherical projection images obtained by the projection conversion means are combined. Further, the image generating apparatus according to the present invention is characterized in that the spherical surface projection conversion means performs projection conversion to a spherical surface having a focus position at the time of photographing as a center. Further, the image generating apparatus according to the present invention generates a plane projection composite image by projecting the composite image obtained by combining the plurality of spherical projection images obtained by the spherical projection converting unit onto the plane by the image combining unit to convert the image onto a plane. It is characterized in that a plane projection conversion means is provided. Further, the image generating apparatus according to the present invention adds, to the image combining means, projection plane type information indicating whether the image to be processed is the spherical projection image or the plane projection combined image, to the image. It is characterized in that additional means is provided. Further, the image generating apparatus according to the present invention is configured such that the composite image and the plane projection composition are combined with the image composition means in accordance with the field of view of the composite image obtained by combining the plurality of spherical projection images obtained by the spherical projection conversion means. It is characterized in that an output means for selectively switching and outputting an image is provided. Further, in the image generating apparatus according to the present invention, a plurality of image pickup means is provided in the image pickup means, and an optical system for controlling the direction of each optical axis of the plurality of image pickup means based on the image pickup condition detected by the detection means. And axis control means. Further, an image generating apparatus according to the present invention is an image generating apparatus that synthesizes a plurality of captured images to generate one image, and means for setting a panoramic shooting mode, and means for detecting an angle of the apparatus. A means for holding the angle information together with a plurality of shot images obtained by shooting in the panorama shooting mode, and a means for synthesizing the plurality of shot images,
When synthesizing the plurality of captured images, the synthesizing method is adaptively switched based on the angle information. Further, an image generating apparatus according to the present invention is an image generating apparatus that synthesizes a plurality of captured images to generate one image, and includes means for setting a panoramic shooting mode, and means for detecting the position of the apparatus. A means for holding the position information together with a plurality of shot images obtained by shooting in the panorama shooting mode, and means for synthesizing the plurality of shot images,
When synthesizing the plurality of captured images, the synthesizing method is adaptively switched based on the position information.

[0010]

According to the present invention, the photographing condition for photographing the subject image by the photographing means is detected by the detecting means. The storage means stores a series of a plurality of images obtained by the photographing means, and also stores the photographing condition obtained by the detecting means in association with each image. Then, the control means adaptively switches the plurality of synthesizing means based on the photographing condition corresponding to each image. As a result, an appropriate combining means is selected and an appropriate combining process is performed on a series of a plurality of images to be combined. Further, according to the present invention, the corresponding point detecting means detects a corresponding point in an overlapping portion of a plurality of images to be combined. The parameter generating means generates a photographing parameter used when the synthesizing process is performed by the coordinate converting means, based on the corresponding points detected by the corresponding point detecting means. Then, the coordinate conversion means performs a combining process using the shooting parameters obtained by the parameter generation means. Further, according to the present invention, the detection means detects focus position information at the time of shooting as the shooting condition. Therefore,
The storage unit stores focus position information corresponding to each image. The control means determines, based on focus position information corresponding to a series of a plurality of images to be combined, whether those images are obtained by short-distance photographing or long-distance photographing. To do. Then, according to the determination result, the control means selects the short-distance combining means when the short-distance photographing is performed, and selects the long-distance combining means when the long-distance photographing is performed. Thereby, the series of plural images obtained by the short-distance photographing are combined by the short-distance combining means by an appropriate combining process, and the series of plural images obtained by the long-distance photographing are combined by the long-distance combining means. Are combined by an appropriate combining process. Further, according to the present invention, the conversion means converts the pixel value of the overlapping portion of each image based on the shooting condition corresponding to the series of plural images to be combined. As a result, the image in the vicinity of the connecting portion between the adjacent images is converted. Further, according to the present invention, the detection means detects exposure information at the time of shooting as a shooting condition. Therefore, the focus position information is stored in the storage means in association with each image. The converting means corrects the density level of the overlapping portion of each image based on the exposure information corresponding to the series of plural images to be combined. Further, according to the present invention, the image synthesizing means synthesizes the plurality of spherical projection images obtained by the spherical projection converting means on the basis of shooting conditions corresponding to a series of plural images to be synthesized. Further, according to the present invention, the spherical projection conversion means project-converts each image into a spherical surface centered on the focus position at the time of shooting to generate a plurality of spherical projection images to be combined. Further, according to the present invention, the image synthesizing means synthesizes a plurality of spherical projection images obtained by the spherical projection transforming means, and the plane projection transforming means projects and transforms the synthesized image again on a plane. Generate a plane projection composite image. Further, according to the present invention, the adding means adds projection plane type information indicating that the image is the spherical projection image to the image obtained by the spherical projection converting means, and obtains by the plane projection converting means. Projection plane type information indicating that the image is a plane projection composite image is added to the image. Further, according to the present invention, the image synthesizing means synthesizes a plurality of spherical projection images obtained by the spherical projection converting means, and outputs the synthetic image and the plane according to the visual field of the synthetic image. The projected composite image is selectively switched and output. Further, according to the present invention, the optical axis control means controls the direction of each optical axis of the plurality of image pickup means based on the photographing condition detected by the detection means. Therefore, by the plurality of imaging means,
The image is divided and captured so that some of the images overlap. Further, according to the present invention, the angle of the device is detected, and the information on the angle is held together with each image obtained by photographing in the panoramic photographing mode. And when synthesizing each image,
The synthesizing method is adaptively switched based on the angle information. As a result, the image synthesizing process is performed by the synthesizing method optimal for the angle of the device. Further, according to the present invention, the position of the device is detected, and the information on the position is held together with each image obtained by photographing in the panoramic photographing mode. Then, when synthesizing each image, the synthesizing method is adaptively switched based on the information on the position. As a result, the image synthesizing process is performed in the optimal synthesizing method for the position of the device.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION First, a first embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system 100 as shown in FIG. The electronic camera system 100 is, for example, an electronic still camera in which each pixel data as luminance information of RGB components of a photographed subject is obtained as digital pixel data of a two-dimensional array. In addition, the electronic camera system 1
In 00, the image composition processing unit 172 is composed of a computer, and the image composition processing unit 172 operates according to a preset program. further,
The image composition processing unit 172 has an image memory 130 which is an external storage device, and the image memory 130 is adapted to store the digital pixel data.

That is, the electronic camera 100 is the same as that shown in FIG.
As shown in FIG. 1, the imaging unit 110, the video signal processing unit 107 to which the output of the imaging unit 110 is supplied, and the video signal processing unit 1
The image memory 130, the focus detection unit 142, the exposure detection unit 143, and the video signal processing unit 1 to which the outputs of 07 are respectively supplied.
07, the white balance detecting unit 141 and the signal processing unit 190, the controller 120 to which the output of the signal processing unit 190 is supplied, and the controller 1
The zoom control unit 121, the focus control unit 122, the aperture control unit 123, the shutter control unit 124, and the flash control unit 125 to which the outputs of 20 are respectively supplied, the shooting mode setting unit 160 connected to the signal processing unit 190, and the input / output. An interface (I / F) unit 170, an image synthesis processing unit 172 to which the output of the I / F unit 170 is supplied, and a display unit 173 to which the output of the image synthesis processing unit 172 is supplied are provided, and signal processing is performed. The output of the unit 190 is also supplied to the imaging unit 110 and the image memory 130, the outputs of the focus detection unit 142 and the exposure detection unit 143 are supplied to the signal processing unit 190, and the output of the image memory 130 is the I / F.
It is adapted to be supplied to the section 170. The electronic camera system 100 also includes a flash 109 controlled by the flash control unit 125 and a release button detection unit 150, and the release button detection unit 150.
Is output to the signal processing unit 190. The imaging unit 110 is supplied with the imaging lens 101, the diaphragm 102, the shutter 108, and the image sensor 103, which are sequentially provided from the subject side, the amplifier 104 to which the output of the image sensor 103 is supplied, and the output of the amplifier 104. An automatic gain control (AGC) circuit 105 and an analog / digital (A) to which the output of the AGC circuit 105 is supplied.
/ D) converter 106 and AGC circuit 105
The output of the signal processing unit 190 is supplied to the video signal processing unit 107, and the output of the A / D converter 106 is supplied to the video signal processing unit 107.

The electronic camera system as described above (hereinafter,
100 is an electronic camera).
By operating 60, the normal photographing mode and the panoramic photographing mode can be set.

Therefore, first, the outline of the case where the electronic camera 100 is set to the panoramic image capturing mode and the object at a short distance or the object at a long distance is panoramic imaged will be described.

For example, when the electronic camera 100 is used to photograph a document 10 at a short distance as shown in FIG. 2, first, the electronic camera 100 is installed at a position P11 so that the area R11 of the document 10 is located as shown in FIG. Take a picture, then electronic camera 100
Is set at the position P12 and the region R12 of the document 10 is photographed. At this time, the region R11 and the region R12 are photographed so as to partially overlap each other. Therefore, an image I11 as shown in FIG. 4 is obtained by the image pickup at the position P11, and an image I12 as shown in the same figure is obtained by the image pickup at the position P12. Here, in the electronic camera 100, when panoramic photographing of an object at a short distance (hereinafter referred to as short-distance panoramic photographing) is performed, as a parameter for moving the electronic camera 100, as shown in FIG. When the image I11 and the image I12 are combined, a coordinate conversion process is performed based on the above-mentioned parameters by using the translations Δx and Δy of A, the rotation angle θ around the optical axis, and the change f of the magnification due to the translation along the optical axis. By doing so, a composite image I13 as shown in FIG. 5 can be obtained.

On the other hand, in the case of panoramic photographing of a long-distance landscape 20 as shown in FIG. 6 with the electronic camera 100, unlike the short-distance panoramic photographing, even if the electronic camera 100 is translated vertically and horizontally, the photographing area is almost the same. Unchanged. For this reason,
As shown in FIG. 6, first, in the state where the electronic camera 100 is installed at the position P21, the coordinate system XYZ, the rotation angles around each coordinate axis are set to “Ψ”, “Φ”, and “θ”, and Y
The region R21 of the landscape 20 is rotated by rotating around the axis (pan) or rotating around the X axis (tilt).
To shoot. Also at positions P22 and P23,
Similar to the case of the position P21, by performing the pan or tilt operation, the regions R22 and R23 of the landscape 20 are obtained.
To shoot. At this time, the regions R21 and R22 are partially overlapped with each other, and the regions R22 and R2 are
3 is photographed so as to partially overlap each other. Therefore, in the shooting at the position P21, the image I as shown in FIG.
21 is obtained, and an image I22 as shown in the same figure is obtained by photographing at the position P22, and an image I23 as shown in the same figure is obtained by photographing at the position P23. By the way, in panoramic photographing of a long-distance target (hereinafter, referred to as long-distance panoramic photographing), since the electronic camera 100 is photographed while panning, as shown in FIG. 7, a central region R22 of the landscape 20 is displayed. When the corresponding image I22 is used as a reference, the image I2 corresponding to the regions R21 and R23 at both ends of the landscape 20 is displayed.
Trapezoidal distortions indicated by dotted lines L21 and L23 occur in the subject images 1 and I23. Such trapezoidal distortion is
Since this is not generally taken into consideration in the image compositing process at the time of short-distance panoramic photography, if the three images I21 to I23 are combined in the same manner as the image compositing process at the time of short-distance panoramic photography, a composite image with degraded image quality is obtained. Will be done.
Therefore, in this electronic camera 100, during long-distance panoramic photography, the rotation angles Ψ, Φ, and θ around each coordinate axis are used as parameters when moving the electronic camera 100.
When the three images I21 to I23 are combined, coordinate conversion processing is performed on the basis of the above parameters, so that a trapezoidal distortion-free combined image I24 as shown in FIG. 8 can be obtained.

That is, when the panoramic shooting mode is set, the electronic camera 100 determines whether the shooting state is short-distance panoramic shooting or long-distance panoramic shooting, and performs image synthesis processing according to the result of the determination. ing.

Hereinafter, referring to FIG. 1, the electronic camera 10 will be described.
0 will be specifically described.

First, the subject image is projected by the image pickup lens 101 through the diaphragm 102 onto the light receiving surface of the image pickup element 103. At this time, the zoom position and the focus position of the imaging lens 101 are controlled by the zoom control unit 121 and the focus control unit 122, and the diaphragm amount of the diaphragm 102 is
It is controlled by the aperture control unit 123.

The image pickup element 103 is a CCD (Charge).
d Coupled Device) and the like, which converts the received subject image into an electric signal and supplies it to the amplifier 104. The amplifier 104 amplifies an electric signal (hereinafter, referred to as a video signal) from the image sensor 103 to amplify the AGC circuit 105.
To supply. The AGC circuit 105 is the signal processing unit 1
Based on the control signal from 90, the video signal from the amplifier 104 is amplified or attenuated and supplied to the A / D converter 106. The A / D converter 106 digitizes the video signal from the AGC circuit 105 and supplies it to the video signal processing unit 107 as image data. At this time, the signal processing unit 1
A control signal 90 detects the signal level of the image data supplied to the video signal processing unit 107, and when the detected signal level is lower than a predetermined level, the control signal for increasing the gain given to the video signal by the AGC circuit 105. Is generated and supplied to the AGC circuit 105, and when the detected signal level is higher than a predetermined level, the AGC circuit 105 generates a control signal for lowering the gain given to the video signal.
05. As a result, the video signal output from the AGC circuit 105 becomes a signal having a predetermined level width suitable for the signal processing performed by the video signal processing unit 107.

The video signal processing unit 107 includes an A / D converter 1
The image data from 06 is subjected to predetermined signal processing, stored in the image memory 130, and supplied to the white balance detection unit 141, the focus detection unit 142, and the exposure detection unit 143. The white balance detection unit 141 detects the white balance state of the image data from the video signal processing unit 107, and supplies the detection result to the video signal processing unit 107. The focus detection unit 142 detects the focus of the imaging lens 101 from the image data from the video signal processing unit 107,
The detection result is supplied to the signal processing unit 190. The exposure detection unit 143 detects the exposure amount in the image sensor 103 from the image data from the video signal processing unit 107, and supplies the detection result to the signal processing unit 190.

The video signal processor 107 adjusts the color balance of the image data from the A / D converter 106 based on the detection result from the white balance detector 141. Therefore, the image memory 130 stores the image data whose color balance has been adjusted. The signal processing unit 190 includes a focus detection unit 142.
And a controller 1 for generating a control signal for setting a shooting condition based on each detection result from the exposure detection unit 143.
Supply to 20. In addition, the signal processing unit 190 stores information regarding shooting conditions, which will be described later, in the image memory 130. The controller 120 includes the signal processing unit 190.
A control signal is supplied to each of the zoom control unit 121, the focus control unit 122, the aperture control unit 123, the shutter control unit 124, and the flash control unit 125 based on the control signal from.

Therefore, the zoom control section 121, the focus control section 122, and the aperture control section 123 are respectively based on the control signal from the controller 120, the zoom position of the imaging lens 101, the focus position of the imaging lens 101, and the aperture 102. It is controlled so that the aperture amount of is in an appropriate state.

As described above, the shooting conditions in the electronic camera 100 are set appropriately.

Next, the photographer selects the photographing mode setting section 160.
By operating, the shooting mode is set to the normal shooting mode or the panoramic shooting mode, and shooting is started. In addition, the photographer operates the first and second strokes (not shown) of the release button detector 150,
It is instructed to lock the shooting conditions or execute the shooting.

The photographing mode setting unit 160 detects which photographing mode is set by the photographer's operation and supplies the detection result to the signal processing unit 190. The release button detection unit 150 detects whether or not each stroke is pushed down by the photographer's operation of the first and second strokes, and outputs two first and second detection signals corresponding to each stroke to the signal processing unit 190. Supply to.

The signal processing unit 190 generates a control signal according to the set shooting mode according to the detection result from the shooting mode setting section 160 and supplies it to the controller 120. Further, when the signal processing unit 190 determines from the first detection signal from the release button detection unit 150 that the first stroke is pushed down, it generates a control signal that locks the shooting condition and releases the release button. When it is determined that the second stroke is pushed down by the second detection signal from the detection unit 150, the controller 12 generates a control signal for performing the shutter operation.
Supply 0. The controller 120 controls the zoom controller 12 based on the control signal from the signal processing unit 190.
1, the focus control unit 122, the aperture control unit 123, the shutter control unit 124, and the flash control unit 125, and supplies the control signals to the shutter control unit 124 and the flash control unit 125.

Therefore, the zoom position of the image pickup lens 101, the focus position of the image pickup lens 101, and the diaphragm 10.
The aperture amount of 2 is in a state according to the operation of the photographer. Further, the shutter control unit 124 controls the shutter 108 based on the control signal from the controller 120, so that the shutter 108 is controlled to the shutter speed according to the operation of the photographer, and the flash control unit 125 controls from the controller 120. By controlling the flash 109 based on the signal, the flash 1 can be operated according to the operation of the photographer.
The ON / OFF operation of 09 is controlled.

When shooting is started as described above, the image data output from the video signal processing unit 107 is stored in the image memory 130 together with the shooting conditions prestored by the signal processing unit 190.

That is, as shown in FIG. 9, the image memory 130 stores image data including a header portion H and a data portion D. In the header part H, the image data number N
o and identification information Px corresponding to the shooting mode are written,
Information fc, fl, s, and v relating to shooting conditions is written in advance by the signal processing unit 190. Here, the information fc, fl, s, regarding the shooting conditions to be written in the header section H
v is, for example, focus information fc, focal length fl, diaphragm s, and shutter speed v. On the other hand, in the data part D,
For example, in FIG. 7 obtained when the panoramic shooting mode is set.
The respective data of the series of images I21, I22, and I23 as shown in (4) are written corresponding to the image data numbers No2, 3, and 4. In this case, the identification information Px written in the header portion H is written as the identification information P1 indicating that the images I21, I22, and I23 are a series of panoramic images.

Therefore, in the plurality of image data stored in the image memory 130, the images having the same identification information Px form a set of panoramic images. As a result, in the electronic camera 100, when a plurality of image data stored in the image memory 130 are combined to generate a panoramic image, the identification information Px added to each image data is determined to automatically It is designed to be able to perform image composition processing. Such image composition processing is
The image combining processing unit 172 is performed, for example, by a user operating an image output operation unit (not shown).

That is, when the user operates the image output operation section, the image output operation section supplies a signal corresponding to the operation to the signal processing unit 190. The signal processing unit 190 supplies, for example, a control signal indicating a panoramic image output operation to the image memory 130 and the I / F circuit 170 based on the signal from the image output operation unit. As a result, the plurality of image data stored in the image memory 130 is supplied to the image synthesis processing unit 172 via the I / F circuit 170.

The image composition processing section 172 is, for example, as shown in FIG.
1, the image data from the I / F circuit 170 shown in FIG. 1 is supplied through the input / output (I / O) circuit 172a to the image information separating section 172f and the image information separating section 172.
The controller 172e and the image memory 172g to which the output of f is supplied, the corresponding point detection unit 172b to which the output of the image memory 172g is supplied, and the short-distance photographing coordinate conversion processing unit 1
72i and long-distance shooting coordinate conversion processing unit 172j, and output of the short-distance shooting coordinate conversion processing unit 172i and long-distance shooting coordinate conversion processing unit 172j.
72h, a selector 172k to which the output of the corresponding point detection unit 172b is supplied, and a short-distance shooting parameter extraction unit 172c and a long-distance shooting parameter extraction unit 172d to which the output of the selector 172k is supplied. The coordinate conversion processing unit 172i is also supplied with the output of the short distance shooting parameter extraction unit 172c, and the long distance shooting coordinate conversion processing unit 172j is also supplied with the output of the long distance shooting parameter extraction unit 172d. Further, the controller 172e includes the image memory 172g and the corresponding point detection unit 1
It is connected to 72b. Then, the composite image memory 17
The output of 2h is supplied to the display unit 173 of FIG. 1 and the like via the I / O circuit 172a.

In the image composition processing section 172, first, the image information separating section 172f separates the image data from the I / O circuit 172a, that is, the image data shown in FIG. 9 into a header section and a data section. The information of the data section (hereinafter referred to as image information) is stored in the image memory 172g, and the information of the header section (hereinafter referred to as header information) is supplied to the controller 172e. The controller 172e controls each unit of the image synthesis processing unit 172 based on the header information from the image information separation unit 172f.

For example, the controller 172e, based on the header information from the image information separating unit 172f, stores a series of plural image information obtained by panoramic photography in the image memory 172.
It is read from g and supplied to the corresponding point detection unit 172b. The corresponding point detecting unit 172b detects corresponding points in the overlapping portions of the images in the plurality of pieces of image information from the controller 172e. A correlation method, a template matching method, or the like is used to detect the corresponding points. Then, the corresponding point detection unit 17
2b supplies the detected corresponding point to the selector 172k.

Here, as described above, in order to perform the optimum image compositing process according to the short-distance panoramic photography and the long-distance panoramic photography, the controller 172e causes the series of plural images to be processed to be the short-distance panoramic photography. The header information from the image information separating unit 172f determines whether the image is obtained by photographing or is obtained by long-distance panoramic photographing.

That is, the controller 172e detects the magnitude relationship between the focus information fc and a predetermined threshold value by using the focus information fc included in the header information from the image information separating unit 172f, and the focus information f is detected.
When c is equal to or larger than the predetermined threshold value, it is determined that the long-distance panoramic photographing is performed, and when the focus information fc is smaller than the predetermined threshold value, the short-distance panoramic photographing is determined. Then, the controller 172e supplies the determination result to the selector 172k. In addition, the controller 172e is
Based on the determination result, the series of plural pieces of image information read from the image memory 172g as described above are supplied to the short-distance photographing coordinate conversion processing unit 172i or the long-distance photographing coordinate conversion processing unit 172j. The selector 172k determines the corresponding point from the corresponding point detection unit 172b according to the determination result from the controller 172e and uses the short-distance photographing parameter extraction unit 1
72c or the long-distance photographing parameter extraction unit 172d.

Therefore, if the series of images to be processed are obtained by the short-distance panoramic photography according to the determination result of the controller 172e, the short-distance photography parameter extraction unit 172c causes the corresponding point detection unit 172b to perform the processing. The obtained corresponding points are supplied, and the short-distance photographing coordinate conversion processing unit 1
A series of plural image information to be combined into 72i is supplied.
In this case, the short-distance shooting parameter extraction unit 172c determines the vertical and horizontal translations Δx and Δy as shown in FIG. 3 from the corresponding points from the corresponding point detection unit 172b, the rotation angle θ around the optical axis, and the optical axis. A change in magnification f due to translation along the axis is extracted as a parameter, and the parameter is supplied to the short-range shooting coordinate conversion processing unit 172i. The short-distance shooting coordinate conversion processing unit 172i includes a short-distance shooting parameter extraction unit 172.
controller 172e based on the parameters from c
The coordinate conversion processing is performed on the series of image information supplied by to generate a combined image, and the combined image is written in the combined image memory 172h.

On the other hand, when the series of a plurality of images to be processed are obtained by the long-distance panoramic photographing according to the determination result of the controller 172e, the long-distance photographing parameter extraction unit 172d is operated by the corresponding point detection unit 172b. The obtained corresponding points are supplied, and the long-distance photographing coordinate conversion processing unit 172j.
A series of plural pieces of image information to be combined with is supplied. In this case, the long-distance photographing parameter extraction unit 172d extracts the rotation angles Ψ, Φ, and θ around each coordinate axis as shown in FIG. 6 from the corresponding points from the corresponding point detection unit 172b as parameters, and the parameters are extracted. Is supplied to the long-distance photographing coordinate conversion processing unit 172j. Long-distance shooting coordinate conversion processing unit 17
2j performs a coordinate conversion process on a series of image information supplied by the controller 172e based on a parameter from the long-distance shooting parameter extraction unit 172d to generate a composite image, and stores the composite image in the composite image memory. 17
Write to 2h.

Therefore, the synthetic image obtained by the appropriate image synthesizing process according to the photographing condition is written in the synthetic image memory 172h, and the synthetic image is stored in the I / O unit 1
It is supplied to the display unit 173 of FIG. 1 via 72a and is displayed on the screen by the display unit 173.

As described above, in the electronic camera 100, when the image data obtained by shooting is stored in the image memory 130, the identification information Px corresponding to the shooting mode is stored in association with each image data, and Focus information fc, focal length fl, diaphragm s, shutter speed v, and the like are stored as information relating to shooting conditions at the time of shooting in association with each image data. It is possible to easily determine whether the image was captured in the mode or the panoramic image capturing mode, that is, whether the image capturing condition is the short-distance panoramic photography or the long-distance panoramic photography. Further, by determining whether or not the identification information Px is the same, the image memory 130
Since a series of panoramic images can be easily discriminated from the plurality of image data stored in, the image combination processing can be automatically performed. Further, in the electronic camera 100, when performing the image combining process, an appropriate image combining process is automatically selected according to the shooting situation, so that a plurality of image data obtained by the short-distance panoramic shooting is acquired. The optimum image combining process can be performed on the image data, and the optimum image combining process can be performed on the plurality of image data obtained by the long-distance panoramic shooting. Also, the electronic camera 1
In 00, in a plurality of images to be combined, the shooting parameters are extracted from the corresponding points of the overlapping portions of the images, and the image processing is performed using the extracted shooting parameters. Therefore, the user combines the images. You do not have to take any special action to do so. Therefore, the electronic camera 100 may be configured such that the plurality of images to be image-synthesized may be either a short-distance panoramic image or a long-distance panoramic image.
It is possible to obtain a panoramic image of high quality easily and always without deterioration of image quality.

In the image synthesizing processing unit 172 of FIG. 10, the image memory 172g for the image to be synthesized and the synthetic image memory 172h for the synthetic image are provided, but one image memory is synthesized. It may be shared for the image to be used and the composite image. Also, the I / O unit 172a
The composite image output from the computer may be stored in a recording unit such as a hard disk.

Next, a second embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system including an image synthesis processing section 182 as shown in FIG. This electronic camera system (hereinafter, simply referred to as an electronic camera) has an image composition processing unit 17 of FIG.
The image synthesis processing unit 182 shown in FIG.

Since this electronic camera is the same as the electronic camera 100 described above except for the configuration and operation of the image composition processing unit 182, detailed description of each unit other than the image composition processing unit 182 will be omitted. In addition, FIG.
In the image synthesizing processing unit 182 of No. 1, the same operation as that of the image synthesizing processing unit 172 of FIG. 10 is denoted by the same reference numeral, and detailed description thereof will be omitted.

That is, as shown in FIG. 11, the image synthesizing processing unit 182 includes a spherical mapping conversion processing unit 182a, and the output of the selector 172k is supplied to the spherical mapping conversion processing unit 182a. Has been done. Further, the corresponding point detection unit 172b is provided in the subsequent stage of the selector 172k, each output of the controller 172e and the image memory 172g is supplied to the selector 172k, and the output of the selector 172k is directly supplied to the corresponding point detection unit 172b. At the same time, it is supplied to the corresponding point detection unit 172b via the spherical mapping conversion processing unit 182a.
Further, in the image synthesis processing unit 182, the short-distance shooting parameter extraction unit 172c and the short-distance shooting coordinate conversion processing unit 172 are used as the shooting parameter extraction unit and the coordinate conversion processing unit.
Only i is provided, and the output of the corresponding point detection unit 172b is supplied to the short-distance shooting parameter extraction unit 172c, and the output of the short-distance shooting parameter extraction unit 172c is supplied to the short-distance shooting coordinate conversion processing unit 172i. Has been done. Therefore, the output of the image memory 172g is supplied to the short-distance shooting coordinate conversion processing unit 172i, and the output of the short-distance shooting coordinate conversion processing unit 172i is output to the composite image memory 172.
It is designed to be supplied to h.

The operation of the image composition processing unit 182 will be described below.

First, the image composition processing section 172 shown in FIG.
Similarly, the image information separating unit 172f separates the image data from the I / O circuit 172a into a header portion and a data portion, stores the image information in the image memory 172g, and
The header information is supplied to the controller 172e. The controller 172e uses the focus information fc included in the header information from the image information separation unit 172f to obtain a series of a plurality of images to be processed by short-distance panoramic photography, or a long-distance panoramic photography. It is determined whether the image is obtained by shooting, and the determination result is selected by the selector 172.
supply to k. The controller 172e also supplies the focal length f1 included in the header information from the image information separation unit 172f to the selector 172k.

The selector 172k is the controller 172.
If the series of images to be processed is obtained by short-distance panoramic photography according to the determination result from e, the series of image information is stored in the image memory 172g.
It is read from and is directly supplied to the corresponding point detecting unit 172b.
On the other hand, if the series of multiple images to be processed were obtained by long-distance panoramic photography, the selector 172k
Reads out the series of image information from the image memory 172g and supplies it to the spherical mapping conversion processing unit 182a, and also supplies the focal length f1 from the controller 172e to the spherical mapping conversion processing unit 182a.

Therefore, the spherical mapping conversion processing unit 182a
When combining a plurality of images obtained by long-distance panoramic photography, the plurality of image information and focal length f1
Will be supplied.

The spherical surface mapping conversion processing unit 182a performs spherical surface mapping conversion processing on the plurality of image information from the selector 172k.

This spherical mapping conversion processing assumes a spherical surface 30 in contact with an arbitrary image I31, as shown in FIG.
An image I3 of the principal point O of the taking lens 101 of FIG.
It is a process of generating a spherical image I32 by projecting 1 onto the spherical surface 30.

Therefore, for example, an image I31 obtained by long-distance panoramic photography at an arbitrary position and an image obtained by arbitrary angle panning of a plurality of image information supplied to the spherical mapping conversion processing unit 182a. When I33 is set, the spherical surface mapping conversion processing unit 182a uses the focal length f1 from the selector 172k, as shown in FIG.
To generate a spherical image I32 by projecting the image I31 onto the spherical surface 30 as the focal length f1.
A spherical image I34 is generated by projecting 33 onto the spherical surface 30.

Therefore, when the focal length f1 is the same and there is no rotation around the optical axis, the spherical mapping conversion processing unit 18
Since the spherical image I32 and the spherical image I34 obtained in 2a are continuous on the spherical surface 30, only the vertical and horizontal translations Δx and Δy as shown in FIG. 3 are used as the parameters used in the coordinate conversion process. Thus, the spherical image I32 and the spherical image I34 can be combined. However, in reality, there are errors in the focal length f1 and the rotation θ around the optical axis, so here, when performing the coordinate conversion process, the vertical and horizontal translations Δx and Δy, the focal length f1, and the optical axis Around rotation θ and the above parameters are used. That is, in this electronic camera, even when a plurality of images obtained by the long-distance panoramic photography are combined, the same parameters as those used for coordinate conversion of the plurality of images obtained by the short-distance panoramic photography described above are used. By performing coordinate conversion processing using parameters, a composite image is obtained.

Therefore, the corresponding point detector 172b is
A plurality of pieces of image information from the selector 172k or a plurality of pieces of spherical image information from the spherical surface mapping conversion processing unit 182a are supplied according to the shooting situation. The corresponding point detector 172b
In the supplied plurality of image information, the corresponding points of the overlapping portions of the respective images are detected, and the detected corresponding points and the selector 17
The plural pieces of image information from the 2k or spherical mapping conversion processing unit 182a are supplied to the short-distance shooting parameter extraction unit 172c. The short-distance shooting parameter extraction unit 172c calculates the vertical and horizontal translations Δx and Δy as shown in FIG. 3 and the rotation angle θ around the optical axis from the corresponding points from the corresponding point detection unit 172b.
And the change in magnification f due to translation along the optical axis are extracted as parameters, and the extracted parameters and the plurality of pieces of image information from the corresponding point detection unit 172b are supplied to the short-distance photographing coordinate conversion processing unit 172i. Close-up shooting coordinate conversion processing unit 1
72i is a short-distance shooting parameter extraction unit 1 based on the parameters from the short-distance shooting parameter extraction unit 172c.
A plurality of image information from 72c is subjected to coordinate conversion processing to generate a composite image, and the composite image is stored in the composite image memory 17
Write to 2h.

As described above, in this electronic camera, when the series of a plurality of images to be processed are those obtained by the long-distance panoramic photography, the spherical mapping conversion processing unit 182a.
Since the spherical mapping conversion process is performed in (1), an image in which the trapezoidal distortion component as shown in FIG. 7 is removed can be obtained. Therefore, even when a series of a plurality of images obtained by long-distance panoramic shooting is combined, the same processing as the shooting parameter extraction processing and coordinate conversion processing when combining a plurality of images obtained by short-distance panoramic shooting Thus, a high quality composite image can be obtained. Therefore,
The electronic camera can always obtain a high-quality panoramic image regardless of the shooting situation. Further, since the electronic camera does not need to be provided with a shooting parameter extraction processing unit and a coordinate conversion processing unit for long-distance panoramic shooting and short-distance panoramic shooting, respectively, the configuration of the device can be simplified. This also leads to cost reduction of the device.

Next, a third embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system including an image synthesis processing section 192 as shown in FIG. This electronic camera system (hereinafter, simply referred to as an electronic camera) has an image composition processing unit 17 of FIG.
The image synthesis processing unit 192 of FIG. Further, in this electronic camera, in the image memory 130 shown in FIG. 9, the information regarding the shooting conditions to be written in the header portion H includes the focus information fc, the focal length fl, the aperture s, the shutter speed v, and the exposure level e.
And the gain level (gain level) g of the AGC circuit 105 of FIG. 1 are added.

Since this electronic camera is the same as the electronic camera 100 described above except for the configuration and operation of the image composition processing unit 192, detailed description of each unit other than the image composition processing unit 192 will be omitted. In addition, FIG.
In the image synthesis processing unit 192 of No. 4, the same operation as that of the image synthesis processing unit 172 of FIG. 10 is denoted by the same reference numeral, and the detailed description thereof will be omitted.

That is, the image composition processing unit 192, as shown in FIG. 14, is the image composition processing unit 17 of FIG.
In addition to the configuration requirements of 2, the short-distance shooting parameter extraction unit 1
72c and a long-distance photographing parameter extraction unit 172d, a short-distance photographing coordinate conversion processing unit 172i, and a long-distance photographing coordinate conversion processing unit 172j provided between a density level correction processing unit 192a and a long-distance photographing coordinate conversion processing unit 172j. The seamless processing unit 192b provided at the latter stage. The density level correction processing unit 192a is connected to the controller 172e, and the image memory 172g.
The output of is supplied. Further, the short-distance shooting coordinate conversion processing unit 17 is stored in the composite image memory 172h.
2i and each output of the seamless processing unit 192b are supplied.

The operation of the image composition processing section 192 will be described below.

First, the plurality of images synthesized by the image synthesis processing unit 192 are, for example, an image I40 as shown in FIG.
L and I40R. These images I40L and I4
0R is obtained by long-distance panoramic photography as shown in FIG. Here, in the three regions R21 to R23 of FIG. 6, an image obtained by photographing the left region R21 is an image I as shown in FIG.
The image obtained by photographing the central region R22 with 40L is referred to as an image I40R as shown in FIG. Also, the image I40L and the image I4 adjacent to the image I40L
The region overlapping with 0R is set as region R41, and image I40R
And an area where an image adjacent to the image I40R overlaps is referred to as an area R42.

Therefore, the image information separation section 172f is operated by the I / O
Image data from the O circuit 172a, that is, image I40
The data of L and I40R is separated into a header part and a data part, and information of the data part (image information) is stored in the image memory 172g.
And the header information (header information) is supplied to the controller 172e.

The controller 172e reads out the exposure level eL of the image I40L and the exposure level eR of the image I40R contained in the header information from the image information separating unit 172f, and compares the level difference between the exposure levels eL and eR with a predetermined value. I do. Then, the controller 172e determines from the comparison result that the density level correction processing unit 192 when the level difference between the exposure levels eL and eR is larger than a predetermined value.
A operation command is issued to a and at the same time, the image information separation unit 1
The header information from 72f is also included in the density level correction processing unit 192.
a. On the other hand, when the level difference between the exposure levels eL and eR is less than or equal to the predetermined value, the controller 172e issues a non-operation instruction to the density level correction processing unit 192a.

At this time, the density level correction processing unit 192a
Is supplied with the parameters obtained by the long-distance photographing parameter extraction unit 172d, and the controller 172e
Image I40 read from the image memory 172g by
L and I40R image information is provided.

The density level correction processing unit 192a receives the operation command from the controller 172e, and the images I40L and I40 supplied by the controller 172e.
In the image information of R, as shown in FIG.
The difference Δm between the average density levels in the overlapping region R41 of 0L and I40R is obtained. Then, the density level correction processing unit 192a in the overlapping region R41 based on the aperture s, the shutter speed v, the gain level g of the AGC circuit 105, and the like included in the header information from the controller 172e supplied at the same time as the operation command. Make sure that the density levels of the images I40L and I40R are the same level.
The difference Δm is used to correct the density level of the image I40R. As a result, an image I40R ′ matched with the average density level in the overlapping region R41 of the image I40L is generated from the image I40R. Then, the density level correction processing unit 192a, the image I40L, the density level-corrected image I40R ', and the long-distance photographing parameter extraction unit 1
The parameters from 72d are supplied to the long-distance photographing coordinate conversion processing unit 172j.

The long-distance photographing coordinate conversion processing unit 172j, based on the parameters from the density level correction processing unit 192a, outputs the image I40L from the density level correction processing unit 192a.
And I40R 'are subjected to coordinate conversion processing to obtain an image I40L.
And a composite image of the image I40R 'are generated, and the composite image is supplied to the seamless processing unit 192b.

The seamless processing section 192b is the same as that shown in FIG.
XY of the composite image in the overlapping area R41 as shown in FIG.
Image I40L and image I40 corresponding to coordinates (i, j)
The pixel values of R ′ are “SL” and “SR” respectively, the width of the overlapping region R41 is “W”, and the pixel value S of the composite image is S.
(I, j) is obtained by a weighted addition arithmetic expression S (i, j) = SL (1.0−X / W) + SRX / W. Then, the seamless processing unit 192b replaces each pixel in the overlapping region R41 with the pixel value S (i, j) obtained by the above arithmetic expression,
The resultant composite image is stored in the composite image memory 172h.
Write in.

Even if the plurality of images synthesized by the image synthesis processing unit 192 are obtained by short-distance panoramic photography, the density-level correction processing unit is used in the same manner as in the case of long-distance panoramic photography described above. After the density level correction processing is performed in 192a, the short-distance shooting parameter extraction unit 172
Coordinate conversion processing is performed by the short-distance photographing coordinate conversion processing unit 172i based on the parameter obtained by c, and a combined image is generated and written in the combined image memory 172h.

Then, the composite image written in the composite image memory 172h is supplied to the display unit 173 of FIG. 1 through the I / O unit 172a and is displayed on the screen by the display unit 173.

As described above, since this electronic camera is constructed so that the density levels of the overlapping areas of the images to be joined are substantially the same, the joined portion can be made inconspicuous. Therefore, this electronic camera can obtain a higher quality panoramic image.

In the electronic camera described above, the photographing conditions used in the density level correction processing unit 192a are the exposure level e
Then, the gain level g of the AGC circuit 105 in FIG. 1 is set, but the present invention is not limited to this. In the electronic camera described above, seamless processing is performed only when combining a plurality of images obtained by long-distance panoramic shooting.
Seamless processing may be performed even when a plurality of images obtained by short-distance panoramic photography are combined. However, when the image to be combined is a character image such as a document obtained by short-distance panoramic shooting, the character edges do not match due to a minute error in the shooting parameters, so the result of combining
It may be a double image. Therefore, in this case, a seamless processing unit corresponding to the seamless processing unit 192b is provided in the subsequent stage of the short-distance shooting coordinate conversion processing unit 172i, and the image to be combined when the short-distance panoramic shooting is determined is the original image. A means (original image discrimination unit) for discriminating whether or not there is provided. The seamless processing unit is set to the inactive state only when it is determined that the document image is a document image based on the determination result of the document image determining unit.

Next, a fourth embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system 200 as shown in FIG. As shown in FIG. 17, the electronic camera system (hereinafter, referred to as an electronic camera) 200 includes an image pickup unit 210 in addition to the configuration requirements of the electronic camera 100 shown in FIG. It is configured by a compound-eye image pickup system including two image pickup units of the unit 210. The image pickup unit 210 has the same configuration as the image pickup unit 110, and includes an image pickup lens 201 and a diaphragm 20 that are sequentially provided from the subject side.
2, shutter 208 and image sensor 203, and image sensor 2
Amplifier 204 to which the output of 03 is supplied, and amplifier 204
AGC circuit 205 to which the output of
05 is supplied to the A / D converter 206, and the output of the A / D converter 206 is the video signal processing unit 20.
7 is supplied. And the imaging unit 2
10, the zoom, focus, aperture, and shutter controls are performed by the zoom control unit 12 as in the image pickup unit 110.
1, the focus control unit 122, the aperture control unit 123, and the shutter control unit 124. In addition, the electronic camera 200 has the optical axis L of the imaging unit 110.
And a convergence angle control unit 220 that controls the direction of the optical axis R of the imaging unit 210, and the convergence angle control unit 220 is controlled by the controller 120. Further, in the electronic camera 200, the controller 120 and the signal processing unit 190 are adapted to perform processing corresponding to the compound eye imaging system.

In the electronic camera 200 shown in FIG. 17, parts that operate similarly to those of the electronic camera 100 shown in FIG. 1 are designated by the same reference numerals, and detailed description thereof will be omitted.

The operation of the electronic camera 200 will be described below.

First, when the focus detection section 142 detects the focus, the signal processing unit 190 causes the focus detection section 14 to move.
Based on the detection result of 2, the control signal for focus control is supplied to the controller 120, and it is determined whether the subject being photographed is a short-distance subject or a long-distance subject. When the signal processing unit 190 determines that the object is a short-distance subject, the signal processing unit 190 performs control such that the optical axis L of the image capturing unit 110 and the optical axis R of the image capturing unit 210 are parallel to each other, as shown in FIG. The signal is supplied to the convergence angle control unit 220 via the controller 120. In addition, when the signal processing unit 190 determines that the object is a long-distance object, the optical axis L of the image capturing unit 110 and the optical axis R of the image capturing unit 210 are outward as shown in FIG. The control signal is supplied to the convergence angle control unit 220 via the controller 120.

Therefore, the convergence angle control unit 220 is controlled by the control signal from the controller 120 so that the optical axis L of the image pickup unit 110 and the optical axis R of the image pickup unit 210 are parallel to each other during the short-distance panoramic photographing. During long-distance panoramic photography, the image capturing unit 110 and the image capturing unit 210 are controlled so that the optical axis L of the image capturing unit 110 and the optical axis R of the image capturing unit 210 face outward.

Here, as shown in FIGS. 18 (a) and 18 (b), the situation in which the direction of the optical axis L of the image pickup section 110 and the optical axis R of the image pickup section 210 is controlled to perform image pickup is as follows. Figure 3 above
The shooting situation is the same as that of FIG. 6 and the shooting situation of FIG. Therefore, a composite image can be obtained from the plurality of images obtained by shooting as shown in FIG. 18A by the above-described combining processing at the time of short-distance panoramic shooting. ), A composite image can be obtained from the plurality of images obtained by photographing as described above by the synthesizing process at the time of the long-distance panoramic photographing described above.

Therefore, the electronic camera 100 shown in FIG.
Similarly, in the image synthesizing processing unit 172, it is determined whether the plurality of images to be synthesized are obtained by the short-distance panoramic photography or the long-distance panoramic photography, and the determination result is By selecting an appropriate synthesizing process accordingly, a high-quality synthesized llama image can be obtained.

As described above, in this electronic camera 200, the directions of the optical axes of the image pickup section 110 and the image pickup section 210 are automatically controlled according to the subject distance, so that the user Does not need to perform an operation for taking a picture so that a part of the image overlaps when taking a panorama picture. Therefore, the electronic camera 200 can improve operability and can easily obtain a high-quality composite image.

In FIG. 17, the image composition processing unit 172 may be the image composition processing unit 182 of FIG. 11 or the image composition processing unit 192 of FIG.

Next, a fifth embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system which performs an image synthesizing process according to the flowchart of FIG. This electronic camera system (hereinafter referred to as an electronic camera) has the same configuration as that of the electronic camera 100 shown in FIG. 1, and particularly, the electronic camera is horizontally panned as shown in FIG. A composite image having a wide angle of view in the horizontal direction is generated from the two generated images. Further, in this electronic camera, a program corresponding to the flowchart of FIG. 19 is set in advance in an image composition processing unit corresponding to the image composition processing unit 172, and the image composition processing unit performs processing according to the program. Is designed to do.

In this electronic camera, except for the image synthesizing process performed by the image synthesizing processing section, the above-mentioned FIG.
The electronic camera 100 is the same as the electronic camera 100 of FIG. Further, the following description will be given by using this electronic camera as the electronic camera 100 with reference to FIGS.

First, in the electronic camera 100, for example, the pixel data of 640 × 480 pixels obtained by the image pickup section 110 is stored in the image memory 130. Further, the image memory 130 stores a series of plural image data obtained by panoramic photography as one file data.

Therefore, in the image composition processing section 172,
First, of the file data stored in the image memory 130, arbitrary file data consisting of a series of two image data is read out onto a memory (not shown) (step S11).

Next, the distortion caused by the taking lens 101 is corrected for each image data read into the memory (step S12).

Specifically, assuming that the distortion of the photographic lens 101 is rotationally symmetric with respect to the image center, the ideal position of the pixel having the image center as the origin when the photographic lens 101 is not distorted is (x , Y) and the position when there is distortion is (xd, xy), xd = x · (1 + k1 · r ² + k2 · r ⁴ ) yd = y · (1 + k1 · r ² + k2 · r ⁴ ) ··・ The relational expression (1) is established. Here, in this relational expression (1), “k1” and “k2” are distortion correction coefficients, and the relational expression of r ² = x ² + y ² is established.

Therefore, in step S12, a process of correcting the distortion of the image data is performed using the relational expression (1).

In this step S12, as shown in FIG. 20, first, an area for storing the corrected image data in which the distortion has been corrected is secured in the memory by the same two-dimensional array size as the input image data (step S12). S121).
Then, the correction image data in which the distortion of the taking lens 101 is corrected is obtained by performing the processes of steps S121 to S125 described below for each pixel of the image data.

That is, for each pixel of the image data, first, the pixel address of the corrected image data is set to the image center by using the size and the pixel pitch of the image pickup surface in the horizontal and vertical directions when the input image data is acquired. The coordinate system is converted to the origin coordinate system (step S122). The coordinates obtained in step S122 become the ideal position (x, y) when there is no distortion.

Next, the ideal position (x, y) obtained in step S122 is substituted into the relational expression (1) to obtain the position (xd, xy) at the time of distortion (step S).
123). Here, in the relational expression (1), the distortion correction coefficients k1 and k2 are values proportional to the third-order and fifth-order distortion aberration coefficients of the imaging lens 101, and the refractive index of the material of the imaging lens 101, the surface shape, and It can be determined by the configuration information including the lens arrangement. Therefore, here, it is assumed that the predetermined value determined by the configuration information is given in advance as the distortion correction coefficients k1 and k2, and the distortion correction coefficient k
Using 1 and k2, the position (xd, xy) at the time of distortion is obtained by the relational expression (1).

Next, using the size of the image pickup surface and the pixel pitch in the horizontal and vertical directions at the time of acquiring the input image data, the inverse conversion process of the conversion process performed in step S122 is performed to obtain the image before the distortion is corrected. In the data, the pixel address corresponding to the position (xd, xy) at the time of distortion obtained in step S123 is obtained (step S124).

Then, the RGB data, which is the pixel value of the pixel address obtained in step S124, is copied as the pixel value of the pixel address of the corrected image data (step S).
125). At this time, if the pixel address obtained in step S124 is outside the image area, for example, a white pixel value is assigned as the dummy pixel value.

By the processing of step S12 as described above, two corrected image data in which the distortion of the image pickup lens 101 is corrected are obtained from the two input image data, and thereafter,
The memory area reserved for storing the corrected image data in step S121 is released.

If the two input image data are images in which the distortion of the image pickup lens 101 can be almost ignored, the process of step S12 need not be performed.

Next, spherical image data is generated by projecting and converting the two corrected image data whose distortions have been corrected in step S12 into spherical surfaces (step S13).

More specifically, this step S13
Then, as shown in FIG. 21, first, an area for storing the image data after the spherical projection conversion is secured in the memory by the same two-dimensional array size as the input image data (corrected image data) (step S131). .

Next, the angular pitch of the pixels when performing the spherical projection conversion is obtained (step S132). At this time, the horizontal and vertical pixel angle pitches are such that the horizontal and vertical field angles of the image data after spherical projection conversion are equivalent to the horizontal and vertical field angles of the original input image data. Set equal pitch. That is, assuming that the size of the horizontal and vertical image planes at the time of input image data acquisition is “h × v” pixels and the focal length of the imaging lens 101 is “f”, the horizontal and vertical angle of view is (2. Tan ⁻¹ (h / (2 · f))) (2 · Tan ⁻¹ (v / (2 · f))). The focal length f indicates the distance between the subject image side principal point of the image pickup lens 101 and the light receiving surface of the image pickup element 103. Therefore, assuming that the size of the corrected image data is “H × V” pixels, the angular pitches dθ and dφ in the horizontal and vertical directions of the spherical projection are as follows: dθ = (2 · Tan ⁻¹ (h / (2 .F))) / H dφ = (2 · Tan ⁻¹ (v / (2 · f))) / V (2) In Expression (2), “Tan ⁻¹ ” indicates an inverse conversion of “Tan”.

Then, for each pixel of the image data, the processes of steps S133 to S137, which will be described later, are performed to obtain spherical projection image data.

That is, for each pixel of the image data, first, the spherical projection image data of the spherical projection image data is obtained using the horizontal and vertical angular pitches dθ and dφ obtained in step S132 and the size of the corrected image data. The pixel address is converted into an angular coordinate system (θ, φ) with the image center as the origin (step S133).

Next, the angular coordinate system (θ, φ) obtained in step S133 is set to the orthogonal coordinate system (X, Y, Z) as follows: X = cosφ · sin θ Y = sin φ Z = cos φ · sin θ. 3) Conversion is performed by the following equation (3) (step S134).
Incidentally, as shown in the equation (3), it is assumed that the polar coordinate with the Y axis as the rotation axis is converted into the orthogonal coordinate conversion, and the coordinate value in the radial direction is not affected by the processing after this step S134, so that "1" is given. ".

Next, the position of the focal length f from the Cartesian coordinate system (X, Y, Z) obtained in step S134 is obtained by the perspective transformation process centered on the focal position of the photographing lens 101 (hereinafter referred to as "viewpoint"). Position (x, y) on the imaging plane at
Is calculated by the equation (4): x = X · f / Z y = Y · f / Z (4) (step S135).

Next, using the size and pixel pitch of the image pickup surface in the horizontal and vertical directions at the time of acquiring the input image data, in the same manner as the above-described processing of step S124, step S124 is performed.
The inverse conversion process of the conversion process performed in 134 is performed to obtain the pixel address corresponding to the position (x, y) obtained in step S135 in the corrected image data before spherical projection conversion (step S136).

Then, similar to the process of step S125 described above, the RGB data which is the pixel value of the pixel address obtained in step S136 is copied as the pixel value of the pixel address of the spherical projection image data (step S1).
37).

By the processing in step S13 as described above, from the two corrected image data obtained in step S12, for example, data of two spherical images I61 and I62 projected on a spherical surface as shown in FIG. 22 can be obtained. . And
After the two spherical image data are obtained, step S131
The memory area reserved for storing the spherical image data is released.

Next, the corresponding points between the two spherical image data obtained in step S13 are extracted (step S1).
4).

Here, in this electronic camera 100, the user can specify a set of corresponding points between two spherical image data with a cursor of several points or the like. Therefore, in step S14, the exact position of the designated corresponding point is obtained by the template matching process.

More specifically, in step S14, as shown in FIG. 23, first, two spherical image data are displayed on the screen by the display unit 173 (step S14).
1). Here, the two spherical image data are also referred to as a left image and a right image hereinafter.

Next, the user operates an operation unit (not shown) to specify several sets of corresponding points, and the coordinates of the specified several sets of corresponding points are read (step S1).
42).

Then, the template matching process shown in steps S143 to S146, which will be described later, is performed on each of the designated sets of corresponding points.

That is, for each set of corresponding points, first, of the two spherical image data, the image data is cut out from the left image as a template (step S14).
3). The cut-out template is image data of a rectangular area of a predetermined size centered on the designated point on the left image.

Next, a region for retrieving points corresponding to the template cut out in step S143 is set from the right image (step S144). The search area is a rectangular area of a predetermined size centered on the designated point on the right image.

Next, in the search area set in step S144, the templates cut out in step S143 are shifted in parallel to obtain the difference between the left image and the right image. This difference is obtained only by the G component of the RGB components of the image data. Then, the position where the sum of the absolute values of the obtained differences is the minimum is set as the corresponding point position (step S145).

Then, the reliability of the corresponding point position obtained in step S145 is judged (step S14).
6). This reliability determination processing is performed by using the sum of absolute values of the differences having the minimum value and the sum of absolute values of the differences having the second smallest values. For example, when the minimum sum of absolute differences is less than or equal to a second predetermined threshold and the second smallest sum of absolute differences is greater than or equal to a first predetermined threshold. First, it is determined that the corresponding point position obtained in step S145 is reliable. In this way, the coordinates of the left and right images of the corresponding point positions determined to be reliable are
The extracted corresponding point data is stored in the memory.

By the processing in step S14 as described above, the position coordinates of the corresponding points between the two spherical image data obtained in step S13 can be obtained.

Next, the parameters for synthesizing the two spherical image data are calculated from the position coordinates of the corresponding points obtained in step S14 (step S15). here,
It is assumed that the focal length of the taking lens 101 before and after the panning does not change, and three parameters of parallel movement and rotation in the horizontal and vertical directions are calculated as the above parameters. The calculation of these three parameters is 2
The least squares method is used from the position coordinates of the corresponding points of the set or more.
Therefore, in this step S15, the parameters of the horizontal and vertical translation and rotation of the right image with respect to the left image are obtained.

In step S15, if the focal length of the taking lens 101 before and after panning has changed, it is necessary to calculate the enlargement / reduction parameters.
Also, among the three parameters of horizontal and vertical translation and rotation, the parameters of translation and rotation in the vertical direction are values close to “0”, so constraint conditions are set for these two parameters. Then, optimization may be performed to calculate the parameters.

Next, the right and left images are synthesized by horizontally moving and rotating the right image in accordance with the parameters obtained in step S15 (step S16). Here, since the two images combined in step S16 are images that have been spherically projected in advance in step S13, the horizontal and vertical translations of the images are the same as the panning in the horizontal and vertical directions of the image before spherical projection. Equivalent to.

More specifically, in step S16, as shown in FIG. 24, first, the size of the two-dimensional array of the combined image data is found, and the area of the found size is the area for storing the combined image data. Is secured in the memory (step S161). Here, since the parameters of the parallel translation and rotation in the vertical direction are close to "0", the size in the vertical direction is set to the same value as the size in the vertical direction of the image data before composition in step S161, and The size of is the value obtained by adding the size corresponding to the number of pixels of the horizontal translation parameter obtained in step S15 to the horizontal size of the image data before composition.

Then, the processing of steps S162 to S165, which will be described later, is performed on each pixel of the composite image data.

That is, for each pixel of the composite image data, first, the pixel address of the composite image data is converted into the angular coordinate system using the angular pitch at the time of spherical projection obtained in step S13 (step S162). ). At this time, the origin of the angular coordinate system is made to coincide with the center point of the left image, so that the pixel data of the left image can be directly copied without coordinate conversion.

Next, the combined image data is translated and rotated in the horizontal and vertical directions in accordance with the parameters obtained in step S15, so that the angular coordinate system of the combined image data obtained in step S162 is changed to the corner of the right image. The coordinate system is converted (step S163).

Next, the size of the right image and step S13
The angular coordinate system of the right image is converted into the pixel address of the right image using the angular pitch at the time of spherical projection obtained in (step S164).

Then, a pixel value is assigned to the pixel address of the composite image data (step S165). At this time, for the pixel whose pixel address is in the image area of the left image and whose pixel address obtained in step S164 is in the image area of the right image, the pixel values of the left and right images Assign an average value. Further, the pixel value of the right image is assigned to the pixel in the image area of only the left image, and the pixel value outside the image area of both the left image and the right image is set as a dummy pixel value, for example, , Assign white pixel values.

By the processing in step S16 as described above, the two spherical image data obtained in step S13 are combined, for example, a combined image I6 as shown in FIG.
3 is obtained, and thereafter, the memory area secured for storing the combined image data in step S161 is released.

Next, the composite image data obtained in step S16 is reprojected onto a plane to obtain image data (step S17).

More specifically, in step S17, as shown in FIG. 26, first, the area for storing the image data after the plane projection conversion is stored in the memory by the same size of the two-dimensional array as the composite image data. It is secured (step S171).

Next, the pixel pitch for the plane projection is obtained (step S172). The pixel pitch in the horizontal and vertical directions at this time is set to a pitch that is equivalent to the angle of view in the horizontal and vertical directions of the image data (planar composite image data) after the plane projection conversion on the imaging surface of the focal length f. To do. That is, for the half angles of view in the horizontal and vertical directions, the size of the composite image data is set to “H × V” pixels, and step S13 is performed.
The angular pitches dθ and d of the spherical projection image data obtained in
By using φ, it is represented by (tan (dθ · (H / 2))) (tan (dφ · (V / 2))). Therefore, the pixel pitch of the image subjected to the plane projection conversion is: dx = ftan (dθ · (H / 2)) / (H / 2) dy = ftan (dφ · (V / 2)) / (V / 2) -(5) It is calculated by the equation (5). The pixel pitch obtained by the equation (5) is equal to the pixel pitch of the image obtained at the time of shooting when an image projected on the imaging surface having the same focal length as that at the time of shooting is generated.

Then, for each pixel of the image data, the processes of steps S173 to S177, which will be described later, are performed to obtain the plane composite image data.

That is, for each pixel of the image data, first, using the pixel pitches in the horizontal and vertical directions and the size of the image data obtained in step S172, in the same manner as the above-described step S122, The coordinate system (x,
y) (step S173).

Next, from the viewpoint, a point (x, y,
The intersection (X, Y, Z) between the straight line drawn in f) and the spherical surface centering on the viewpoint is obtained (step S174). At this time, the radius of the spherical surface is set to "1" because it does not affect the processing after step S174. Next, step S174.
Intersection obtained in, ie, Cartesian coordinate system (X, Y, Z)
Is converted to a spherical coordinate system by the equation (6) such that θ = sin ⁻¹ (X / sqrt (X ² + Z ² )) φ = sin ⁻¹ (Y / Z) (6) (step S175 ). Here, in formula (6), “sin ⁻¹ ”
Represents the inverse transformation of "sin", and "sqrt" represents the square root.

Next, using the size and angle pitch of the image pickup surface in the horizontal and vertical directions of the combined image data, the pixel address corresponding to the spherical coordinates obtained in step S175 in the image data before the plane projection conversion is determined. Obtain (step S17
6).

Then, similar to the processing of step S125 described above, the RGB data which is the pixel value of the pixel address obtained in step S176 is copied as the pixel value of the pixel address of the image data after the plane projection conversion (step). S177).

By the processing in step S17 as described above, for example, a plane composite image I64 projected on a plane as shown in FIG. 27 is obtained from the composite image data obtained in step S16, and then in step S171. The memory area reserved for storing the plane composite image data is released.

Finally, the plane composite image data obtained in step S17 is displayed on the screen by the display unit 173,
If necessary, the image is stored in the image memory 130 (step S18).

As described above, in the electronic camera 100, the taking lens 101 is configured so that the parallel movement of the images during the image synthesizing process corresponds to the horizontal and vertical panning of the images during the photographing. When synthesizing two images obtained by panning around the focus position of the image, even if some camera tilt occurs at the time of shooting, a double border appears in the overlapping portion of the combined image. Never. Therefore, the electronic camera 100 can obtain a natural composite image as if it were taken with a camera lens having a wider angle than the actually taken camera lens.

Although the electronic camera described above has the same configuration as the electronic camera 100 in FIG. 1, it may have the same configuration as the electronic camera 200 in FIG. 2, for example.

Next, a sixth embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system including an image synthesizing processing unit 300 as shown in FIG. This electronic camera system (hereinafter referred to as an electronic camera) has the same configuration as the electronic camera 100 shown in FIG.
Instead of the above, the image composition processing unit 300 of FIG. 28 is provided. Further, a predetermined program is set in advance in the image synthesizing processing unit 300, and the image synthesizing processing unit 300 is adapted to perform processing according to the program.

In this electronic camera, each unit other than the image synthesis processing unit 300 is the same as the electronic camera 100 shown in FIG.
Therefore, detailed description of each unit other than the image synthesis processing unit 300 will be omitted.

First, the image synthesizing processor 300 operates as shown in FIG.
As shown in FIG. 8, it includes a control unit 301, and an image input unit 302, an image conversion unit 303, and an image synthesis unit 304 which are connected to the control unit 301.

The control unit 301 controls the operation of the entire apparatus so as to perform the processing in accordance with the above-mentioned program. For example, the image input unit 302 and the image are input according to the operation of the operation unit (not shown) by the user. Control signals are supplied to the conversion unit 303 and the image synthesis unit 304, respectively. As a result, the composite image and the like obtained by the image composition processing unit 300 is displayed on the screen by the display unit 173 of FIG. Image input unit 3
In 02, a program for performing the same processing as step S11 shown in FIG. 19 is preset. As a result, the image input unit 302 reads out a series of a plurality of image data designated by the user to a memory (not shown) based on the control signal from the control unit 301. The image conversion unit 303 has steps S12 and S1 shown in FIG.
3 and a program for performing the same processing as S17 are set in advance. As a result, the image conversion unit 303 causes the image input unit 30 to operate based on the control signal from the control unit 301.
The distortion correction process, the spherical projection conversion process, and the planar projection conversion process are performed on the plurality of image data read in step 2.
The image synthesizing unit 304 has the step S shown in FIG.
A program that performs the same processing as 14 to S16 is set in advance. As a result, the image synthesizing unit 304, based on the control signal from the control unit 301,
Corresponding point extraction processing for the plurality of image data read in step 2, parameter calculation processing for synthesizing a plurality of image data from the coordinates of the corresponding points obtained by the corresponding point extraction processing, and the above parameter calculation A synthesizing process for generating a synthetic image is performed by performing parallel movement and rotation according to the parameters obtained by the process.

Hereinafter, the image composition processing section 300 as described above.
In the following, for example, as shown in FIG. 36, a case will be described in which images c1 to c4 obtained by panning in both the horizontal and vertical directions and shooting with four times of framing are combined.

Here, the user operates the operation unit (not shown) to display images c1 and c2 and images c3 and c.
4 is given, and an instruction (command) is given to the apparatus so as to synthesize the images of the respective synthesis results. Also, the image c1
It is assumed that parameters c4 to c4 are stored in the image memory 130, and parameters such as an image size, a pixel pitch, and a focal length necessary for the image conversion process are given in advance.

Therefore, FIG. 29 is a flow chart showing the processing of the image composition processing unit 300 in the above case. Hereinafter, the image composition processing unit 3 will be described with reference to FIG.
The operation of each unit of 00 and the image combining process will be described.

First, the image input unit 302 includes the control unit 301.
The data of the image c1 stored in the image memory 130 is read out onto a memory (not shown) based on the control signal from the (step S201). This read image c1
Is displayed on the screen of the display unit 173 under the control of the control unit 301.

Next, the user confirms the image c1 displayed on the screen of the display unit 173 and operates an operation unit (not shown) to cause the image conversion unit 303 to perform distortion correction processing and spherical projection on the image c1. Give the device a command by which the conversion process is performed. Based on this command, the control unit 3
When 01 supplies the control signal to the image conversion unit 303, the image conversion unit 303 performs the distortion correction process and the spherical projection conversion process on the image c1 (step S20).
2). Then, the control unit 301 causes the display unit 173 to display a screen of the image c1 that has been subjected to the distortion correction processing and the spherical projection conversion processing by the image conversion unit 303.

Next, the image c2 is also read into the above memory, subjected to the distortion correction processing and the spherical projection conversion processing, and displayed on the screen of the display unit 173 in the same manner as the processing of steps S201 and S202. Steps S203 and S20
4).

Next, the user operates an operation unit (not shown) to specify several sets of corresponding points of the two images c1 and c2 displayed on the display unit 173. Based on this designation, the control unit 301 supplies a control signal to the image synthesizing unit 304, so that the image synthesizing unit 304 obtains the coordinates of the designated sets of corresponding points and detects the position of the accurate corresponding points. Therefore, template matching processing is performed for each corresponding point. Then, the image synthesizing unit 304 obtains the parameters of translation and rotation between the two images c1 and c2 from the positions of the corresponding points detected by the template matching process by the method of least squares, and based on the obtained parameters, 2 Two images c1 and c2 are combined (step S
205).

Then, the control unit 301 controls the image synthesizing unit 30.
The composite image c12 obtained in No. 4 as shown in FIG. 30 is displayed on the screen by the display unit 173. In addition, the control unit 301
The composite image c12 is temporarily stored in the image memory 130 (step S206).

Next, as in steps S201 to S206, the two images c3 and c4 are combined to generate a combined image c34 as shown in FIG.
(Steps S207 to S212).

Next, the image input unit 302 is controlled by the control unit 301.
The two composite images c12 and c34 stored in the image memory 130 are read out onto a memory (not shown) on the basis of the control signal from (steps S213 and S214).

Next, the image synthesizing section 304 carries out step S2.
The two combined images c12 and c34 are combined in the same manner as the processing of 05 and S211 (step S215). Here, since the composite images c12 and c34 are images that have undergone distortion correction processing and spherical projection conversion processing by the image conversion unit 303, there is no need to perform distortion correction processing and spherical projection conversion processing by the image conversion unit 303. . By this step S215, a combined image I71 obtained by combining the four images c1 to c4 as shown in FIG. 31 is obtained. Since the composite image I71 is a composite of two spherical projection-converted images, it will be referred to as a spherical composite image I71 hereinafter.

Next, the image conversion section 303 uses the image composition section 3
From the spherical composite image I71 obtained in 04, a plane composite image I72 as shown in FIG. 32 is obtained by projecting and converting the spherical composite image I71 (step S216).

Then, the control unit 301 controls the image conversion unit 30.
The plane composite image I72 obtained in 3 is stored in the image memory 130 (step S217).

As described above, since this electronic camera is constructed so as to perform the image synthesizing process in accordance with the predetermined command given by the user operating the operation unit (not shown), the user A desired composite image can be obtained. In addition, this electronic camera sequentially repeats a process of combining two images with respect to a plurality of images obtained by panning in both the horizontal and vertical directions around the focal position of the camera lens. Since the image is generated and the composite image is projected and converted again on a plane, a natural composite image with a wide angle of view, which is taken by a camera lens wider than the actual camera lens, is obtained. be able to.

If the image to be combined in the image combining processing unit 300 shown in FIG. 28 is always an image subjected to spherical projection conversion, the process of the image converting unit 303 may be automated. More specifically, a flag (distortion correction flag) indicating whether or not the image data to be combined by the image combining processing unit 300 is distortion correction processed data, and spherical projection conversion processed data or plane. A flag (projection plane flag) indicating whether the data has been subjected to the projection conversion process is added as additional information. Therefore, the image conversion unit 303 sets the distortion correction flag to processed when the distortion correction processing is performed, and sets the projection surface flag to spherical projection conversion processing when the spherical projection conversion processing is performed. Then, when the plane projection conversion process is performed, the projection plane flag is set to the plane projection conversion process completed. Also,
When performing the image synthesizing process, the image converting unit 303 always performs the distortion correcting process based on the additional information of the image data to be synthesized, and the image synthesizing unit 304 synthesizes the spherical projection-converted images. To do. Further, when outputting the combined image, the image combining unit 304 always outputs the combined image subjected to the plane projection conversion based on the additional information of the image data to be combined. With the configuration as described above, a high quality composite image can be efficiently obtained.

The above-described electronic camera is designed to obtain a natural composite image as if it was taken by a camera lens having a wider angle than the actually taken camera lens by performing the plane projection conversion process. It is not necessary to output the composite image that has undergone the plane projection conversion process. More specifically, since the electronic camera is configured to sequentially repeat the process of synthesizing two images to generate a synthetic image, it is possible to synthesize, for example, five or more images. However, when the field of view of the combined image reaches 180 °, the plane projection conversion process cannot be performed. Therefore, in such a case, it is desirable to output the combined image subjected to the spherical projection conversion process. Therefore, the image obtained by the plane projection conversion process and the image obtained by the spherical projection conversion process may be selectively output according to the visual field of the composite image.

In the electronic camera described above, parameters such as image size, pixel pitch, and focal length necessary for image conversion processing are given in advance. However, these parameters are used by the image synthesizing processing unit 300. It may be added as additional information to the image data to be combined.
In this case, the image conversion unit 303 and the image composition unit 304
A parameter is read from the additional information added to the image data to be combined, and processing is performed based on the parameter.
At this time, the pitch on the imaging surface is used as the pixel pitch parameter for the image subjected to the plane projection conversion process, and the pixel pitch parameter is used for the image subjected to the spherical projection conversion process. Therefore, it is desirable to use the angular pitch.

In the electronic camera described above, the above-described processes are performed according to a predetermined program. However, the program is directly implemented as hardware,
It can also be used as an image composition processing device.

Further, in the electronic camera described above,
Although the electronic camera 100 has the same configuration as that of the electronic camera 100, it may have the same configuration as the electronic camera 200 of FIG.

Next, a seventh embodiment of the present invention will be described with reference to the drawings.

The image generating apparatus according to the present invention is applied to, for example, an electronic camera system 400 as shown in FIG.

As shown in FIG. 33, this electronic camera system (hereinafter, simply referred to as an electronic camera) 400 has, in addition to the structural requirements of the electronic camera 100 of FIG. 1, an angle connected to the signal processing unit 190. The detector 401 is provided. The angle detection unit 401 uses a gyro or the like, and detects a panning angle generated due to the movement of the electronic camera 400 during shooting. Then, the electronic camera 400 is configured to perform an image synthesizing process based on the information of the panning angle detected by the angle detection unit 401.

In the electronic camera 400 shown in FIG. 33, parts that operate in the same manner as in the electronic camera 100 described above are given the same reference numerals, and detailed description thereof will be omitted. Also,
Regarding the operation of the image synthesizing processing unit 172 described later, a part different from the operation in the above-described first embodiment will be specifically described, and the other parts are the same as those in the above-described first embodiment. , Its detailed description is omitted.

That is, as shown in FIG. 34, the angle detector 401 includes the angular velocity sensor 401a and the angular velocity sensor 4a.
A / D converter 401b to which the output of 01a is supplied,
Angle calculator 40 to which the output of the D / D converter 401b is supplied
1c, and the output of the angle calculator 401c is supplied to the signal processing unit 190.

First, the angle sensor 401a supplies the A / D converter 401b with the output signal V according to the angle change generated by the movement of the device. The level of this output signal V is
It is proportional to the angular velocity. A / D converter 401b
Outputs the output signal V from the angle sensor 401a as digital data D to the angle calculator 401c. The angle calculator 401c is the A / D converter 401
An average level Davg is obtained by integrating the digital data D from b for 1 s, for example, and this average level Davg is converted into an angle component to obtain an angle signal. Then, the angle signal obtained by the angle calculator 401c is supplied to the signal processing unit 190.

Then, the signal processing unit 190 writes the information of the angle signal obtained by the angle detector 401 in the header portion of the image memory 130 in correspondence with the image data written in the image memory 130.

Next, in the image composition processing section 172,
As shown in FIG. 10, the image information separating section 172f separates the image data read from the image memory 130 into a header section and a data section, and the controller 172e is supplied with header section information (header information). Information of the data section (image information) is written in the image memory 172g.

The controller 172e compares the information of the angle component at the time of photographing included in the header information supplied from the image information separating unit 172f with a preset threshold value, and the angle component is the threshold value. If it is greater than or equal to the above, it is determined to be the long-distance panoramic shooting, and if the angle component is equal to or less than the threshold value, it is determined to be the short-distance panoramic shooting and the determination result is supplied to the selector 172k.

The selector 172k is the controller 172.
According to the determination result from e, the short-distance parameter extraction unit 1
Either 72c or the long-distance parameter extraction unit 172d is selected, and the information of the corresponding points obtained by the corresponding point detection unit 172b is supplied.

Thereafter, the same processing as in the above-described first embodiment is performed to generate a composite image.

As described above, in the electronic camera 400, based on the angle component generated by the movement of the device at the time of photographing,
Since the image synthesizing process is automatically selected, it is possible to perform an appropriate image synthesizing process with respect to movement of the device such as panning. Therefore, the electronic camera 40
With 0, a high quality panoramic image can be obtained efficiently.

In the electronic camera 400 described above, the electronic camera 40 is provided by providing the angular velocity sensor 401a.
Although the angle component generated by the movement of 0 is detected, by providing an acceleration sensor or the like, the electronic camera 4
The translational component (translational movement amount) generated by the movement of 00 may be detected. In this case, the image composition processing unit 1
At 72, the controller 172e determines to be a short-distance panoramic shooting when the detected translational movement amount is larger than a preset threshold value, and when the translational movement amount is smaller than the threshold value. Is a long-distance panoramic shot. Also in this case, a high quality panoramic image can be efficiently obtained as in the case of detecting the angle component.

[0178]

As described above, according to the present invention, since the composite image is generated by the appropriate image composition processing selected according to the photographing conditions, a high quality composite image is always obtained. be able to. According to the present invention,
By configuring to automatically generate a composite image without the user performing an operation to combine the images,
The load on the user can be reduced, and a composite image can be easily obtained. Further, according to the present invention, it is automatically determined whether the image to be combined is obtained by short-distance photography or long-distance photography based on the focus position information, and the discrimination result Since the composite image is configured to be generated by the appropriate image composition processing selected by, the high-quality composite image can be easily obtained without depending on the subject distance. Further, according to the present invention, when synthesizing a plurality of images, an image in the vicinity of a connecting portion between adjacent images in the plurality of images is converted based on the shooting condition so that the vicinity of the connecting portion becomes inconspicuous. With the configuration, it is possible to obtain a high-quality composite image without a feeling of strangeness. Further, according to the present invention, the density level of the image in the vicinity of the connecting portion between the adjacent images in the plurality of images is corrected based on the photographing condition so as to make the image in the vicinity of the connecting portion inconspicuous, thereby making the user feel uncomfortable. It is possible to obtain a high-quality composite image with no image. Further, according to the present invention, since the spherical conversion processing is performed on the plurality of images to be combined based on the shooting condition, it is possible to obtain a natural high-quality combined image from which distortion is removed. Further, according to the present invention, even when a plurality of images obtained by panning in both the horizontal and vertical directions around the focus position at the time of shooting are combined, a plurality of images in the vicinity of a connection portion with adjacent images are combined. By configuring so that the images do not overlap, a natural high quality composite image can be obtained. Further, according to the present invention, since the composite image is configured so as to be re-projected on a plane, a high quality composite image as if taken by a camera lens having a wider angle than the actually taken camera lens can be obtained. You can For example, by sequentially repeating the combining process of two images, it is possible to obtain a combined image having a wider angle of view from a large number of images photographed by free framing. Further, according to the present invention, since the spherical projection conversion processing and the plane projection conversion processing are automatically performed without instructing, a high quality composite image can be easily obtained. Further, according to the present invention, since the composite image is configured to be generated by the appropriate image composition processing according to the visual field of the composite image, the visual field of the image to be captured does not depend on the visual field.
It is possible to always obtain a high quality composite image. Further, according to the present invention, the user is configured to automatically perform short-distance photography and long-distance photography according to the subject distance without performing operations for short-distance photography and long-distance photography. Therefore, a composite image can be easily obtained. Further, according to the present invention, since the image synthesizing method is switched according to the photographing information at the time of photographing, the optimum image synthesizing process can be performed on the movement information of the device. Therefore, a high-quality composite image can be efficiently obtained. For example, it is possible to perform optimal image composition processing with respect to movement of the device such as panning.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an electronic camera system to which an image generating apparatus according to the present invention is applied in a first embodiment of the present invention.

FIG. 2 is a plan view showing a subject for short-distance panoramic photography.

FIG. 3 is a diagram for explaining a shooting condition of the short-distance panoramic shooting.

FIG. 4 is a plan view showing two images obtained by the short-distance panoramic photography.

FIG. 5 is a plan view showing a combined image obtained by combining the two images.

FIG. 6 is a diagram for explaining a shooting situation of long-distance panoramic shooting.

FIG. 7 is a plan view showing three images obtained by the long-distance panoramic photography.

FIG. 8 is a plan view showing a combined image obtained by combining the above three images.

FIG. 9 is a diagram for explaining image data stored in an image memory of the electronic camera system.

FIG. 10 is a block diagram showing a configuration of an image composition processing unit of the electronic camera system.

FIG. 11 is a block diagram showing a configuration of an image composition processing unit of an electronic camera system to which an image pickup apparatus according to the present invention is applied in the second embodiment of the present invention.

FIG. 12 is a diagram for explaining spherical projection processing of the image composition processing unit.

FIG. 13 is a diagram for explaining a spherical projection image obtained by the spherical projection processing.

FIG. 14 is a block diagram showing a configuration of an image composition processing unit of an electronic camera system to which an image pickup apparatus according to the present invention is applied in the third embodiment of the present invention.

FIG. 15 is a plan view showing an image to be processed by the image composition processing unit.

FIG. 16 is a diagram for explaining a density level correction process of the image composition processing unit.

FIG. 17 is a block diagram showing a configuration of an electronic camera system to which an image pickup apparatus according to the present invention has been applied in the fourth embodiment of the present invention.

FIG. 18 is a diagram for explaining a process of controlling each optical axis of two imaging units.

FIG. 19 is a flowchart showing a process of an image synthesizing processing unit of an electronic camera system to which an image pickup apparatus according to the present invention has been applied, in the fifth embodiment of the present invention.

FIG. 20 is a flowchart specifically showing an image input process in the process of the image composition processing unit.

FIG. 21 is a flowchart specifically showing spherical projection conversion processing in the processing of the image composition processing section.

FIG. 22 is a plan view showing two spherical projection images obtained by the spherical projection conversion processing.

FIG. 23 is a flowchart specifically showing a corresponding point extraction process in the process of the image composition processing unit.

FIG. 24 is a flowchart specifically showing an image synthesizing process in the process of the image synthesizing processing unit.

FIG. 25 is a plan view showing a spherical projection combined image obtained by the image combining processing.

FIG. 26 is a flowchart specifically showing a plane projection conversion process in the process of the image composition processing unit.

FIG. 27 is a plan view showing a plane projection combined image obtained by the plane projection conversion process.

FIG. 28 is a block diagram showing a configuration of an image composition processing unit of an electronic camera system to which an image pickup apparatus according to the present invention has been applied, in the sixth embodiment of the present invention.

FIG. 29 is a flowchart showing the processing of the image composition processing unit.

FIG. 30 is a plan view showing two synthetic images obtained by synthesizing two spherical projection images in the image synthesis processing unit.

FIG. 31 is a plan view showing a composite image obtained by combining the two composite images.

FIG. 32 is a plan view showing an image obtained by performing a plane projection conversion process on the composite image.

FIG. 33 is a block diagram showing the configuration of an electronic camera system to which an image generating apparatus according to the present invention has been applied in the seventh embodiment of the present invention.

FIG. 34 is a block diagram showing a configuration of an angle detection unit of the electronic camera system.

FIG. 35 is a diagram for explaining a case where two images are captured by panning a conventional electronic camera system in the horizontal direction.

FIG. 36 is a plan view showing two images obtained by the electronic camera system.

FIG. 37 is a plan view showing a combined image obtained by combining the two images.

FIG. 38 is a plan view showing four images obtained when the rectangular frame is photographed by framing four times by the electronic camera system.

FIG. 39 is a plan view showing a combined image obtained by combining the above four images only by parallel movement without projecting the image on the cylindrical surface.

FIG. 40 is a plan view showing a combined image obtained by temporarily projecting an image on a cylindrical surface and translating the image and combining the four images.

[Explanation of symbols]

 172 Image synthesis processing unit 172a Input / output unit 172b Corresponding point detection unit 172c Short-distance shooting parameter extraction unit 172d Long-distance shooting parameter extraction unit 172e Controller 172f Image information separation unit 172g Image memory 172h Composite image memory 172i Short-distance shooting coordinate conversion process Part 172j long-distance photographing coordinate conversion processing part 172k selector

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kenji Hatori 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc.

Claims (13)

[Claims] 1. A photographing device divides an image of a subject into a plurality of screens so that a part of the screens overlaps each other, and a series of a plurality of images obtained by the photographing device is combined to generate a combined image. An image generating apparatus, a detection means for detecting shooting conditions at the time of shooting, and a storage means for storing the plurality of images obtained by the shooting means and the shooting conditions detected by the detection means corresponding to each image. And an image synthesizing unit that synthesizes a series of images stored in the storage unit to generate a synthetic image, and a control unit that controls the image synthesizing unit. And a control means for selectively switching the plurality of composition means based on a photographing condition corresponding to each image, and the composition means selectively switched by the control means is Image generation apparatus characterized by combining a plurality of images of communicating. 2. The image synthesizing means, the corresponding point detecting means for detecting corresponding points in the overlapping portions of the respective images, the coordinate converting means for subjecting the respective images to a coordinate converting process to generate a synthetic image, and the correspondence check. Parameter generating means for generating a photographing parameter based on the corresponding point detected by the outputting means, wherein the coordinate converting means performs the coordinate converting process using the photographing parameter generated by the parameter generating means. The image generation apparatus according to claim 1, characterized in that 3. The detecting means detects focus position information at the time of photographing as the photographing condition, and the image synthesizing means combines short-distance synthesizing means for synthesizing a series of a plurality of images obtained by short-distance photographing. A long-distance synthesizing means for synthesizing a series of a plurality of images obtained by the long-distance shooting; It is discriminated whether it is the one obtained by long-distance photographing or the one obtained by long-distance photographing, and the short-distance composition means and the long-distance composition means are selectively switched based on the discrimination result. 1. The image generation device according to 1. 4. The image generating apparatus according to claim 1, wherein the image synthesizing unit includes a converting unit that converts a pixel value of an overlapping portion of each image based on a shooting condition corresponding to each image. 5. The detecting means detects exposure information at the time of shooting as a shooting condition, and the converting means corrects the density level of an overlapping portion of each image based on the exposure information corresponding to each image. The image generating apparatus according to claim 4, characterized in that 6. The image synthesizing means has a spherical projection converting means for projecting and converting each image onto a spherical surface to generate a spherical projection image based on a photographing condition corresponding to each image, and the spherical projection converting means. The image generation apparatus according to claim 1, wherein the plurality of spherical projection images obtained in (1) are combined. 7. The image generation apparatus according to claim 6, wherein the spherical projection conversion means performs projection conversion into a spherical surface centered on a focal position at the time of photographing. 8. The plane image conversion means is a plane projection conversion means for projecting a composite image obtained by combining the plurality of spherical projection images obtained by the spherical projection converting means onto a plane to generate a plane projection combined image. The image generating apparatus according to claim 6, further comprising: 9. The image synthesizing means comprises addition means for adding projection plane type information indicating whether the image to be processed is the spherical projection image or the plane projection synthetic image to the image. The image generating apparatus according to claim 8, which is characterized in that. 10. The image synthesizing means selectively switches between the synthetic image and the plane projection synthetic image according to the field of view of the synthetic image obtained by synthesizing the plurality of spherical projection images obtained by the spherical projection converting means. The image generating apparatus according to claim 8, further comprising an output unit that outputs the image. 11. The image pickup means comprises a plurality of image pickup means, and optical axis control means for controlling the directions of the respective optical axes of the plurality of image pickup means based on the image pickup conditions detected by the detection means. The image generation apparatus according to claim 1, wherein 12. An image generating apparatus for synthesizing a plurality of captured images to generate one image, comprising means for setting a panoramic shooting mode, means for detecting an angle of the apparatus, and shooting in the panoramic shooting mode. A means for holding the information of the angle together with the plurality of captured images obtained by, and means for synthesizing the plurality of captured images, when synthesizing the plurality of captured images, based on the information of the angle An image generating apparatus characterized by adaptively switching a synthesizing method. 13. An image generating apparatus for synthesizing a plurality of photographed images to generate one image, comprising means for setting a panoramic photographing mode, means for detecting the position of the apparatus, and photographing in the panoramic photographing mode. A means for holding the position information together with the plurality of captured images obtained by, and means for synthesizing the plurality of captured images, based on the position information when synthesizing the plurality of captured images An image generating apparatus characterized by adaptively switching a synthesizing method.