JPH09322055A - Electronic camera system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide the high performance electronic camera system in which a high quality panorama image is simply obtained. SOLUTION: An angle detection section 126 detects an angle component of a device caused by movement of the device. An image memory 130 stores information of angle components detected by the angle detection section 126 and an image pickup condition detected by a signal processing unit 190 as additional information in cross reference with each image obtained through image pickup. An overlap area prediction means of an image synthesis processing section 172 predicts an overlapped area of the images to be synthesized based on the additional information stored in the image memory 130. Furthermore, a cross reference detection means detects a cross reference point in the overlapped area predicted by the overlap area prediction means. Then the image synthesis means synthesizes a series of images based on the cross reference point detected by the cross reference detection means.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an electronic camera system for synthesizing a plurality of images obtained by photographing in a panoramic photographing mode to generate a panoramic image.

[0002]

2. Description of the Related Art Conventionally, a subject image is divided into a plurality of screens so as to overlap a part of adjacent screens, and a panorama image is taken by combining the plurality of images obtained by the shooting. As an electronic camera system for generating the, there is, for example, the electronic camera and its reproducing device disclosed in Japanese Patent Laid-Open No. 4-52635. This electronic camera system is designed to detect information relating to frames or cuts such as a three-dimensional position and orientation of the device, store the information together with a video signal, and reproduce the video signal based on the information. is there. As a result, the electronic camera system can support various applications such as reproduction in a three-dimensional space inside the computer by retaining the positional relationship between frames or cuts of a captured image and reproducing the image. I am going to do it.

[0003]

However, in the conventional electronic camera system as described above, when a plurality of images are combined to generate a panoramic image, each image is converted into an image of equal magnification based on the zoom magnification information. In addition to converting it to a frame memory and recording it in a frame memory, information such as three-dimensional position and orientation is recorded simultaneously as supplementary information in association with each image, and the recorded image data are simply combined to generate a panoramic image. It was That is, the electronic camera system only mounts a plurality of images obtained by panoramic photography on the frame memory according to information such as the three-dimensional position and orientation. It was just placed. Therefore, the seams were discontinuous, resulting in a very unsightly panoramic image.

Therefore, the present invention has been made to eliminate the above-mentioned drawbacks, and an object of the present invention is to provide a high-performance electronic camera system which can easily obtain a high-quality panoramic image.

[0005]

An electronic camera system according to the present invention divides a subject image into a plurality of screens so that a part of the screens is overlapped by the image pickup means, and photographs it. An electronic camera system for synthesizing a plurality of images to generate a panoramic image, the shooting condition detecting means for detecting shooting conditions at the time of shooting, and a moving state detecting means for detecting a moving state of the device at the time of shooting, A storage unit that stores, as additional information, a plurality of images obtained by shooting, together with the information of the shooting condition detected by the shooting condition detecting unit and the moving state detected by the moving state detecting unit corresponding to each image. And an overlap area prediction unit that predicts and detects an overlap area of each image based on the additional information stored in the storage unit, and the overlap of each image using the overlap area obtained by the overlap area prediction unit. A panoramic image is generated by synthesizing a corresponding point detecting means for detecting a corresponding relationship in the area and a series of a plurality of images stored in the storing means on the basis of the information on the corresponding relationship obtained by the corresponding point detecting means. And an image synthesizing means. Further, the electronic camera system according to the present invention is characterized in that the moving state detecting means detects an angular component generated by the movement of the apparatus. Also, the electronic camera system according to the present invention is characterized in that the moving state detecting means detects the angle component by an angular velocity sensor. Further, the electronic camera system according to the present invention is characterized in that the movement state detecting means detects the translational component generated by the movement of the apparatus. Further, an electronic camera system according to the present invention is characterized in that the movement state detecting means detects the translational component by an acceleration sensor. Also, the electronic camera system according to the present invention includes position detection means for detecting the position of the device at the time of photographing, and the storage means stores the position information detected by the position detection means in the additional information. It is characterized by doing. The electronic camera system according to the present invention is characterized in that the position detecting means detects the three-dimensional position of the device. Further, the electronic camera system according to the present invention is arranged such that every time an image is inputted to the image pickup means, based on the information of the photographing condition detected by the photographing condition detecting means and the moving state detected by the moving state detecting means. In the overlapping area detecting means for sequentially detecting the overlapping degree indicating the overlapping area of each image, the comparing means for comparing the overlapping degree and the predetermined value obtained by the overlapping area detecting means, based on the comparison result of the comparing means. And a control means for controlling the photographing timing. Further, the electronic camera system according to the present invention is stored in the storage means, and a spherical projection conversion means for projecting and converting a series of images stored in the storage means onto a spherical surface to generate a spherical projection image. Switching means for selectively switching the series of a plurality of images and the plurality of spherical projection images obtained by the spherical projection conversion means for output to the corresponding point detecting means, and photographing from the additional information stored in the storage means A shooting situation detecting means for detecting a situation; and a switching control means for controlling the operation of the spherical projection converting means based on the detection result of the shooting situation detecting means and controlling the switching operation of the switching means. Characterize. Further, the electronic camera system according to the present invention is characterized by comprising an output unit for externally outputting the panoramic image obtained by the image synthesizing unit.

[0006]

According to the present invention, the moving state detecting means detects the posture of the device generated by the movement of the device as information on the moving state. The storage means stores the movement state information of the device detected by the movement state detection means and the photographing condition detected by the photographing condition detection means as additional information corresponding to each image obtained by photographing. It The overlapping area predicting means predicts an overlapping area between the images to be combined based on the additional information stored in the storage means. The correspondence detection unit detects, in each image to be combined, a corresponding point in the overlap region predicted by the overlap region prediction unit. The image composition means
A series of plural images are combined based on the corresponding points detected by the correspondence detecting means. Further, according to the present invention, the movement state detecting means detects the angular component of the device generated by the rotational movement of the device during photographing. Further, according to the present invention, the moving state detecting means detects the angle component by the angular velocity sensor. Further, according to the present invention, the movement state detecting means detects the translational component of the device generated by the translational movement of the device at the time of photographing. Further, according to the present invention, the moving state detecting means detects the translational component by an acceleration sensor. Further, according to the present invention, the position detecting means detects the position of the device generated by the movement of the device during photographing. The storage means stores the position information obtained by the position detection means in the additional information. Further, according to the present invention, the position detecting means detects the three-dimensional position of the device generated by the movement of the device at the time of photographing.
Further, according to the present invention, the overlapping area detecting unit is configured to sequentially input each image based on the information on the moving state of the device detected by the moving information detecting unit and the photographing condition detected by the photographing condition detecting unit. The degree of overlap of is sequentially detected by calculation.
The comparing means compares the degree of overlap detected by the overlapping area detecting means with a preset threshold value. The control means controls the operation of the entire device so that the device is in the shooting state at the timing when it is determined from the comparison result of the comparison device that the degree of overlap is equal to the threshold value. Further, according to the present invention, the photographing condition detecting means detects from what additional photographing the plurality of images to be combined were obtained from the additional information stored in the storage means. The switching control means, according to the detection result of the shooting situation detection means,
When the plurality of images to be combined are obtained by rotational movement, the spherical projection conversion means is controlled to project and convert the plurality of images onto the spherical surface to generate a spherical projection image. The switching means is controlled so that the spherical projection image obtained by the spherical projection conversion means is selected and output. According to the invention, the output means outputs the panoramic image generated by the image synthesizing means to the outside.

[0007]

BEST MODE FOR CARRYING OUT THE INVENTION First, a first embodiment of the present invention will be described with reference to the drawings.

The electronic camera system according to the present invention is applied to, for example, an electronic camera system 100 as shown in FIG.

As shown in FIG. 1, an electronic camera system (hereinafter, simply referred to as an electronic camera) 100 includes an imaging lens 101 and a diaphragm 10 which are sequentially provided from a subject side.
2, shutter 108 and image sensor 103, and image sensor 1
Amplifier 104 to which the output of 03 is supplied, and amplifier 104
Gain control (AGC) circuit 105 supplied with the output of
, An analog / digital (A / D) converter 106 to which the output of the AGC circuit 105 is supplied, and an A / D converter 10
6 is supplied to the video signal processing unit 107, and the output of the video signal processing unit 107 is supplied to the image memory 13 respectively.
0, the focus detection unit 142 and the exposure detection unit 143, and the white balance detection unit 1 connected to the video signal processing unit 107.
41 and the signal processing unit 190, the controller 120 to which the output of the signal processing unit 190 is supplied, and the zoom control unit 1 to which the output of the controller 120 is respectively supplied.
21, focus control unit 122, aperture control unit 123, shutter control unit 124, flash control unit 125, and shooting mode setting unit 1 connected to the signal processing unit 190.
60 and input / output interface (I / F) unit 170
An image synthesis processing unit 172 to which the output of the I / F unit 170 is supplied, and a display unit 173 to which the output of the image synthesis processing unit 172 is supplied. In addition, the electronic camera 100
Includes a flash 109 controlled by the flash controller 125 and a release button detector 150. Further, the electronic camera 100 includes a position sensor (not shown), and the output of the position sensor is supplied to the signal processing unit 190. The output of the signal processing unit 190 is the AGC circuit 105.
And the focus detection unit 142 is also supplied to the image memory 130.
Each output of the exposure detection unit 143 is the signal processing unit 19.
It is designed to be supplied to 0. The output of the image memory 130 is supplied to the I / F unit 170, and the output of the release button detection unit 150 is supplied to the signal processing unit 190.

Here, the electronic camera 100 is provided with an angle detecting section 126 connected to the signal processing unit 190 in addition to the above-mentioned respective constituents.

The angle detection unit 126 uses a gyro or the like, and detects the posture of the apparatus generated as the electronic camera 100 is moved at the time of photographing as an angle component. That is, as shown in FIG. 2, the angle detector 126 includes the angular velocity sensor 126a and the angular velocity sensor 12a.
A / D converter 126b to which the output of 6a is supplied, and A / D converter 126b
Angle calculator 126 to which the output of the D converter 126b is supplied
c, and the output of the angle calculator 126c is supplied to the signal processing unit 190.

The angle sensor 126a uses the A / D converter 1 to output the output signal V according to the angle change generated by the movement of the device.
26b. The level of the output signal V is proportional to the angular velocity. The A / D converter 126b digitizes the output signal V from the angle sensor 126a and supplies it to the angle calculator 126c as digital data D. The angle calculator 126c obtains an average level Davg by integrating the digital data D from the A / D converter 126b for 1 s, for example, and calculates the average level Davg.
An angle signal is obtained by converting vg into an angle component. Then, the angle signal obtained by the angle calculator 126c is supplied to the signal processing unit 190.

Therefore, the electronic camera 100 is adapted to generate a panoramic image by using the angle signal obtained by the angle detecting unit 126 as described above together with the information regarding the photographing condition at the time of photographing. .

Therefore, first, panoramic photography will be described.

The electronic camera 100 includes a photographing mode setting section 1
By operating 60, a shooting mode such as a normal shooting mode and a panoramic shooting mode can be set. For example, the panorama shooting mode is set, and a long-distance landscape 20 as shown in FIG. Unlike in the case of shooting a subject at a short distance, the shooting area of the camera is almost unchanged even when the electronic camera 100 is translated vertically and horizontally. Therefore, as shown in FIG. 3, first, the electronic camera 100 is installed in the position P21 (state 1).
In the coordinate system XYZ, the rotation angles around each coordinate axis are “Ψ”, “Φ”, and “θ”, and the rotation around the Y axis (pan) or the rotation around the X axis (tilt) is performed. A region R21 of the landscape 20 is photographed. Also,
Even in the state where the electronic camera 100 is installed at the positions P22 and P23 (states 2 and 3), the regions R22 and R23 of the landscape 20 are photographed by performing the pan or tilt operation as in the case of the position P21. . At this time, the regions R21 and R22 are photographed so as to partially overlap each other, and the regions R22 and R23 are photographed so as to partially overlap each other.

Therefore, when the image is taken at the position P21, the image I21 shown in FIG. 4 is obtained, and when the image is taken at the position P22, the image I22 shown in FIG.
When shooting at 23, an image I23 as shown in the figure is obtained. Then, the three images I21, I22, I23 are
As a result of being synthesized by the image synthesis processing unit 172 using the angle signal obtained by the angle detection unit 126 described above, FIG.
A panoramic image I24 in which the joints (dotted line portions) of the respective images are continuous is obtained as shown in FIG.

Next, the operation of the electronic camera 100 will be specifically described.

First, the subject image is projected by the image pickup lens 101 through the diaphragm 102 onto the light receiving surface of the image pickup element 103. At this time, the zoom position and the focus position of the imaging lens 101 are controlled by the zoom control unit 121 and the focus control unit 122, which are connected to the controller 120, respectively. The aperture amount of the aperture 102 is also controlled by the aperture controller 123 connected to the controller 120.

The image sensor 103 is a CCD (Charge).
d Coupled Device) and the like, which converts the received subject image into an electric signal and supplies it to the amplifier 104. The amplifier 104 amplifies an electric signal (hereinafter, referred to as a video signal) from the image sensor 103 to amplify the AGC circuit 105.
To supply. The AGC circuit 105 is the signal processing unit 1
Based on the control signal from 90, the video signal from the amplifier 104 is amplified or attenuated and supplied to the A / D converter 106. The A / D converter 106 digitizes the video signal from the AGC circuit 105 and supplies it to the video signal processing unit 107 as image data. At this time, the signal processing unit 1
A control signal 90 detects the signal level of the image data supplied to the video signal processing unit 107, and when the detected signal level is lower than a predetermined level, the control signal for increasing the gain given to the video signal by the AGC circuit 105. Is generated and supplied to the AGC circuit 105, and when the detected signal level is higher than a predetermined level, the AGC circuit 105 generates a control signal for lowering the gain given to the video signal.
05. As a result, the video signal output from the AGC circuit 105 becomes a signal having a predetermined level width suitable for the signal processing performed by the video signal processing unit 107.

The video signal processing unit 107 includes an A / D converter 1
The image data from 06 is subjected to predetermined signal processing, stored in the image memory 130, and supplied to the white balance detection unit 141, the focus detection unit 142, and the exposure detection unit 143. The white balance detection unit 141 detects the white balance state of the image data from the video signal processing unit 107, and supplies the detection result to the video signal processing unit 107. The focus detection unit 142 detects the focus of the imaging lens 101 from the image data from the video signal processing unit 107,
The detection result is supplied to the signal processing unit 190. The exposure detection unit 143 detects the exposure amount in the image sensor 103 from the image data from the video signal processing unit 107, and supplies the detection result to the signal processing unit 190.

The video signal processing unit 107 adjusts the color balance of the image data from the A / D converter 106 based on the detection result from the white balance detection unit 141. Therefore, the image memory 130 stores the image data whose color balance has been adjusted. The signal processing unit 190 includes a focus detection unit 142.
And a controller 1 for generating a control signal for setting a shooting condition based on each detection result from the exposure detection unit 143.
Supply to 20. Further, the signal processing unit 190 stores information regarding the shooting conditions in the image memory 130. The controller 120, based on the control signal from the signal processing unit 190, the zoom control unit 121, the focus control unit 122, the aperture control unit 123, and the shutter control unit 124.
And a control signal to the flash controller 125.

Therefore, the zoom control section 121, the focus control section 122, and the aperture control section 123 are respectively based on the control signal from the controller 120, the zoom position of the imaging lens 101, the focus position of the imaging lens 101, and the aperture 102. It is controlled so that the aperture amount of is in an appropriate state.

As described above, the shooting conditions in the electronic camera 100 are set appropriately.

Next, the photographer selects the photographing mode setting section 160.
By operating, the shooting mode is set to, for example, the panoramic shooting mode, and shooting is started. In addition, the photographer operates a release button (not shown) to instruct the setting (lock) of the photographing condition or the execution of the photographing.

The photographing mode setting section 160 detects which photographing mode is set by the photographer's operation and supplies the detection signal to the signal processing unit 190. The release button detection unit 150 detects whether the release button is pressed down to the first stroke position or the second stroke position and supplies the detection signal to the signal processing unit 190.

The angle detection unit 126, as described above,
An angle signal based on an angle change generated by the movement of the electronic camera 100 is generated and supplied to the signal processing unit 190. A position sensor (not shown) detects the three-dimensional position, azimuth, etc. of the electronic camera 100 and supplies the detection signal to the signal processing unit 190.

The signal processing unit 190 generates a control signal according to the set shooting mode according to the detection signal from the shooting mode setting section 160 and supplies it to the controller 120. In addition, when the signal processing unit 190 determines from the detection signal from the release button detection unit 150 that the button has been pushed down to the first stroke position, the signal processing unit 190 generates a control signal to lock the shooting condition and causes the controller 120 to do so. Supply. At the same time, the signal processing unit 1
Reference numeral 90 denotes a detection signal of the position sensor and the angle detection unit 1
Reset the angle signal from 26. On the other hand, if it is determined that the signal processing unit 190 is pushed down to the second stroke position, the signal processing unit 190 generates a control signal for performing the shutter operation and supplies the control signal to the controller 120.

The controller 120 controls the zoom controller 12 based on the control signal from the signal processing unit 190.
1, the focus control unit 122, the aperture control unit 123, the shutter control unit 124, and the flash control unit 125, and supplies the control signals to the shutter control unit 124 and the flash control unit 125.

Therefore, the zoom position of the image pickup lens 101, the focus position of the image pickup lens 101, and the diaphragm 10.
The aperture amount of 2 is in a state according to the operation of the photographer. Further, the shutter control unit 124 controls the shutter 108 based on the control signal from the controller 120, so that the shutter 108 is controlled to the shutter speed according to the operation of the photographer, and the flash control unit 125 controls from the controller 120. By controlling the flash 109 based on the signal, the flash 1 can be operated according to the operation of the photographer.
The ON / OFF operation of 09 is controlled.

When shooting is started as described above, the image data output from the video signal processing unit 107 is stored in the image memory 130. At this time, the signal processing unit 190, on the basis of the detection signal from the release button detection unit 150, information regarding the shooting conditions such as the focus distance, the focal length, the identification information of the shooting mode, the detection signal of the position sensor and the angle detection described above. Information of the angle signal from the unit 126 is stored as additional information in the image memory 130 in association with the image data output from the video signal processing unit 107.

The image composition processing unit 172 is connected to the image memory 13
The data stored in 0 is read out via the I / F unit 170 to generate a panoramic image.

More specifically, as shown in FIG. 6, the image synthesizing processing unit 172 is configured so that the I / F circuit 1 of FIG.
Image data from the input / output (I / O) circuit 172a
Image information separating unit 172f supplied via the controller 17 and the controller 17 to which the output of the image information separating unit 172f is supplied.
2e and the image memory 172g, the corresponding point detection unit 172b to which the output of the image memory 172g is supplied, and the parameter extraction unit 172 to which the output of the corresponding point detection unit 172b is supplied.
d, a coordinate conversion processing unit 172h to which the output of the parameter extraction unit 172d is supplied, and a combined image memory 172i to which the output of the coordinate conversion processing unit 172h is supplied. The output of the combined image memory 172i is I / O circuit 17
It is configured to be supplied to the display unit 173 of FIG. 1 via 2a. Further, the image composition processing unit 172 receives the output of the controller 172e, and the overlapping region prediction unit 1
72c, and the output of the overlapping area prediction unit 172c is supplied to the corresponding point detection unit 172b. Then, the controller 172e controls the image memory 17
2g is connected, and the output of the image memory 172g is
The coordinate conversion processing unit 172h is also supplied.

In the image composition processing section 172 as described above, first, the image information separating section 172f is operated by the I / O circuit 17.
The data from 2a, that is, the data consisting of the image data stored in the image memory 130 and the additional information is separated, the image data is stored in the image memory 172g, and the additional information is supplied to the controller 172e.

The controller 172e controls each unit by the additional information from the image information separating unit 172f. In addition, the controller 172e sequentially reads out image data corresponding to a series of a plurality of images obtained by panoramic photography from the image memory 172g based on the photography mode identification information included in the additional information, and the corresponding point detection unit 172.
b and the coordinate conversion processing unit 172h. Further, the controller 172e supplies the overlap region prediction unit 172c with information on the detection signal such as the focal length and the three-dimensional position and the angle signal included in the additional information.

The overlap area prediction unit 172c obtains the overlap area of each image by decoding the information of the detection signal such as the focal length and the three-dimensional position and the angle signal from the controller 172e.

Here, the processing for obtaining the overlapping area will be specifically described. For example, as shown in FIG. 7, when panoramic image capturing is performed by panning around the origin O, the information of the three-dimensional position and the angle signal is as follows. , Information about only the rotation “Φ” around the Y axis shown in FIG. Therefore,
In FIG. 7, the pan angle (angle formed by the optical axes 31a and 31b before and after the panning operation) is “Φ”, the angle of view of the electronic camera 100 is “2α”, and the focal length (from the origin O to the sensor surfaces 32a and 32b). The overlapping angle β is obtained by the equation (1): β = 2α−β (1). In addition, each sensor surface 32
Assuming that each size of a and 32b is “H (mm) × V (mm)”, each area dx on the sensor surfaces 32a and 32b.
Is obtained by the equation (2): dx = f · (tan (α) −tan (α−β)) (2) From these equations (1) and (2), the overlapping region S of the images 33a and 33b is
The images 33a and 33 read from the image memory 172g
When each size of b is “nx × ny”, S = nx · dx / H (3) is obtained by the equation (3).

Therefore, the overlapping area prediction unit 172c
The overlapping area of each image is obtained using Expressions (1) to (3). That is, the overlapping area is predicted based on the information of the detection signal such as the three-dimensional position and the angle signal. Then, the overlapping area prediction unit 172c supplies information on the predicted overlapping area to the corresponding point detection unit 172b.

The corresponding point detector 172b is shown in FIG.
As shown in the image 33a from the image memory 172a.
Information of the overlapping area from the overlapping area prediction unit 172c
From the indicated area (overlapping area) S, an arbitrary "m × m" size
Cut out the template T. Then, the corresponding point detection unit 17
2b is next to the cut-out template T and the image 33a.
A test is performed between the matching image 33b and the image in the search area S '.
By performing the calculation using the template matching method,
Determine the position corresponding to the template T from the search area S '
Set. The size of the search area S'is equal to that of the overlapping area S.
Have the same size. Therefore, as shown in FIG.
Calculation processing by the template matching method as described above
, Each template T included in the overlapping area S₁,
T_Two, ..., T _nBy performing sequentially for
Corresponding points as indicated by arrows in the figure are detected. Soshi
Information of the corresponding points detected by the corresponding point detecting unit 172b.
Is supplied to the parameter extraction unit 172d.

The parameter extracting unit 172d extracts a parameter for the combining process based on the information of the corresponding points from the corresponding point detecting unit 172b, and supplies the parameter to the coordinate conversion processing unit 172h.

The coordinate conversion processing section 172h uses the parameters from the parameter extraction section 172d to generate a panoramic image by performing coordinate conversion processing such as affine transformation on a series of a plurality of image data read by the controller 172e. Then, the panoramic image is written in the composite image memory 172i.

The panoramic image written in the composite image memory 172i is transmitted through the I / F circuit 172a as shown in FIG.
Are displayed on the screen by the display unit 173.

As described above, the electronic camera 100 predicts the overlapping area of each image based on the three-dimensional position and the angle component of the apparatus, detects the correspondence of each image in the predicted overlapping area, and performs the combining process. Therefore, it is possible to obtain a natural panoramic image with continuous seams.
Further, since the electronic camera 100 is configured to store the information of the angle component in the image memory 130 in association with each image, the data of each image is detected when the correspondence relationship in the overlapping area of each image is detected. At the same time, information on the angle component corresponding to each image can be instantaneously obtained. For this reason,
It is possible to reduce the occurrence of erroneous correspondence and to significantly shorten the detection processing time, as compared with the case where the correspondence in the overlapping area is detected only by the image data.

In the electronic camera 100 described above, the electronic camera 10 is provided by providing the angular velocity sensor 126a.
Although the angular component generated by the movement of 0 is detected, the electronic camera 1 is provided by providing an acceleration sensor or the like.
The translational component generated by the movement of 00 may be detected. Although the panoramic image obtained by the image composition processing unit 172 is displayed on the screen of the display unit 173, it may be stored in a recording medium such as a hard disk. Further, in the image synthesizing processing unit 172, the image memory 172g for the images to be synthesized and the synthetic image memory 172h for the panoramic image are respectively provided.
One image memory may be shared for an image to be combined and for a panoramic image. In addition, even when a tilting operation is performed during panoramic shooting, high-quality images can be obtained by predicting the overlapping area of each image based on the three-dimensional position and angular components of the device, as in the case of shooting by the panning operation described above. It goes without saying that it is possible to generate a panoramic image of.

Next, a second embodiment of the present invention will be described with reference to the drawings.

The electronic camera system according to the present invention is applied to, for example, an electronic camera system 200 as shown in FIG.

This electronic camera system (hereinafter, simply referred to as an electronic camera) 200 is the electronic camera 100 shown in FIG.
The overlap area calculation unit 201 is provided in the. And
The output of the video signal processing circuit 107 is supplied to the overlapping area calculating unit 201, and the output of the overlapping area calculating unit 201 is supplied to the release button detecting unit 150.

In the electronic camera 200 shown in FIG. 10, parts that operate similarly to those of the electronic camera 100 shown in FIG. 1 are designated by the same reference numerals, and detailed description thereof will be omitted.

Here, as shown in FIG. 3 above, the long-distance landscape 20 is photographed by the panning operation so that a part of each image overlaps, and based on the image information of the overlapping regions 25 and 26. To generate a panoramic image,
The overlapping areas 25 and 26 need to be set appropriately.

Therefore, in the electronic camera 200, the overlapping area calculation unit 201 sequentially detects the overlapping areas each time an image is input.

For example, as shown in FIG. 11, after the image F ₁ of the first frame is input to the video signal processing unit 107 by the panning operation, the image F _{2 of} the second frame,
..., the image F _n of the n-th frame, the (n + 1) image F _{n + 1} of the frame, the (n + 2) image F _{n + 2} of the frame are sequentially inputted. Then, the image data of each frame subjected to the predetermined signal processing in the video signal processing unit 107 is sequentially supplied to the overlapping region calculation unit 201.

The overlap area calculation unit 201 receives, for example, the first data every time the image data is supplied from the video signal processing unit 107.
Overlapping area 40a of the image F _n of the n-th frame image F ₁ of the frame, the image F ₁ of the first frame (n + 1) th frame image F _{n + 1} overlap region 40b, the image F ₁ of the first frame The overlapping area 40c of the image F _{n + 2} of the (n + 2) th frame is sequentially obtained. That is, the overlap area calculation unit 201
Is the same as the detection processing of the overlapping area in the electronic camera 100 of FIG. 1, the panning angle is obtained by using the information such as the three-dimensional position and the angle component of the electronic camera 200, and the above equations (1) and (2) ) To obtain the overlap area. Then, the overlapping area calculation unit 201 sequentially compares the obtained overlapping area with a preset threshold value k, and when the overlapping area and the threshold value k become substantially equal, the release button detecting unit 150 Provides a shutter control signal. Therefore, in FIG. 11 above, the image F of the (n + 2) th frame
_A shutter control signal is supplied to the release button detector 150 when _{n + 2} is input.

When the shutter control signal is supplied from the overlap area calculation unit 201, the release button detection unit 150 is a signal corresponding to a detection signal indicating that the shutter button has been pushed down to the second stroke position, that is, detection for instructing a shutter operation. The signal is supplied to the signal processing unit 190.

The signal processing unit 190 supplies a control signal to the shutter controller 124 based on the detection signal from the release button detector 150. As a result, the shutter control unit 124 controls the operation of the shutter 108 and the shutter is released.

As described above, in the electronic camera 200, similarly to the detection processing of the overlapping area in the electronic camera 100 of FIG. 1, the overlapping areas of the sequentially input images are sequentially obtained, and the obtained overlapping area is Since the shutter operation is automatically performed when the threshold value is substantially equal to the preset threshold value, it is possible to obtain a series of a plurality of images with an appropriate degree of overlap for generating a panoramic image. Therefore, it is possible to obtain a natural panoramic image with continuous seams without failing in shooting.

Next, a third embodiment of the present invention will be described with reference to the drawings.

The electronic camera system according to the present invention is applied to, for example, an electronic camera system including an image synthesis processing section 300 as shown in FIG.

This electronic camera system (hereinafter, simply referred to as an electronic camera) has an image composition processing unit 300 of FIG. 12 instead of the image composition processing unit 172 of FIG. 6 provided in the electronic camera 100 of FIG. Is provided. Further, in addition to the constituent requirements of the electronic camera 100 shown in FIG. 1, the electronic camera has an angular velocity sensor for detecting a translational component generated when the electronic camera moves as described above, although not shown. The detector is provided.

Since the configuration and operation of the image composition processing unit 300 are the same as those of the electronic camera 100 shown in FIG. 1, detailed description of each unit other than the image composition processing unit 300 will be omitted. In addition, the image composition processing unit 3 of FIG.
00, the same operation as that of the image composition processing unit 172 of FIG. 6 is denoted by the same reference numeral, and the detailed description thereof will be omitted.

Here, for example, as shown in FIG. 13, when the electronic camera 100 takes a panoramic image of an original 40 at a short distance, the electronic camera 100 is installed at a position P41 and the original 4 is placed.
The region R41 of 0 is photographed, and the electronic camera 100 is moved to the position P4.
The region R42 of the document 10 is photographed by performing translational movement (translations Δx and Δy in the vertical and horizontal directions) from 1 to the position P12. On the other hand, when shooting a long-distance subject, since the shooting area hardly changes even if the electronic camera 100 is translated vertically and horizontally, as shown in FIG. 3, the landscape 20 or the like is panned as shown in FIG. Take a picture. In this panning photography, for example, when an image of the central region R22 of the landscape 20 is used as a reference, trapezoidal distortion occurs in the images of the regions R21 and R23 at both ends thereof. Therefore, when composing each image, it is necessary to consider the generated distortion of the image.

However, since the image distortion caused by the panning photographing as described above is not generally taken into consideration in the image synthesizing process at the time of the translational photographing, it is the same for the panning photographing and the translational photographing. When the image combining process is performed, the image is deteriorated.

Therefore, in this electronic camera, the image synthesizing unit 300 predicts the overlapping area of each image to be synthesized, and performs the optimum image synthesizing process for each of the photographing by panning and the photographing by translation. Has been done.

That is, as shown in FIG. 12, the image synthesizing processor 300 has the image synthesizing processor 172 shown in FIG.
And a selector 304 to which each output of the controller 172e and the image memory 172g is supplied,
The output of the selector 304 is supplied to the spherical mapping conversion processing unit 302, and the corresponding point detection unit 172b is directly supplied with the output of the selector 304 and the output of the selector 304 is converted into the spherical mapping conversion processing unit 302. It is designed to be supplied via.

The operation of the image composition processing section 300 will be described below.

First, the controller 172a obtains the panning angle by extracting the information of the three-dimensional position and the angle component included in the additional information from the image information separating unit 172f. Then, the controller 172a compares the panning angle with a preset threshold value, and if the panning angle is larger than the threshold value, it is determined that the image was taken by the panning as shown in FIG. It discriminates and supplies the discrimination signal to the selector 304. Further, when the controller 172a determines that the image is captured by panning, it also supplies the focal length information included in the additional information to the selector 304. Further, the controller 172a includes an image information separation unit 172.
Information on the focus position (subject distance), the focal length, the three-dimensional position, and the angle component included in the additional information from f is supplied to the overlap region prediction unit 301.

The selector 304 is the controller 172a.
When the image is captured by panning, the series of plural image data written in the image memory 172g is read out and supplied to the spherical mapping conversion processing unit 302, and the focal length information from the controller 172a is also output. It is supplied to the spherical mapping conversion processing unit 302. On the other hand, in the case of shooting by translational movement, the selector 304 directly supplies the series of plural image data read from the image memory 172g to the corresponding point detecting unit 172b.

The sphere mapping conversion processing unit 302 includes a selector 3
Spherical mapping conversion processing is performed on each image data from 04.

More specifically, first, in the spherical mapping conversion processing, as shown in FIG. 14, an image of the principal point O of the taking lens 101 of FIG. It is a process of generating a spherical image I52 by projecting I51 onto the spherical surface 50.

Therefore, as shown in FIG. 15, the image data supplied to the spherical mapping conversion processing unit 302 is obtained by, for example, panning an image I51a photographed at a long distance at an arbitrary position and an arbitrary angle. In the case of the image I51b, the spherical mapping conversion processing unit 302 uses the information on the focal length from the selector 304 to set the image I51a as the focal length f1.
Is projected onto the spherical surface 50 to generate a spherical image I52a, and the image I51b is taken as the spherical surface 5 as the focal length f1.
A spherical image I52b is generated by projecting it on 0.

Therefore, when the focal lengths f1 are the same and there is no rotation around the optical axis, the spherical mapping conversion processing unit 30
The spherical image I52a and the spherical image I52b obtained in 2 are
Since they are continuous on the spherical surface 50, the parameters used for coordinate conversion of these spherical images can be parameters for only the vertical and horizontal translations Δx and Δy as shown in FIG. However, in reality, there are errors in the focal length f1 and the rotation θ around the optical axis, and so on.
And Δy, the focal length f1, and the rotation θ around the optical axis are used as the above parameters. Accordingly, the same parameters as those used for the image capturing by the translational movement can be used for the coordinate conversion process during the image capturing by the panning.

As described above, in the case of an image taken by panning, the spherical mapping conversion processing unit 30
2 performs spherical mapping conversion processing, and a series of a plurality of spherical image data obtained by the processing is used as the corresponding point detection unit 172b.
Is supplied to. In the case of an image photographed by translational movement, a series of a plurality of image data written in the image memory 172g is the corresponding point detection unit 172b.
Is supplied to.

On the other hand, the overlapping area predicting section 301 predicts the overlapping area of each image using the information on the object distance, the focal length, the three-dimensional position and the angle component from the controller 172e.

For example, assuming that the positional relationship of each image can be roughly represented by vertical and horizontal translations Δx and Δy, FIG.
6, the subject distance is “L” and the translation amount is “Δ”.
x ", the width of the sensor surface is" h ", and the focal length is" f ", the overlapping region S can be obtained by the following equation (5): S = hf ?? x / L (5) it can.

Therefore, the overlapping area prediction unit 301 uses the above (5) to convert the overlapping area S into the size of the image data written in the image memory 172g, predict the overlapping area, and obtain the result. Information on the overlapping area is supplied to the corresponding point detecting unit 172b.

The corresponding point detection unit 172b, based on the overlap area information from the overlap area prediction unit 301, selects the selector 30.
4 or the corresponding point in the overlapping area of each image from the spherical mapping conversion processing unit 302 is detected and supplied to the parameter extraction unit 303.

The parameter extracting unit 303 extracts a parameter for the combining process based on the information of the corresponding points from the corresponding point detecting unit 172b, and supplies the parameter to the coordinate conversion processing unit 172h.

The coordinate conversion processing section 172h uses the parameters from the parameter extraction section 172d to generate a panoramic image by performing coordinate conversion processing such as affine transformation on a series of a plurality of image data read by the controller 172e. Then, the panoramic image is written in the composite image memory 172i.

As described above, in this electronic camera, the spherical mapping conversion process is performed on the image determined to be the image obtained by panning. The same image composition processing can be performed regardless of the shooting situation such as shooting.
That is, it is possible to always perform an appropriate image synthesizing process regardless of the shooting condition. Therefore, it is possible to always obtain a high quality panoramic image regardless of whether the panning is performed or the moving translation is performed. Further, since the panoramic image is generated by predicting the overlapping area of each image based on the information of the three-dimensional position and the angle component, the processing time can be shortened and the panoramic image can be generated with high accuracy. Can be generated.

In the electronic camera described above, the image composition processing section 300 is provided instead of the image composition processing section 172 provided in the electronic camera 100 shown in FIG.
It may be provided instead of the image composition processing unit 172 provided in the electronic camera 200 of FIG.

As described above, according to the present invention, it is possible to store information on the shooting condition at the time of shooting held together with the image obtained by shooting and the moving state of the device generated by the movement of the device at the time of shooting. It is configured so that the overlap area between each image can be predicted instantly based on the information, and the correspondence relationship of each image in the overlap area can be obtained. The processing time can be remarkably shortened as compared with the case of obtaining the correspondence in the overlapping area of the images. Further, the occurrence of false detection of correspondence can be significantly reduced. Therefore, a high quality panoramic image can be obtained in a short time. That is, the performance of the device can be improved, and a high-quality panoramic image can be easily obtained. Further, according to the present invention, since the overlapping area between the images to be combined is predicted based on the shooting condition at the time of shooting and the information of the angle component generated by the movement of the device at the time of shooting, A high quality panoramic image can be obtained in time. Further, according to the present invention, since the angular component is configured to be obtained by the angular velocity sensor, a high quality panoramic image can be obtained in a short time. Further, according to the present invention, since the overlapping area between the images to be combined is predicted based on the shooting condition at the time of shooting and the information of the translational component generated by the movement of the device at the time of shooting, A high quality panoramic image can be obtained in time. Moreover, according to the present invention, since the translational component is obtained by the acceleration sensor, a high quality panoramic image can be obtained in a short time. Further, according to the present invention, based on the information on the position of the device generated by the movement of the device at the time of shooting, the overlapping area between the images to be combined is further instantly predicted, so that the time is further shortened. You can get high quality panoramic images. Further, according to the present invention, based on the information of the three-dimensional position of the device generated by the movement of the device at the time of shooting, the overlapping area between the images to be combined is configured to be more instantaneously predicted. It is possible to obtain a high quality panoramic image in a short time. Further, according to the present invention, it is suitable for sequentially obtaining an overlapping area of each image based on the information on the movement state of the device generated by the movement of the device at the time of image capturing and synthesizing the images. Since the apparatus is configured to automatically capture images at various overlapping degrees, the performance of the apparatus can be further enhanced. For example,
It is possible to prevent the failure of shooting and significantly improve the operability. According to the present invention,
A high quality panoramic image is always simplified by adopting a configuration that performs an appropriate image composition process for the shooting situation determined based on the information of the movement state of the device generated by the movement of the device at the time of shooting. Can be obtained. Further, according to the present invention, the generated panoramic image is externally output, so that the panoramic image can be given to an externally connected device.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an electronic camera system according to the present invention in a first embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of an angle detection unit of the electronic camera system.

FIG. 3 is a diagram for explaining a case where panning is performed to panoramicly photograph a long-distance subject.

FIG. 4 is a diagram for explaining a series of images obtained by the panoramic photography.

FIG. 5 is a diagram for explaining a panoramic image obtained by synthesizing a series of a plurality of images.

FIG. 6 is a block diagram showing a configuration of an image composition processing unit of the electronic camera system.

FIG. 7 is a diagram for explaining a process of predicting an overlapping area in the image synthesis processing unit.

FIG. 8 is a diagram illustrating a process of detecting corresponding points in an overlapping area by the image synthesis processing unit.

FIG. 9 is a diagram for explaining corresponding points obtained by detecting the corresponding points.

FIG. 10 is a block diagram showing a configuration of an electronic camera system according to the present invention in a second embodiment of the present invention.

FIG. 11 is a diagram for explaining a process of predicting an overlapping area in the electronic camera system.

FIG. 12 is a block diagram showing a configuration of an image composition processing unit of the electronic camera system according to the third embodiment of the present invention.

FIG. 13 is a diagram for explaining a case where a panoramic image of a subject at a short distance is obtained by translational movement.

FIG. 14 is a diagram for explaining a spherical mapping.

FIG. 15 is a diagram for explaining a process of performing spherical mapping conversion in the image synthesis processing unit.

FIG. 16 is a diagram for explaining a process of predicting an overlapping area in the image synthesis processing unit.

[Explanation of symbols]

 100 electronic camera system 101 shooting lens 102 aperture 103 shutter 104 amplifier 105 automatic gain control circuit 106 A / D converter 107 video signal processing circuit 109 flash 120 controller 121 zoom control unit 122 focus control unit 123 aperture control unit 124 shutter control unit 126 Angle detection unit 130 Image memory 141 White balance detection unit 142 Focus detection unit 143 Exposure detection unit 150 Release button detection unit 155 Reset button 160 Shooting mode setting unit 170 I / F unit 172 Image composition unit 173 Display unit

Front page continuation (72) Inventor Hideo Takiguchi 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc.

Claims (10)

[Claims] 1. A panoramic image is generated by dividing an image of a subject into a plurality of screens by an image pickup means so as to overlap a part of the screens, and synthesizing a series of a plurality of images obtained by the image pickup means. An electronic camera system comprising: shooting condition detection means for detecting shooting conditions at the time of shooting; moving state detection means for detecting a moving state of the device at the time of shooting; and a plurality of images obtained by shooting Storage means for storing, as additional information, information on the shooting condition detected by the shooting condition detecting means and the moving state detected by the moving state detecting means, and based on the additional information stored in the storing means. An overlapping area prediction unit that predicts and detects an overlapping area of each image, and a corresponding point detection unit that detects a correspondence relationship between the images in the overlapping area using the overlapping area obtained by the overlapping area prediction unit. , Up An electronic camera comprising: an image synthesizing unit for synthesizing a series of a plurality of images stored in the storage unit to generate a panoramic image based on the information on the correspondence obtained by the corresponding point detecting unit. system. 2. The electronic camera system according to claim 1, wherein the movement state detecting means detects an angle component generated by movement of the device. 3. The electronic camera system according to claim 2, wherein the moving state detecting means detects the angular component by an angular velocity sensor. 4. The electronic camera system according to claim 1, wherein the moving state detecting means detects a translational component generated by the movement of the device. 5. The electronic camera system according to claim 4, wherein the moving state detecting means detects the translational component by an acceleration sensor. 6. A position detecting means for detecting the position of the apparatus at the time of photographing is provided, and the storage means stores the information of the position detected by the position detecting means in the additional information. The electronic camera system according to claim 1. 7. The electronic camera system according to claim 6, wherein the position detecting means detects a three-dimensional position of the device. 8. An overlapping area of each image, every time an image is input to the image pickup means, based on the information of the image pickup condition detected by the image pickup condition detection means and the movement state detected by the movement state detection means. The overlapping area detecting means for sequentially detecting the overlapping degree, the comparing means for comparing the overlapping degree obtained by the overlapping area detecting means with a predetermined value, and the photographing timing is controlled based on the comparison result of the comparing means. The electronic camera system according to claim 1, further comprising a control unit. 9. A spherical projection conversion means for projecting a series of a plurality of images stored in the storage means onto a spherical surface to generate a spherical projection image, and a series of a plurality of images stored in the storage means. And a switching means for selectively switching a plurality of spherical projection images obtained by the spherical projection conversion means to output to the corresponding point detecting means, and a shooting situation for detecting the shooting situation from the additional information stored in the storage means. 2. A detection means and a switching control means for controlling the switching operation of the switching means while controlling the operation of the spherical projection conversion means based on the detection result of the photographing condition detection means. Electronic camera system described. 10. The electronic camera system according to claim 1, further comprising output means for externally outputting the panoramic image obtained by the image synthesizing means.