JPH10155104A - Compound eye image pickup method and device and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To attain panorama image pickup display of a moving image and a stereoscopic image pickup display in response to an output form by adopting a plurality of synthesis methods which generate a synthesis image from two left/right images picked up by two image pickup optical devices placed to the left and the right sides synchronously with each other so as to select and switch a desired processing mode based on a plurality of synthesis methods. SOLUTION: In the case that a through-display mode is selected, two left/right images formed by lenses 8a, 8b in the compound eye image pickup device 1 are picked up by respective CCDs 9a, 9b and the images are given to a personal computer 2 and stored in a memory 13 as the images synchronously with each other. An image synthesis section 15 overlaps the two images based on a preset overlap amount and the result is displayed on a display device 3. In the recording mode, an image correction/overlap amount calculation section 16 calculates the overlapped amount of the two left/right images. Thus, a panorama synthesis image of a moving image with excellent image quality in response to the output form is obtained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a compound-eye imaging method and apparatus having a high-definition and high-definition moving image panoramic image pickup and display function or a moving image stereoscopic image pickup and display function, and a storage medium used in the compound eye image pickup method and apparatus.

[0002]

2. Description of the Related Art Conventionally, a panoramic imaging display method and a stereoscopic imaging display method using a compound-eye imaging device have been known. In the panoramic imaging display method, for example, two imaging optical systems arranged on the left and right are arranged and imaged using mirrors or the like so that the viewpoints of the images to be imaged coincide. In the imaging optical system, the imaging is performed such that the two captured images overlap each other. Then, the obtained two left and right images are overlapped by a certain amount of overlap to create one panoramic composite image, and displayed on an image output device such as a display.

On the other hand, in the stereoscopic imaging display method, two imaging optical systems are arranged in parallel at an interval given by a certain base length, and images are taken from two viewpoints.

On the other hand, the human left and right eyes have an average of 65
It is said that there is a distance of about mm, and it is standard that the baseline length of the two imaging optical systems is 65 mm in stereoscopic imaging display.

As described above, when a subject of interest is imaged from two viewpoints on the left and right, the positions of the objects in the images captured by the respective imaging systems are different from each other. That is, this is the parallax, and the user can view an image having a stereoscopic effect by performing stereoscopic viewing (stereoscopic viewing) of the parallax.

There are various methods for stereoscopically viewing an image obtained from two viewpoints on the left and right. One is to output the obtained left and right images alternately to one image area using a display (image output device). On the user side, an image with a three-dimensional effect can be viewed by viewing the image with liquid crystal shutter glasses that switch the left and right shutters in synchronization with the display switching of the left and right images.

In another display method, two left and right images are alternately arranged every other line in the horizontal direction in an area of one stereoscopic image prepared in advance to form a striped image composed of two left and right images. Create an image. The display screen has a polarizing plate whose polarization direction alternates every other line in the horizontal direction, similarly to the created stereoscopic image, and the image is displayed in a stripe shape with different polarization directions. Then, when the created striped stereoscopic image is displayed on a display, the image (right image) captured by the right imaging optical system is displayed by transmitting only polarized light in a certain direction, and the left imaging optical system. (Left image)
Is displayed by transmitting only polarized light different from the right image.

On the other hand, the user wears polarized glasses having a function of transmitting only the same polarized light as the image displayed on the display on the left and right sides, so that only the polarized light displaying the right image is displayed on the right eye. The left eye transmits only polarized light in which the left image is displayed. Using these polarized glasses, the user sees the right image only with the right eye and the left image only with the left eye, and the user can see an image with a three-dimensional effect.

Further, as described above, in the stereoscopic imaging display, the parallax of images captured from different viewpoints is used. That is, the user creates an image with a three-dimensional effect by superimposing two images having parallax on a subject of interest (hereinafter, referred to as a main subject), that is, by fusing.

[0010] Generally, when a user fuses images of two viewpoints on the left and right of a main subject and stereoscopically views the images, the user can select two images on the left and right.
The smaller the parallax of the main subject between the images, the easier the fusion of the main subject.

Therefore, when taking an image, it is necessary to dispose the imaging optical system so that the parallax of the main subject becomes small. Conventionally, this problem has been solved by (1) disposing the imaging optical system with a convergence angle and (2) moving the imaging optical system in parallel.

FIG. 23 is a view showing an example in which two imaging optical systems are arranged without having a convergence angle, that is, an example of arrangement of imaging optical systems for stereoscopic imaging by parallel vision. In the figure, 2
The two imaging optical systems 701a and 701b are arranged in parallel with each other at an interval given by the base line length 1 around the origin O1, and have lenses 702a and 702b and CCDs 703a and 703b as imaging elements, respectively. Lens 702a and CCD 703a, lens 702b and CCD 7
The interval of 03b is v. It is also assumed that the main subject 904 is located at a position A that is separated by z in the imaging direction from the origin O1.

In FIG. 23, a main subject 904 forms images at different positions on the surfaces of the left and right CCDs 703a and 703b. CCDs 703a and 703b at this time
Is referred to as a parallax d. That is, two imaging optical systems 701 arranged in parallel
In a and 701b, the main subject 904 is imaged with a certain parallax d. Therefore, the image pickup optical systems 701a and 701b are provided with a convergence angle so that the user can easily fuse the image, and the parallax d of the main subject 904 is reduced.

In FIG. 23, lenses 702a, 702
Angles O1AB and O1AC formed by the centers B and C of b, the position A where the main subject 904 as an object exists, and the origin O1 are obtained as θ by the following equation (1).

Θ = arctan l / 2z Equation (1) where z is the distance between the two imaging optical systems 701 a and 701 b and the main subject 904, and l is the two imaging optical systems 7
01a and 701b represent the base line length. Therefore, the two imaging optical systems 701a and 701b are connected to the respective lenses 702a and 702b.
With the centers B and C of 2b as centers of rotation, an angle θ
By rotating only both CCDs 703a, 7
The position of the main subject 904 to be imaged on 03b is set to the center of the image, and the parallax can be set to 0.

FIG. 24 shows an image pickup optical system 701a, 701a, 7b so that the parallax between two images of the main object 904 to be imaged is set to 0.
It is a figure which shows the example which arrange | positioned giving a convergence angle to 01b.

As described above, the imaging optical systems 701a and 701
By giving b a convergence angle, the imaging optical system 701
The parallax of the main subject 904 can be set to 0 as long as there is no physical restriction such as a collision between a and 701b.

On the other hand, in the method of arranging the imaging optical systems 701a and 701b by moving them in parallel, the imaging optical systems 701a and
The base line length l of 701b is shortened or the CCD 703a,
For example, there is a method of moving the 703b in parallel.

FIG. 25 is a diagram showing an arrangement example in which the base line length of the image pickup optical systems 701a and 701b is reduced from l to l '.
When the base line length of the imaging optical systems 701a and 701b is reduced in this way, the parallax of the captured image can be reduced.

FIG. 26 is a view showing an example in which the CCDs 703a and 703b in the imaging optical systems 701a and 701b are translated. As shown in the figure, the CCDs 703a, 703
Each of the imaging optical systems 701a and 701 for imaging by moving b in parallel
The parallax of the main subject 904 can be reduced in the two left and right images captured at 01b.

[0021]

However, in the above-described panoramic imaging display method and stereoscopic imaging display method using the conventional compound-eye imaging apparatus, there are only processing and display methods for still images, No treatment and labeling method has been established. In moving image display, processing and a display method that provide a high frame rate and provide a good image as a moving image are required, but have not yet been realized.

When a convergence angle is given to the above-mentioned conventional imaging optical system, the conjugate plane of the imaging plane changes.

In FIG. 24, the conjugate plane of the imaging plane is shifted from the conjugate plane 905a of the imaging plane when viewed in parallel from each imaging optical system 70.
1a, conjugate plane of the imaging plane imaged by 701b, here, conjugate plane 9 of the imaging plane by left imaging optical system 701a
05b to the conjugate plane 905c of the imaging plane by the right imaging optical system 701b. Due to this change in the conjugate plane of the imaging plane, distortion occurs in the peripheral portion except for the main subject 904 at the center of each image. This distortion increases as the convergence angle of the imaging optical systems 701a and 701b increases, and it becomes more difficult to perform stereoscopic vision. In order to perform better stereoscopic imaging display, there is a problem that the convergence angles provided to the imaging optical systems 701a and 701b are limited.

In the method of reducing the parallax of the main subject 904 by arranging the imaging optical systems 701a and 701b in parallel movement, first, when the base line length is reduced,
An image without parallax is obtained not only for the main subject 904 but also for the entire image, and there is a problem that a sufficient three-dimensional effect cannot be obtained.

In the method of moving an image pickup device such as a CCD in parallel, it is required to control the image pickup device with high accuracy.
Therefore, when using the left and right images having a large parallax, there is a problem that the amount of movement is large and the control is difficult in order to move the image sensor in parallel to reduce the parallax to zero.

The present invention has been made in view of the above-mentioned problems of the prior art, and a first object of the present invention is to provide panoramic imaging and display of a moving image according to an output mode and stereoscopic viewing. An object of the present invention is to provide a compound eye imaging method and apparatus capable of performing image display.

It is a second object of the present invention to provide a compound eye imaging method and apparatus capable of obtaining a stereoscopic image which facilitates fusion of a main subject.

A third object of the present invention is to provide a compound eye imaging method and apparatus capable of easily controlling an imaging optical system by manual operation.

A fourth object of the present invention is to provide a storage medium capable of smoothly controlling the compound-eye imaging device.

[0030]

According to a first aspect of the present invention, there is provided a compound-eye imaging method according to the first aspect, wherein one image is synthesized from two left and right images taken by two imaging optical systems arranged on the left and right. It is characterized by comprising a combining step having a plurality of combining methods for creating an image, and a switching step of switching between the plurality of combining methods.

In order to achieve the first object, the compound eye imaging method according to claim 2 is the compound eye imaging method according to claim 1, wherein the first of the plurality of synthesizing methods has a synthesizing speed. A combining method 1 is a combining method in which combining is performed preferentially, and a second method among the plurality of combining methods is a combining method 2 in which the image quality of a combined image is combined with priority.

In order to achieve the first object, a compound eye imaging method according to claim 3 is the compound eye imaging method according to claim 2, wherein the synthesizing method 1 comprises the steps of: The combining method 2 is performed by giving an amount of overlap, and the left and right images captured are corrected for left and right differences in luminance and color information and trapezoidal distortion, and an overlap area between the two images is detected. And combining using the overlap amount calculated from this.

According to a fourth aspect of the present invention, there is provided a compound-eye imaging method according to the second or third aspect, wherein the switching step is performed in the through-display mode. In the recording and reproducing mode, the method is switched to the synthesizing method 2.

According to a fifth aspect of the present invention, there is provided a compound-eye imaging method according to the first, second or third aspect, wherein the combined image is a panoramic combined image. It is characterized by.

According to a sixth aspect of the present invention, there is provided a compound-eye imaging method according to the first, second or third aspect, wherein the composite image is a stereoscopic image. It is characterized by.

In order to achieve the first object, the compound-eye imaging apparatus according to claim 7 forms one composite image from two left and right images taken by two imaging optical systems arranged on the left and right. It is characterized by comprising a combining means having a plurality of combining methods, and a switching means for switching between the plurality of combining methods.

In order to achieve the first object, the compound-eye imaging apparatus according to claim 8 is the compound-eye imaging apparatus according to claim 7, wherein the first of the plurality of combining methods includes a combining speed. A combining method 1 is a combining method in which combining is performed preferentially, and a second method among the plurality of combining methods is a combining method 2 in which the image quality of a combined image is combined with priority.

According to a ninth aspect of the present invention, there is provided a compound-eye imaging apparatus according to the ninth aspect, wherein the synthesizing method 1 comprises the steps of: The combining method 2 is performed by giving an amount of overlap, and the left and right images captured are corrected for left and right differences in luminance and color information and trapezoidal distortion, and an overlap area between the two images is detected. And combining using the overlap amount calculated from this.

According to a tenth aspect of the present invention, there is provided a compound-eye imaging apparatus according to the seventh or eighth aspect, wherein the switching means comprises the first synthesizing method in the through display mode. In the recording and reproducing mode, the method is switched to the synthesizing method 2.

In order to achieve the first object, the compound-eye imaging apparatus according to the eleventh aspect is characterized in that the compound-eye imaging apparatus according to the seventh, eighth or ninth aspect
In the compound eye imaging apparatus according to the aspect, the composite image is a panoramic composite image.

According to a twelfth aspect of the present invention, there is provided a compound-eye imaging apparatus according to the seventh or eighth aspect.
In the compound eye imaging apparatus according to the aspect, the composite image is a stereoscopic image.

In order to achieve the second object, the compound-eye imaging method according to claim 13 is a compound-eye imaging method in which a pair of images having parallax is captured by two imaging optical systems. And a control step of controlling to adjust the parallax of the main subject selected from within.

In order to achieve the second object, a compound-eye imaging method according to claim 14 is the compound-eye imaging method according to claim 13, wherein the control step includes the step of: Is set.

According to a fifteenth aspect of the present invention, there is provided a compound-eye imaging method according to the thirteenth aspect, wherein the control step includes the step of: setting a limit value of a convergence angle of the imaging optical system. In the above, the parallax of the main subject in the image is adjusted by moving the image to be displayed in parallel.

Further, in order to achieve the second object, in the compound-eye imaging method according to claim 16, in the compound-eye imaging method according to claim 15, the parallel movement includes parallel movement of the imaging optical system. Refers to, the control step is to adjust the parallax of the main subject in the image by shortening the base line length of the imaging optical system when the imaging angle is equal to or greater than the limit value of the convergence angle of the imaging optical system. And

According to a seventeenth aspect of the present invention, in order to achieve the second object, in the multi-eye imaging method according to the fifteenth aspect, the parallel movement is performed by using an imaging element in the imaging optical system. Refers to parallel movement, and the control step includes, when the convergence angle of the imaging optical system is equal to or greater than a limit value, moving the imaging device of the imaging optical system in parallel away from the center of the two imaging optical systems. Thus, the parallax of the main subject in the image is adjusted.

In order to achieve the second object, the compound-eye imaging method according to claim 18 is the compound-eye imaging method according to claim 15, wherein the translation includes translating the captured left and right images. The control step,
When the convergence angle of the imaging optical system is equal to or more than the limit value, the parallax of the main subject in the image is adjusted by moving the captured left and right images in parallel to create a stereoscopically captured display image. An imaging method according to claim 16.

In order to achieve the third object, the compound-eye imaging method according to claim 19 is the compound-eye imaging method according to claim 15, wherein a pair of images having parallax between two imaging optical systems is used. In the compound eye imaging method for imaging,
Setting a limit value of the convergence angle of the imaging optical system, controlling the imaging optical system by the convergence angle or the parallel movement below the limit value, and controlling the imaging optical system by the parallel movement above the limit value. It has a control step.

According to another aspect of the present invention, there is provided a compound-eye image pickup apparatus for picking up a pair of images having parallax by two image pickup optical systems. It is characterized by having control means for controlling so as to adjust the parallax of the main subject selected from inside.

According to a twenty-first aspect of the present invention, in order to achieve the second object, the compound-eye imaging apparatus according to the twentieth aspect, wherein the control means includes a limit value of a convergence angle of the imaging optical system. Is set.

According to a twenty-second aspect of the present invention, there is provided a compound-eye imaging apparatus according to the twenty-second aspect, wherein the control means comprises a limit value of a convergence angle of the imaging optical system. In the above, the parallax of the main subject in the image is adjusted by moving the image to be displayed in parallel.

According to a twenty-third aspect of the present invention, in order to achieve the second object, in the multiple-eye imaging apparatus according to the twenty-second aspect, the translation means that the imaging optical system is translated. The control means adjusts the parallax of the main subject in the image by shortening the base line length of the imaging optical system and imaging the image when the convergence angle of the imaging optical system is equal to or more than the limit value. And

According to a twenty-fourth aspect of the present invention, in order to achieve the second object, in the multi-eye imaging apparatus according to the twenty-second aspect, the parallel movement means that an imaging element in the imaging optical system is used. Refers to parallel movement, and when the convergence angle of the imaging optical system is equal to or more than a limit value, the control unit moves the imaging device of the imaging optical system in parallel away from the center of the two imaging optical systems to perform imaging. Thus, the parallax of the main subject in the image is adjusted.

According to a twenty-fifth aspect of the present invention, in order to achieve the second object, in the compound eye imaging apparatus according to the twenty-second aspect, the translation means that the captured left and right images are translated. The control means, when the convergence angle of the imaging optical system is equal to or more than the limit value, creates a stereoscopically captured display image by translating the captured left and right images in parallel, thereby obtaining a main subject in the image. It is characterized by adjusting parallax.

According to another aspect of the present invention, there is provided a compound-eye image pickup apparatus for picking up a set of images having parallax by two image pickup optical systems. Limit value setting means for setting a limit value of a convergence angle of the system, and controls the imaging optical system by the convergence angle or the parallel movement below the limit value set by the limit value setting means; and Control means for controlling the imaging optical system by parallel movement.

According to a twenty-seventh aspect of the present invention, in order to achieve the third object, in the multi-eye imaging apparatus according to the twenty-sixth aspect, the control means is a user interface.

In order to achieve the fourth object, the storage medium according to the twenty-eighth aspect is a storage medium for storing a program for controlling a compound-eye imaging device, comprising two imaging optical systems arranged on the left and right. A program is stored which includes a combining module having a plurality of combining methods for creating one combined image from two captured left and right images, and a switching module for switching the plurality of combining methods.

In order to achieve the fourth object, the storage medium according to claim 29 is the storage medium according to claim 28, wherein the first of the plurality of synthesizing methods gives priority to the synthesizing speed. And a second one of the plurality of synthesizing methods is a synthesizing method 2 for prioritizing the image quality of the synthesized image.

In order to achieve the fourth object, the storage medium according to claim 30 is the storage medium according to claim 29, wherein the synthesizing method 1 includes the steps of: In the synthesis method 2, the two left and right images that have been imaged are given the respective brightness,
It is characterized by correcting left-right differences and trapezoidal distortion of color information, detecting overlapping areas of two images, and synthesizing using an overlap amount calculated from the overlapping areas.

In order to achieve the fourth object, the storage medium according to claim 31 is the storage medium according to claim 28 or 29, wherein the switching module includes: In the recording and reproducing mode, the mode is switched to the synthesizing method 2.

Further, in order to achieve the fourth object, the storage medium according to claim 32 is characterized in that:
0, wherein the composite image is a panoramic composite image.

Further, in order to achieve the fourth object, the storage medium according to claim 33 is characterized in that:
0, wherein the composite image is a stereoscopic image.

In order to achieve the fourth object, the storage medium according to claim 34 stores a program for controlling a compound-eye imaging apparatus for capturing a pair of images having parallax by two imaging optical systems. And storing a program having a control module for controlling a parallax of a main subject selected from a captured image to be adjusted.

In order to achieve the fourth object, the storage medium according to claim 35 is the storage medium according to claim 34, wherein the control module sets a limit value of a convergence angle of the imaging optical system. It is characterized by doing.

In order to achieve the fourth object, the storage medium according to claim 36 is the storage medium according to claim 34, wherein the control module is configured to determine whether or not the convergence angle of the imaging optical system is equal to or larger than a limit value. In addition, the parallax of the main subject in the image is adjusted by moving the image to be displayed in parallel.

Further, in order to achieve the fourth object, in the storage medium according to claim 37, in the storage medium according to claim 36, the translation means that the imaging optical system is translated. The control module adjusts the parallax of the main subject in the image by shortening the base line length of the imaging optical system and performing imaging when the convergence angle of the imaging optical system is equal to or greater than the limit value. .

In order to achieve the fourth object, the storage medium according to claim 38 of the present invention is the storage medium according to claim 36, wherein the translation is defined as an image pickup device in the image pickup optical system. Refers to moving in parallel, the control module, at or above the limit value of the convergence angle of the imaging optical system,
The image pickup device of the image pickup optical system is moved in parallel away from the center of the two image pickup optical systems to pick up an image, thereby adjusting the parallax of the main subject in the image.

In order to achieve the fourth object, the storage medium according to claim 39 of the present invention is characterized in that, in the storage medium according to claim 36, the translation means that the captured left and right images are translated. When the convergence angle of the imaging optical system is equal to or greater than the limit value, the control module translates the captured left and right images to create a stereoscopically captured display image, thereby obtaining a main subject in the image. Is characterized by adjusting parallax.

In order to achieve the fourth object, a storage medium according to claim 40 of the present invention controls a compound-eye imaging apparatus that captures a pair of images having parallax by two imaging optical systems. A storage medium for storing a program, wherein a limit value setting module that sets a limit value of a convergence angle of the imaging optical system, and the limit value set by the limit value setting module is less than or equal to the convergence angle or the parallel movement. A control module that controls the imaging optical system and controls the imaging optical system by the parallel movement when the imaging optical system is equal to or more than the limit value.

[0070]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) First, the first embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS. FIG. 1 is a block diagram showing a system configuration having a compound-eye imaging device according to the first embodiment of the present invention.
It comprises a compound-eye imaging device 1, a personal computer 2, and a display device 3.

The compound-eye imaging apparatus 1 comprises two left and right imaging optical systems 4a and 4b, a synchronizing signal generator 5, and an A / D converter 6.
a, 6b and a memory 7. Imaging optical systems 4a, 4
b denotes lenses 8a and 8b and an image sensor CC, respectively.
D9a and 9b. The CCDs 9a and 9b are connected to the synchronization signal generator 5, and the two CCDs 9a and 9b
Can be imaged synchronously. The compound-eye imaging device 1 is connected to the personal computer 2 by an interface cable 10.

The personal computer 2 has a parallel interface 11, a CPU (central processing unit) 1
2, memory 13, display controller 14, image synthesis unit 15, image correction / overlap amount calculation unit 16,
It has a storage device 17 and a mode selection section 18, which are each connected to a CPU bus 19. The display controller 14 has a VRAM (video random access memory) 20.

The input of image signals from the compound-eye imaging device 1 is performed by the parallel interface 11, and the output of images to the display device 3 is performed by the display controller 14.
Is to be done through.

Next, a panoramic image capturing and displaying method of a moving image by the compound-eye image capturing apparatus 1 according to the present embodiment will be described with reference to the flowchart of FIG. First, step S201
Is used to select and switch the processing mode. There are three types of processing modes: a through display mode for capturing, processing, and displaying an image in real time; a recording mode for temporarily recording a captured image; and a reproduction mode for reproducing a recorded image. The user selects a desired processing mode from these three types and switches the processing mode. In the present embodiment, the processing mode is selected and switched by the mode selection unit 18 which is hardware in the personal computer 2 in FIG. 1, but the processing mode is not selected and switched by hardware. It is possible to select and switch the processing mode by software.

Next, the flow of processing when each processing mode is selected will be described.

First, when the through display mode is selected, right and left images are picked up in step S202. In FIG. 1, two left and right images formed by the lenses 8a and 8b in the compound-eye imaging apparatus 1 are acquired by the respective CCDs 9a and 9b. The image is acquired synchronously on the left and right based on the signal from the synchronization signal generator 5. The obtained image signals are converted into digital images by A / D converters 6a and 6b, respectively, and stored in the memory 7. These two types of image signals are input to the personal computer 2 via the interface cable 10. The input image signal is transferred to the memory 13 through the CPU bus 19.

Here, since the CCDs 9a and 9b perform imaging in synchronization with each other, subsequent analog / digital image conversion of image signals and image transfer from the compound-eye imaging device 1 to the personal computer 2 are performed in synchronization. Even if they are not, the two images on the left and right transferred to the memory 13 are images synchronized with each other.

Therefore, one panoramic composite image is created from the left and right two-system images transferred to the personal computer 2. FIG. 3 shows a method of displaying a panoramic composite image on the display device 3 in the through display mode. In the figure, 201a is a left image, and 201b is a right image. Here, in performing panoramic synthesis of the two left and right images 201a and 201b, it is necessary to determine an overlap region where the two left and right images captured from one viewpoint are connected.
This overlap amount is set to a certain value in step S203 in the through display mode. This sets the appropriate value by the user. Then, based on this value, the image combining unit 15 in FIG. 1 combines the two left and right images 201a and 201b into a panoramic image in step S204.

In synthesizing a panoramic image, two images 20
The overlapping areas 1a and 201b are superimposed by substituting pixel values in one of the left and right images. The combined panoramic image is transferred to the VRAM 20 by the display controller 14 of FIG. This is displayed on the display device 3 only for the frame desired by the user, and it is determined in step S206 whether or not to end the display. If not, the process returns to step S202.
If the processing is to be terminated, the display device 3 is determined in step S207.
The display of is ended.

As described above, in the through display mode, a certain amount of overlap is set in advance, and the panorama synthesizing process is performed using the set value.

Next, the processing when the recording mode is selected will be described. FIG. 4 shows a panoramic imaging display method in the recording mode. If the recording mode has been selected, right and left images are captured in step SS208. The imaging of the left and right image signals by the compound eye imaging apparatus 1 is the same as in the above-described through display mode. Also, in the recording mode, display in the through display mode is performed on the display device 3. That is, in parallel with the recording of the panorama composite image, the overlap amount is set to a certain value in step SS209. Based on this value, in step S210, the image combining unit 15 in FIG. 1 combines the two left and right images 201a and 201b into a panoramic image. The combined panoramic image is stored in the VRAM 2 by the display controller 14 in FIG.
0 and is displayed on the display device 3. The user can record the panoramic composite image while watching the image displayed on the display device 3.

However, in recording a panoramic composite image, in the through display mode, the two images 201a and 201
When one panoramic composite image is created from b, the two images 201a and 201b are combined by giving a certain amount of overlap, which is a state as it is. First, step S2
At 14, the amount of overlap between the two left and right images 201a 'and 201b' is calculated by the image correction / overlap amount calculator 16 of FIG. These two left and right images 201a ',
The amount of overlap of 201b 'is determined by associating the pixel values of two captured left and right images 201a' and 201b 'using an algorithm such as template matching, and detecting the overlap area. It has been calculated. Therefore, the amount of overlap used for image synthesis is a variable value.

Next, in step S215, the image correction /
The overlap amount calculation unit 16 corrects an image using information of the captured image. That is, the compound-eye imaging device 1
Correction of the luminance and color information of the left and right two images and the trapezoidal distortion caused by the imaging optical systems 4a and 4b. FIG.
FIG. 9 shows a state in which the luminance that is discontinuous at the joint between the right image 201b 'and the left image 201a' is corrected. After performing these corrections and calculating the amount of overlap, panorama synthesis is performed by the image synthesis unit 15 in FIG. 1 in step S216. This is the same as the combination in step S204 in the through display mode, but the method of substitution is different for the amount of overlap between the left and right images, which is a parameter at that time. Further, the two images to be combined have been subjected to image correction in the recording mode.

The image subjected to the panoramic synthesis as described above is transferred to the storage device 17 of FIG. 1 and recorded in step S217. This recording is performed for a plurality of frames according to the user's setting, and it is determined in step S212 whether display display and recording are to be terminated. If not, the process returns to the step S208. If the process is to be ended, the display and recording are terminated in a step S213. Here, the image to be recorded is a panorama-combined image, but instead, it is also possible to record an image-corrected two left and right images and an overlap amount as attribute information.

Finally, the processing when the reproduction mode is selected will be described. When the reproduction mode is selected, the moving image file previously stored in the storage device 17 of FIG. 1 is read in step S218 and displayed on the display device 3. The display by the display device 3 is transferred to the VRAM 20 under the control of the display controller 14 in FIG.

As described above, the display according to the purpose of the user can be performed by switching the mode. In the through display mode, since a panorama composite image is created and displayed with a predetermined overlap amount set in advance, the processing time of one frame is reduced, and the same image with a high real-time frame rate can be provided. Further, in the recording mode, after performing image correction and accurate calculation of the overlap amount, in order to create a panoramic composite image, when displaying again on the display device 3 in the reproduction mode,
It is possible to obtain a panoramic composite image having a smooth joint between two images of good image quality.

In this embodiment, the image correction / overlap amount calculation unit 16 and the image synthesis unit 15 for processing the two left and right image signals in FIG. 1 are provided in the personal computer 2. It can also be provided in the device 1. Further, a device such as a workstation can be used instead of the personal computer 2. An interface cable 10 for connecting the compound-eye imaging apparatus 1 and the personal computer 2 is separately transferred to each system by two interfaces, and is divided into two systems by using one interface to time-divide the left and right image signals into the personal computer 2. Can be transferred to Other CPs in the personal computer 2
As the U bus 19, buses of various standards such as an ISA bus and a PCI bus can be used.

(Second Embodiment) Next, a second embodiment of the present invention will be described with reference to FIGS. While the first embodiment described above obtains a panoramic composite image, the present embodiment obtains a stereoscopic image. Note that the system configuration including the compound-eye imaging device according to the present embodiment is the same as that of FIG. 1 in the above-described first embodiment, and furthermore, stereoscopic imaging of a moving image by the compound-eye imaging device according to the present embodiment. Since the flowchart of the display process is the same as that of FIG. 2 in the first embodiment, the description will be made with reference to both drawings.

First, at step S201, the processing mode is selected and switched. This processing mode has three types: a through display mode, a recording mode, and a reproduction mode, which are the same as those in the first embodiment. The user selects a desired processing mode from these three types and switches the processing mode. Also in the present embodiment, the processing mode is selected and switched by the mode selection unit 18 which is hardware in the personal computer 2 in FIG. 1, but the processing mode is not selected and switched by hardware. Then, select the processing mode by software,
It is possible to switch.

Next, the flow of processing when each processing mode is selected will be described.

First, when the through display mode is selected, two images on the left and right are captured in step S202.
The method of capturing two images on the left and right by the compound-eye imaging apparatus 1 is the same as the above-described panoramic imaging display. However, in the panoramic imaging display, the images are captured while the viewpoints are matched, whereas in the stereoscopic imaging display, the imaging optical systems 4a, 4b
Are arranged at intervals given by the base line length, and images are captured from two viewpoints.

Next, the captured left and right two-system images are transferred to the personal computer 2 to create one stereoscopic image. FIG. 5 shows a method of displaying a stereoscopic image on the display device 3 in the through display mode. In the figure, 30
1a is a left image, 301b is a right image, and 302 is a main subject. Here, it is necessary to determine an overlap amount for creating a stereoscopic image from the two left and right images 301a and 301b. The amount of overlap in a stereoscopic image refers to the amount by which two left and right images are overlapped. By changing the amount of overlap, the parallax of the stereoscopic image to be created, that is, the stereoscopic effect can be controlled. This overlap amount is set to a certain value in step S203. In the present embodiment, the value of the overlap amount is set so as to reduce the parallax of the main subject 302 in the captured image. This is set to facilitate the fusion of the main subject 302, but the amount of overlap can be freely set by the user. Then, one stereoscopic image is created by the image synthesizing unit 15 in FIG. 1 in step S204 based on the set values.

In displaying a stereoscopic image, as described in the section of the prior art, the left and right images are alternately output on a display device, and viewed with liquid crystal shutter glasses that shutter in synchronization with the switching of the images. The method and two left and right images are alternately arranged every other line in the horizontal direction,
There is a method in which a sheet that alternates the direction of polarization is covered every other line, and viewing is performed from above using glasses with different polarizations on the left and right. As described above, there are a plurality of types of stereoscopic image display methods, but any display method is applicable here.

The image created as described above is processed in step S20 in the same manner as in the above-described panoramic imaging display.
5 at the display controller 14 of FIG.
The data is transferred to the AM 20 and displayed on the display device 3. Next, in step S206, the display is performed for the time and the number of frames set by the user, and it is determined whether to end the display. If the processing is not to be terminated, the above-described step S is performed.
Returning to step 202, if the processing is to be ended, the display on the display device 3 is ended in step S207.

As described above, in the through display mode, a stereoscopic image is created as it is from the two left and right images and displayed on the display device 3. This makes it possible to perform stereoscopic imaging display of a moving image with a high frame rate.

Next, a process when the recording mode is selected will be described. FIG. 6 shows a stereoscopic imaging display method in the recording mode. In the figure, 301a 'is a left image, 301
b 'is a right image, and 302 is a main subject. If the recording mode has been selected, right and left images are captured in step SS208. The imaging of the left and right image signals by the compound eye imaging apparatus 1 is the same as in the above-described through display mode. Also, in the recording mode, display in the through display mode is performed on the display device 3. That is, in parallel with the recording of the stereoscopic image, step SS209
Sets the overlap amount to a certain value. Based on this value, in step S210, the image combining unit 15 in FIG. 1 combines the two left and right images 301a and 301b into a stereoscopic image. The synthesized stereoscopic image is obtained in step S
At 211, the data is transferred to the VRAM 20 by the display controller 14 of FIG. The user can record a stereoscopic image while viewing the image displayed on the display device 3.

However, in recording a panoramic composite image, in the through display mode, the two images 301a and 301
In the case of creating one stereoscopic image from the image b, the two images 301a and 301b are combined while giving a certain amount of overlap as they are, but in the recording mode, the stereoscopic image is created. Before performing, first, step S21
In step 4, the amount of overlap between the left and right images 301a 'and 301b' is calculated by the image correction / overlap amount calculation unit 16 in FIG. These two left and right images 301a ', 3
For the overlap amount of 01b ', the main subject 302 is extracted from the two left and right images 301a' and 301b 'taken, and the parallax is set to 0. That is, the two left and right images 301a ',
It shows the amount of overlap when 301b 'is overlapped.
Therefore, the amount of overlap used for creating a stereoscopic image is a variable value. Here, in the present embodiment, parallax is set to 0.
However, it is also possible to set the parallax to a value other than 0.

Next, in step S215, the image correction /
The overlap amount calculation unit 16 corrects an image using information of the captured image. That is, the compound-eye imaging device 1
Correction of the luminance and color information of the left and right two images and the trapezoidal distortion caused by the imaging optical systems 4a and 4b. FIG.
FIG. 9 shows a state in which the luminance that is discontinuous at the joint between the right image 301b 'and the left image 301a' is corrected. After performing these corrections and calculating the amount of overlap, a stereoscopic image is created by the image combining unit 15 of FIG. 1 in step S216. This is the same as the combination in step S204 in the through display mode, but the amount of overlap between the left and right images, which is a parameter at that time, is
The assignment method is different. Further, the two images to be combined have been subjected to image correction in the recording mode.

The stereoscopic image created as described above is transferred to the storage device 17 in FIG. 1 and recorded in step S217. This recording is performed for a plurality of frames according to the user's setting, and it is determined in step S212 whether display display and recording are to be terminated. If not, the process returns to the step S208. If the process is to be ended, the display and recording are terminated in a step S213.

Here, the image to be recorded is a created stereoscopic image, but instead, it is also possible to record an image-corrected two left and right images and an overlap amount as attribute information.

Finally, the processing when the reproduction mode is selected will be described. When the reproduction mode is selected, the moving image file previously stored in the storage device 17 of FIG. 1 is read in step S218 and displayed on the display device 3. The display by the display device 3 is transferred to the VRAM 20 under the control of the display controller 14 in FIG.

As described above, the moving image can be displayed according to the purpose of the user by switching the mode. Since the stereoscopic image is created and displayed as it is from the two left and right images captured in the through display mode, the processing time for one frame is reduced, and a moving image with a high real-time frame rate can be provided. Also,
In the recording mode, after the image correction and the calculation of the appropriate amount of overlap are performed, a stereoscopic image is created. Therefore, when the stereoscopic image is displayed on the display device 3 again in the reproduction mode, a high-quality stereoscopic image can be obtained. it can.

Here, in the first and second embodiments described above, the through display mode and the recording mode are selected in advance before capturing an image, and then each process is performed. The recording mode is turned on (ON) and off (OFF) by the mode selection unit 18 from the state of the through display mode.
, It is also possible to use a processing algorithm for synthesizing and recording images in parallel with the live view display on the display device 3.

In a compound-eye image pickup apparatus, the image pickup optical system is arranged so that the viewpoint of an image picked up by using a mirror or the like is coincident in a panoramic image pickup display.
Images are taken by arranging them in parallel at mm intervals, but the arrangement of these imaging systems can be easily changed. Therefore, it is possible to perform both two-dimensional panoramic imaging display and three-dimensional stereoscopic imaging display in one compound-eye image input / output device.

(Third Embodiment) Next, a third embodiment of the present invention will be described with reference to FIGS. In the present embodiment, by changing the convergence angle and the base line length of the imaging optical system, the parallax of the main subject in the stereoscopically captured display image is set to 0, and the user's main subject is fused. It was made.

FIG. 7 is a block diagram showing a configuration of a compound eye imaging apparatus according to the third embodiment of the present invention. In the figure, 7
Reference numeral 00 denotes a compound-eye imaging apparatus, and two left and right imaging optical systems 701.
a, 701b, signal processing unit 704, subject position detection unit 7
05, finder 706, interface 707,
It has imaging optical systems 708a and 708b and a user interface 709 for controlling the convergence angle / parallel movement amount.

The imaging optical systems 701a and 701b include lenses 702a and 702b, and a CCD 7 serving as an imaging device, respectively.
03a and 703b. Two imaging optical systems 70
1a and 701b are captured by the signal processing unit 7
04, where image processing such as synthesis of a stereoscopic image, image correction, and image output is performed. The signal processing unit 704 is connected to the subject position detection unit 705, finder 706, and interface 707. Finder 706
Outputs an image on which image correction and synthesis have been performed. By looking through the viewfinder 706, an image stereoscopically viewed can be viewed. When an image is edited by an external device (not shown) such as a personal computer or an image is displayed on a display device (not shown), the image is transferred to another external device via the interface 707. Has become.

An object position detection unit 705 is an interface used when the user selects a main object in a stereoscopically displayed image, and sets the depth of the selected main object from the imaging optical system and the parallax of the main object to zero. And a calculation unit for calculating a convergence angle required for the calculation.

Here, when a point included in the object of interest is designated by a pointing device such as a mouse from an image captured by the left imaging optical system displayed on the viewfinder 706, a certain size centered on the point is specified. The corresponding point in the right image is detected by matching using the template. From this set of corresponding points, the parallax at that position is determined, and the position of the main subject, that is, the depth from the imaging optical system is calculated from the parallax. Further, the amount of convergence angle required when only the convergence angle is controlled to reduce the parallax of the main subject to zero is calculated.

However, the method of selecting the main subject in the image is not limited to using only the user interface 709 for controlling the angle of convergence / parallel movement, and the main subject in the image is automatically extracted. You may. Further, the main subject may be located at the center of the image, and may be determined in advance at one point of the center of the image, and the parallax of that portion may be adjusted.

Now, in FIG. 23, two imaging optical systems 70
It is assumed that a subject 904 is imaged as an imaging target of 1a and 701b. Here, the compound eye imaging device 70 of FIG.
When the VD7QE 904 is selected by the 0 subject position detection unit 705, the depth z of the selected subject 904 from the two imaging optical systems 701a and 701b is detected.

The subject position detecting section 705 is connected to the image pickup optical system driving devices 708a and 708b, and the position information of the main subject 904 obtained here is picked up by the image pickup optical systems 701a and 701b connected to the two image pickup optical systems 701a and 701b. The data is transferred to the optical system driving devices 708a and 708b. The imaging optical system driving devices 708 a and 708 b use the imaging optical system 7 based on the position information of the main object 904 transferred from the object position detection unit 705.
01a and 701b are automatically controlled.

Next, the image pickup optical system driving devices 708a and 708
A method of automatically controlling the convergence angles and the amount of parallel movement of the imaging optical systems 701a and 701b by b.

As described above, conventionally, there has been a method of adjusting the parallax only by the convergence angle or only the parallel movement.
As for the control method based on the convergence angle, as shown in FIG. 24, the angle θ determined by the distance z between the imaging optical systems 701a and 701b and the main subject 904 and the base length l is obtained by the following equation (1).

Θ = arctan l / 2z (1) If the imaging optical systems 701a and 701b have a convergence angle by this angle θ, the parallax of the main subject 904 can be made zero. However, in this case, the left and right imaging optical systems 70
Since the conjugate surfaces 905b and 905c of the imaging plane are different between the left and right imaging optical systems 701a and 701b by giving the convergence angles to 1a and 701b, respectively,
The images distorted from each other are obtained in the peripheral portion except for the region No. 4. This distortion increases as the convergence angle increases, and it becomes impossible to obtain a good stereoscopic image.

Therefore, in the present embodiment, the imaging optical system 70
For the convergence angles given to 1a and 701b, a limit value that can be viewed as a stereoscopic image is determined, and no more convergence angle is given. Here, as the limit value of the convergence angle, a value based on a certain setting condition or a empirical value of a human may be used. The limit value of the convergence angle is stored in a memory provided in the imaging optical system driving devices 708a and 708b in FIG.
It is determined whether the convergence angles of 1a and 701b are the set limit values. In the present embodiment, the imaging optical system 70
1a and 701b are set so that the base length l is 65 mm and the convergence angle can be controlled so that parallax is 0 for an object located from infinity to 1 m. That is, when l = 65 mm and z = 1 m in the above equation (1), the limit value of the angle θ is 0.0325 (rad) (1.86 °), and this imaging system sets the convergence angle to 1.86. Can be given up to °.

Next, each of the imaging optical systems 701a and 701b
FIG. 8 shows a method of controlling the imaging optical systems 701a and 701b for setting the parallax of the main subject 904 located m in front to zero.
A description will be given based on the flowchart of FIG.

FIG. 9 shows two image pickup optical systems 701a and 701.
It is a figure which shows the example which has arrange | positioned giving a convergence angle to the limit value which set b. Here, the limit value of the convergence angle is θ ′. 8, first, in step S801, the main subject 904 in FIG. 9 is detected by the subject position detection unit 705 in FIG. 7 to detect the depth from the imaging optical systems 701a and 701b. Next, in step S802, step S801 is performed.
The amount of convergence angle required when only the convergence angle is controlled to reduce the parallax of the main subject 904 to 0 from the depth detected in the above is calculated, and the calculated convergence angle is determined in advance by the imaging optical system drive shown in FIG. It is compared with the limit value of the convergence angle stored in the memories of the devices 708a and 708b, and it is determined whether or not the parallax can be adjusted to 0 with θ <θ ′. Then, imaging optical systems 701a and 701 necessary for setting the parallax of the main subject 904 to 0 are set.
If the convergence angle of b is equal to or smaller than the limit value θ ′, that is, θ <θ ′
If the parallax can be adjusted to 0 in step S803, the parallax is adjusted to 0 by changing only the convergence angle in step S803, and then in step S803.
At 804, the adjustment for setting the parallax of the main subject 904 to 0 ends.

On the other hand, in step S802, the image pickup optical system 7 necessary for setting the parallax of the main subject 904 to zero is set.
01a and 701b are equal to or larger than the limit value θ ',
That is, if the parallax cannot be adjusted to 0 at θ <θ ′, the convergence angle is changed to θ = θ ′ (stop the convergence angle at the limit value) in step S805, and the imaging of FIG. 7 is performed in the next step S806. The imaging optical system 7 is controlled by the optical system driving devices 708a and 708b.
After the parallax is adjusted to 0 by changing the base line length of 01a and 701b, the adjustment for setting the parallax of the main subject 904 to 0 in step S804 ends.

Now, the left image pickup optical system 701a in FIG.
It is assumed that the optical axis La intersects with the conjugate plane 905a of the imaging plane when viewed in parallel at A ″. At this time, the distance d between the position A where the main subject 904 exists and A ″ is represented by the following (2). ).

D = 1 / 2-z × tan θ ′ (2) Accordingly, by moving the left imaging optical system 701a by d in the direction of the origin O1, the optical axis La of the left imaging optical system 701a and the imaging are moved. The intersection A ″ of the plane with the conjugate plane 905a can be adjusted to the position A where the main subject 904 exists.

Further, the right image pickup optical system 701 in FIG.
b, the imaging optical system 701a on the left centering on the origin O1
And the right imaging optical system 701
The main subject 904 can be aligned on the optical axis Lb by moving b parallel to the origin O1 so as to shorten the base line length l. Thereby, the two imaging optical systems 7
0a, 701b, main subject 90 of the image picked up
The parallax of 4 can be adjusted to 0.

In FIG. 10, the convergence angle θ is limited to the limit value θ ′, and is parallel-translated so as to shorten the base line length l, so that the two imaging optical systems 70a, 70a,
The example of arrangement of 701b is shown.

As described above, for the purpose of adjusting the amount of parallax, 2
For the convergence angles given to the two imaging optical systems 701a and 701b, a limit value θ ′ that can be viewed as a stereoscopic image is determined.
For further adjustment, the two imaging optical systems 701a and 701
It is adjusted by shortening the base length l of b. Accordingly, it is possible to easily fuse the main subject 904 and obtain a good stereoscopically displayed image.

In the present embodiment, the purpose is to make the parallax of the main subject 904 zero.
The present invention is not limited to this, and can be applied to a case where the parallax amount is adjusted to an arbitrary amount.

(Fourth Embodiment) Next, a fourth embodiment of the present invention will be described with reference to FIGS. In the present embodiment, the convergence angle of the imaging optical system is changed, and furthermore, the position of the imaging device (CCD) in the imaging optical system is moved in parallel to reduce the parallax of the main subject in the stereoscopically displayed image. Thus, the fusion of the main subject of the user is facilitated.

FIG. 11 is a block diagram showing a configuration of a compound eye imaging apparatus according to the fourth embodiment of the present invention. In FIG. 11, the same parts as those in FIG. 7 in the third embodiment described above are the same. The code is attached. The difference between FIG. 11 and FIG. 7 is that the configuration of FIG.
CCD driving device 710 for driving each of b
a, 710b.

Also in this embodiment, the imaging optical systems 701a, 701a, 70
1b can have a convergence angle.

In the third embodiment, when the convergence angle exceeds the limit value θ ′, the convergence angle is limited to the limit value θ ′, and the base length l of the imaging optical systems 701a and 701b is reduced. It was controlled by shortening it.

However, in the present embodiment, the imaging optical system 70
Instead of shortening the base line length 1 of 1a and 701b, CCDs 703a and 703b in the imaging optical systems 701a and 701b are used.
Are moved in parallel by the CCD driving devices 710a and 710b to reduce the parallax of the main subject 904 to zero.

Next, the flow of processing for setting the parallax of the main subject 904 to 0 in the compound-eye imaging apparatus according to the present embodiment will be described with reference to the flowchart of FIG.

First, in step S1201, the main object 904 is detected by the object position detector 5 in FIG. 11, and the depth from the imaging optical systems 701a and 701b is detected. Next, at step S1202, at step S120
The amount of convergence angle required when only the convergence angle is controlled to reduce the parallax of the main subject 904 to 0 from the depth detected in 1 is calculated, and the calculated convergence angle θ is determined in advance in FIG.
1 is compared with the limit value θ ′ of the convergence angle held in the memory of the imaging optical system driving devices 708a and 708b.
It is determined whether parallax can be adjusted to zero.

If the convergence angle of the imaging optical systems 701a and 701b required to reduce the parallax of the main subject 904 to 0 is equal to or smaller than the limit value θ ′, that is, if the parallax can be adjusted to 0 with θ <θ ′, After setting the parallax to 0 only by controlling the convergence angle in step S1203, the adjustment for setting the parallax of the main subject 904 to 0 in step S1204 ends.

On the other hand, necessary imaging optical systems 701a and 701
There is also a case where the convergence angle of b is equal to or larger than the limit value θ ′. Therefore, if the convergence angles of the imaging optical systems 701a and 701b required to reduce the parallax of the main subject 904 to 0 are equal to or larger than the limit value θ ′, that is, if the parallax cannot be adjusted to 0 with θ <θ ′, In step S1205, the convergence angle is stopped at the limit value θ ′,
The CCD driving devices 710a and 710b of FIG.
After the parallax is set to 0 by moving the CDs 703a and 703b in parallel, the main subject 9 is determined in step S1204.
The adjustment for setting the parallax of 04 to 0 ends.

In FIG. 9, the position of the main subject 904 formed on the CCD 703a in the left imaging optical system 701a is at the point b. Therefore, the image of the main subject 904 is now represented by C
Let us consider moving the CD 703a to the center O2 of the surface.
Assuming that the distance between the rear principal plane of the lens 703a and the CCD 703a is v, and the distance between b and O2 is x, x
Is determined by the following equation (3).

X = v × tan (θ−θ ′) = v × l−2ztan θ ′ / 2z + ltan θ ′ (3) Accordingly, only the x is set so that the CCD 703a moves the point b closer to O2 in the direction perpendicular to the optical axis La. When shifted, the main subject 90
4 can be moved by the distance x and brought to the center O2 of the CCD 703a.

FIG. 13 shows an example of the arrangement of the imaging optical systems 701a and 701b when the parallax of the main subject 904 is set to 0 by moving the CCD 703a in a direction perpendicular to the optical axis La in a direction away from the origin O1. .

As described above, the parallax amount imaging optical system 7
The convergence angle given to each of the convergence angles 01a and 701b defines a limit value that can be viewed as a stereoscopic image.
By moving the CDs 703a and 703b in parallel in a direction away from the origin O1, the parallax of the main subject 904 is set to zero. Thus, the main subject 904 can be easily fused and a good stereoscopically displayed image can be viewed. Also, C
In the adjustment of the parallax amount only by the parallel movement of the CDs 703a and 703b, there is a problem that the parallel movement amount is large and the control is difficult. However, according to the present embodiment, the control of the convergence angle and By using them together, the load of parallel movement control can be reduced.

By the way, in the present embodiment, the purpose is to set the parallax of the main subject 904 to 0, as in the case of the third embodiment described above. The present invention is also applicable.

Further, in this embodiment, the CCD is used as the image pickup device, but the present invention is not limited to this.
Other image sensors may be used.

(Fifth Embodiment) Next, a fifth embodiment of the present invention will be described with reference to FIGS. 7, 9 and 14. FIG. Note that the configuration of the compound eye imaging apparatus according to the present embodiment is the same as that of FIG. 7 in the above-described third embodiment, and will be described with reference to FIG.

In the third and fourth embodiments described above, the convergence angle θ of the imaging optical systems 701a and 701b is changed to a predetermined limit value θ ′, and further control is performed by hardware. .

In the present embodiment, the imaging optical system 701a,
After changing the convergence angle θ of 701b to the defined limit value θ ′, when the stereoscopic image is actually created from the two captured left and right images, the captured image is translated, and the stereoscopic image The parallax is adjusted.

FIG. 14 is an explanatory diagram of convergence angle control up to the limit value θ ′ of the convergence angle θ in the compound-eye imaging apparatus according to the present embodiment and stereoscopic imaging display by software. Similarly, in the present embodiment, it is assumed that the imaging optical systems 701a and 701b can have a convergence angle up to the limit value θ ′ determined in FIG.

In the above-described fourth embodiment, it has been described that the position of the image of the main subject 904 on the CCD is shifted by x from the center O2 of the CCD. Accordingly, the position of the main subject 904 in the images captured by the left and right imaging optical systems 701a and 701b is shifted by 2x on the CCD surface. This shift amount 2x and the CCD element size s
Is known, the shift amount p (pixel) in the captured image
l) is obtained by the following equation (4).

P = 2x / s (4) This shift amount p is x in the imaging optical system driving device 708a of FIG.
Is calculated and then transferred to the signal processing unit 704, where it is converted into a shift amount p in the captured image.

Therefore, in this signal processing unit 704, FIG.
In FIG. 7, the right imaging optical system 7 with respect to the image 1401a captured by the left and right imaging optical systems 701a and 701b in FIG.
01b, an image 1401b captured by p (pixel)
l) A stereoscopic image 1402 synthesized in a state shifted by only
Is displayed on the display of the viewfinder 706.

For example, when the left and right images are alternately output to the left and right on the display of the viewfinder 706, and the user views with liquid crystal shutter glasses that switches the left and right shutters in synchronization with the switching of the display of the left and right images. In this case, the right and left images may be alternately output by being shifted from each other by p (pixel). In addition, one image in the horizontal direction is added to an area of one stereoscopic image in which two left and right images are created in advance.
In the case of creating a striped image composed of two left and right images by alternately arranging every other line, it is only necessary to give a p (pixel) shift when synthesizing a stereoscopic image.

When the parallax is controlled after exceeding the limit value of the convergence angle as described above, a stereoscopic image is created by translating the right image with respect to the left image on software when creating the stereoscopic image. Thus, the parallax of the main subject 904 can be set to zero. Accordingly, it is possible to easily and excellently view a stereoscopically-captured display image without imposing a load on hardware.

By the way, the present embodiment also aims at setting the parallax of the main subject 904 to 0 as in the case of the third embodiment described above. The present invention is also applicable. Further, according to the present embodiment, the left and right images once captured and recorded in the memory can be displayed with an arbitrary parallax amount even when a stereoscopic image is newly created and reproduced. Further, the main subject 904, which is the object of imaging through the third to fifth embodiments described above, always exists on the bisector of the left and right imaging optical systems 701a and 701b. The parallax can be controlled as a main subject even for an object existing at an arbitrary position in a region to be imaged.

(Sixth Embodiment) Next, a sixth embodiment of the present invention will be described with reference to FIG. 7, FIG. 10, and FIGS. Note that the configuration of the compound eye imaging apparatus according to the present embodiment is the same as that of FIG. 7 in the above-described third embodiment, and will be described with reference to FIG.

In the third and fourth embodiments described above, the method in which the imaging optical system automatically moves when the user selects the main object 904 in the captured image by the object position detection unit 705.

In the present embodiment, the user can control the imaging optical system via an interface instead of automatically controlling the imaging optical system.

FIG. 15 is a view showing a compound-eye imaging apparatus having a user interface for controlling the angle of convergence / parallel movement. In the figure, reference numeral 1501 denotes a compound-eye imaging device, which is provided with a finder 1502 and control buttons 1503 on the rear surface. The control button 1503 indicates the convergence angle / parallel movement amount control user interface 709 in FIG.

In FIG. 7, the convergence angle / parallel movement amount control user interface 709 is connected to two image pickup optical system drive units 708a and 708b. And
The user sends a control signal to the imaging optical system driving devices 708a and 708b via the control button 1503 to control the two imaging optical systems 701a and 701b, respectively. The control button 1503 has a (+) direction and a (−) direction for providing a convergence angle, and when the control button 1503 is pressed, the imaging optical systems 701a and 701b can be moved.

Next, the flow of the process of moving the imaging optical systems 701a and 701b by the control button 1503 will be described with reference to FIGS.
7 will be described.

In this embodiment, as shown in FIG. 26, the imaging optical systems 701a and 701a,
The control is performed by changing the base line length 1 of 01b.

First, the operation when the control button 1503 is pressed in the (+) direction will be described with reference to FIG. FIG. 18 shows an operation state of the imaging optical systems 701a and 701b when the control button 1503 is pressed in the (+) direction.

When the control button 1503 is pressed in the (+) direction, first, in step S1601, the current imaging optical system 70 is pressed.
The convergence angles 1a and 701b are measured, and a magnitude relationship (θ ≦ θ ′) between the measured convergence angle θ and a preset limit value θ ′ of the convergence angle is determined. If θ ≦ θ ′, that is, if the current convergence angle is equal to or smaller than the limit value θ ′, the imaging optical systems 701a and 701b are moved to increase the convergence angle in step S1602.

FIG. 18 shows a state in which two imaging optical systems 701a and 701b having a convergence angle equal to or less than the limit value θ 'are arranged at the positions indicated by the broken lines by increasing the convergence angle from the positions indicated by the solid lines. I have.

Here, while the control button 1503 is pressed in the (+) direction, it is determined in step S1603 whether or not the convergence angle θ of the imaging optical systems 701a and 701b at that time is equal to or less than the limit value θ ′. I have. Further, the control button 15
03 in the (+) direction, and when the convergence angle θ has reached its limit value θ ′, this time in FIG.
As shown in (5), the two imaging optical systems 701a and 701b move in parallel in a direction in which the distance between them decreases. Here, first, in step S1601, the current convergence angle θ of the imaging optical systems 701a and 701b is compared with a preset limit value θ ′ of the convergence angle, but not θ ≦ θ ′ but θ ≧ θ. In the case of ', the steps S1602 and S1603 are skipped, and the process of step S1604 is directly executed.

Next, in step S1605, the control button 15
03 and two imaging optical systems 701a and 701b
Gradually decreases the distance between the two imaging optical systems 701
It is determined whether or not the distance between a and 701b has become 0, that is, whether or not the two imaging optical systems 701a and 701b have contacted each other at the origin O1 in FIG. If no contact is made, the process returns to step S1604. If contact is made, the movement of the imaging optical systems 701a and 701b in the (+) direction ends. FIG. 18 shows a state in which the imaging optical systems 701a and 701b at the positions indicated by the broken lines having the limit value θ ′ of the convergence angle have been translated to the positions indicated by the two-dot chain lines.

When the control button 1503 is pressed in the (+) direction as described above, first, the imaging optical systems 701a and 701a
The convergence angle of 1b is increased to a limit value θ ′, and when the limit value θ ′ is reached, control is performed so that the two imaging optical systems 701a and 701b move in parallel in a direction in which the distance gradually decreases. However, this operation is an example, and the convergence angle becomes the limit value θ ′.
It is also possible to control so as to translate in parallel even if it has not been reached.

Next, the operation when the control button 1503 is pressed in the (-) direction will be described with reference to FIG.

FIG. 19 shows the operation state of the imaging optical systems 701a and 701b when the control button 1503 is pressed in the (-) direction.

When the control button 1503 is pressed in the (-) direction, first, in step S1701, the current imaging optical system 70 is pressed.
The convergence angle θ of 1a and 701b is measured, and it is determined whether or not the convergence angle θ is 0 (θ ≠ 0). If θ ≠ 0, the imaging optical systems 701a and 701b are moved so as to reduce the convergence angle θ in step S1702.

FIG. 19 shows a state in which the imaging optical systems 701a and 701b in the state where the convergence angle is present are arranged at the position shown by the solid line by reducing the convergence angle from the position shown by the broken line.

Next, in step S1703, it is determined whether the convergence angle θ is 0 or not until it becomes 0.
When the convergence angle θ becomes 0 when the 503 is continuously pressed in the (−) direction, the two imaging optical systems 7
01a and 701b move in parallel in the direction in which the distance increases.
Here, first, in step S1701, the current convergence angle θ of the imaging optical systems 701a and 701b is not θ な く 0 but θ = 0.
If it is, the above-mentioned steps S1702 and S170
Step 1703 is skipped and step S1704 is directly performed.
Execute the processing of

Next, in step S1705, the imaging optical system 70
It is determined whether or not 1a and 701b have reached (contacted) the point D which is the limit point of the parallel movement. If the point D has not been reached, the process returns to step S1704. If the point D has been reached, the imaging optical systems 701a and 701b in the (-) direction.
Movement ends. FIG. 19 shows a state in which the imaging optical systems 701a and 701b located at a position indicated by a broken line where the convergence angle is 0 are translated to a position indicated by a two-dot chain line.

When the control button 1503 is pressed in the (-) direction as described above, first, the imaging optical systems 701a and 701a
The convergence angle of 1b decreases to 0 in some cases,
At this time, control is performed so as to translate in the direction in which the distance between the imaging optical systems 701a and 701b increases. However, this operation is an example, and it is possible to perform control so as to perform parallel movement even if the convergence angle is not zero.

As described above, the user operates the control button 15
03, the imaging optical systems 701a, 701
b can be controlled so as to have a convergence angle up to a limit value θ ′ at which a good stereoscopic image can be obtained, and to move in parallel at a limit value θ ′ or more. This allows the user to freely adjust stereoscopic images having different parallaxes according to his / her preference.

Here, the operation method of the image pickup optical systems 701a and 701b in the present embodiment is an example, and the convergence angle control and the parallel movement control are performed within the range of the preset limit value θ ′ of the convergence angle. Arbitrarily selected imaging optical systems 701a, 701
The position of the main subject 904 can be controlled by adjusting the position of the main subject 01b.

In the third to fifth embodiments described above, the automatic control of the parallax in the two left and right images has been described. In the present embodiment, the control of the base length l of the imaging optical systems 701a and 701b in FIG. 10 has been described, but the same applies to the case of the parallel movement control of the CCDs 703a and 703b. Further, the imaging optical system 701
Although the control method using the control button 1503 has been described as an example of the control of a and 701b, other devices may be used as long as the user interface can control the convergence angle and the translation amount.

(Seventh Embodiment) Next, a storage medium used in the compound-eye imaging apparatus of the present invention will be described with reference to FIGS.

As shown in FIG. 20, at least a “compositing module” and a “switching” are stored in a storage medium for storing a program for controlling a compound-eye imaging apparatus that captures a pair of images having parallax by two imaging optical systems. module"
The program code of each module may be stored.

Here, the "synthesis module" is a program module having a plurality of synthesis methods for creating one composite image from two left and right images picked up by two imaging optical systems arranged on the left and right. The "switching module" is a program module for switching the plurality of combining methods.

The first of the plurality of synthesizing methods is the synthesizing method 1 in which the synthesizing speed is prioritized, and the second of the plurality of synthesizing methods is to prioritize the image quality of the synthesized image. This is a synthesis method 2 for synthesis. Further, the synthesis method 1 includes:
The two synthesized left and right images are synthesized by giving a certain amount of overlap, and the synthesizing method 2 corrects the two left and right images to correct the left-right difference and the trapezoidal distortion of the luminance and color information, and further corrects the left and right. The overlap area of two images is detected, and the two images are combined using the overlap amount calculated from the overlap area. In addition, the "switch module"
In the through display mode, the synthesizing method 1 is switched, and in the recording and reproducing mode, the synthesizing method 2 is switched.

Further, as shown in FIG. 21, at least the program code of the "control module" may be stored in another storage medium for storing the program for controlling the above-described compound eye imaging apparatus.

Here, the "control module" is a program module for performing control to adjust the parallax of the main subject selected from the captured image. Also,
The “control module” sets a limit value of the convergence angle of the imaging optical system. In addition, the “control module” adjusts the parallax of the main subject in the image by moving the displayed image in parallel when the convergence angle of the imaging optical system is equal to or more than the limit value. The translation refers to translating the imaging optical system, and the “control module” shortens the baseline length of the imaging optical system when the imaging optical system is at or above the limit value of the convergence angle. By doing so, the parallax of the main subject in the image is adjusted. In addition, the parallel movement refers to parallel movement of an imaging device in the imaging optical system, and the “control module” performs imaging of the imaging optical system when the convergence angle of the imaging optical system is equal to or more than a limit value. The parallax of the main subject in the image is adjusted by moving the element in parallel away from the center of the two image pickup optical systems and picking up an image. Further, the translation refers to translating the captured right and left images, and the “control module” translates the captured left and right images when the convergence angle of the imaging optical system is equal to or more than a limit value. Then, the parallax of the main subject in the image is adjusted by creating a stereoscopically captured display image.

Further, as shown in FIG. 22, another storage medium for storing a program for controlling the above-described compound eye imaging apparatus includes a “limit value setting module” and a “control module”.
The program code of each module may be stored.

Here, the "limit value setting module" is a program module for setting a limit value of the convergence angle of the imaging optical system. Further, the “control module” controls the imaging optical system by the convergence angle or the parallel movement below the limit value set by the “limit value setting module”, and controls the imaging optical system by the parallel movement above the limit value. Is a program module for controlling.

[0183]

As described above in detail, according to the first compound-eye imaging method and apparatus of the present invention, it is possible to perform panoramic imaging display and stereoscopic imaging display of a moving image according to an output mode. .

Further, according to the second compound eye imaging method and apparatus of the present invention, it is possible to obtain a stereoscopic image which facilitates fusion of a main subject.

Further, according to the third compound eye imaging method and apparatus of the present invention, there is an effect that the control of the imaging optical system can be easily performed by manual operation.

Further, according to the storage medium of the present invention, there is an effect that the above-described compound eye imaging apparatus can be smoothly controlled.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a system configuration including a compound-eye imaging device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a processing flow of the compound eye imaging apparatus.

FIG. 3 is an explanatory diagram of a panoramic image synthesizing method in a through display mode of the compound eye imaging apparatus.

FIG. 4 is an explanatory diagram of a panoramic image synthesizing method in a recording mode of the compound eye imaging apparatus.

FIG. 5 is an explanatory diagram of a stereoscopic image creating method in a through display mode of the compound eye imaging apparatus according to the second embodiment of the present invention.

FIG. 6 is an explanatory diagram of a stereoscopic image creating method in a recording mode of the compound eye imaging apparatus.

FIG. 7 is a block diagram showing a system configuration having a compound-eye imaging device according to a third embodiment of the present invention.

FIG. 8 is a flowchart showing a flow of a process of setting the parallax of a main subject to 0 in the compound-eye imaging apparatus.

FIG. 9 is an explanatory diagram of stereoscopic imaging display by convergence angle control up to the convergence limit in the compound-eye imaging apparatus.

FIG. 10 is an explanatory diagram of stereoscopic imaging display by convergence angle control up to a convergence limit and base line length control in the compound-eye imaging apparatus.

FIG. 11 is a block diagram showing a system configuration having a compound-eye imaging device according to a fourth embodiment of the present invention.

FIG. 12 shows that the parallax of the main subject in the compound eye imaging apparatus is 0.
9 is a flowchart showing the flow of the processing to be performed.

FIG. 13 is an explanatory diagram of convergence angle control up to the convergence limit and stereoscopic imaging display by parallel movement of the CCD in the compound-eye imaging apparatus.

FIG. 14 is an explanatory diagram of convergence angle control up to a convergence limit and stereoscopic imaging display by software in the compound-eye imaging apparatus according to the fifth embodiment of the present invention.

FIG. 15 is an explanatory diagram of a finder for a compound-eye imaging device and control buttons in the compound-eye imaging device according to the sixth embodiment of the present invention.

FIG. 16 is a flowchart illustrating a flow of a method of moving the imaging optical system in a plus (+) direction by a user interface in the compound-eye imaging apparatus.

FIG. 17 is a flowchart showing a flow of a method for moving the imaging optical system in the plus (-) direction by a user interface in the compound-eye imaging apparatus.

FIG. 18 is an explanatory diagram of convergence angle control up to the convergence limit and base line length control manually in the (+) direction in the compound-eye imaging apparatus.

FIG. 19 is an explanatory diagram of the convergence angle control up to the convergence limit manually and the base line length control in the (−) direction in the compound-eye imaging apparatus.

FIG. 20 is a diagram showing modules of a program code stored in a storage medium used in the compound eye imaging apparatus of the present invention.

FIG. 21 is a diagram showing modules of program codes stored in a storage medium different from that of FIG. 20 used in the compound eye imaging apparatus of the present invention.

FIG. 22 is a diagram illustrating modules of program codes stored in a storage medium different from those in FIGS. 20 and 21 used in the compound-eye imaging apparatus of the present invention.

FIG. 23 is an explanatory diagram of stereoscopic imaging display by parallel vision.

FIG. 24 is an explanatory diagram of stereoscopic imaging display by convergence angle control.

FIG. 25 is an explanatory diagram of stereoscopic imaging display by base line length control.

FIG. 26 is an explanatory diagram of stereoscopic image pickup display by CCD parallel movement in the compound eye image pickup apparatus.

[Explanation of symbols]

Reference Signs List 1 compound eye imaging device 2 personal computer 3 display device 4a imaging optical system 4b imaging optical system 5 synchronization signal generator 6a A / D converter 6b A / D converter 7 memory 8a lens 8b lens 9a CCD (imaging element) 9b CCD (imaging element) 10) Interface cable 11 Parallel interface 12 CPU (Central processing unit) 13 Memory 14 Display controller 15 Image synthesis unit 16 Image correction / overlap amount calculation unit 17 Storage device 18 Mode selection unit 19 CPU bus 20 VRAM 700 Compound eye imaging device 701a Imaging optical system 701b Imaging optical system 702a Lens 702b Lens 703a CCD (Imaging device) 703b CCD (Imaging device) 704 Signal processing unit 705 Subject position detection unit 706 Inder 707 Interface 708a Imaging optical system driving device 708b Imaging optical system driving device 709 Vergence angle / parallel movement user interface 710a CCD driving device 710b CCD driving device

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Katsuhiko Mori 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inside the corporation

Claims (40)

[Claims] 1. A combining step having a plurality of combining methods for creating one combined image from two left and right images taken by two imaging optical systems arranged on the left and right, and a switching step for switching between the plurality of combining methods. A compound eye imaging method comprising: 2. The first of the plurality of synthesizing methods is a synthesizing method 1 in which the synthesizing speed is prioritized, and the second of the plurality of synthesizing methods prioritizes the image quality of the synthesized image. 2. The method according to claim 1, wherein the combining method is a combining method 2.
The compound eye imaging method according to the above. 3. The combining method 1 combines the two captured left and right images by giving a certain amount of overlap, and the combining method 2 combines the captured left and right images into left and right luminance and color information, respectively. 3. The compound eye imaging method according to claim 2, wherein a difference and a trapezoidal distortion are corrected, and an overlapping area between the two images is detected and synthesized using an overlap amount calculated from the overlapping area. 4. The compound eye imaging method according to claim 2, wherein the switching step switches to the synthesizing method 1 in the through display mode and to the synthesizing method 2 in the recording and reproduction mode. 5. The compound eye imaging method according to claim 1, wherein the composite image is a panoramic composite image. 6. The compound eye imaging method according to claim 1, wherein the composite image is a stereoscopic image. 7. A synthesizing unit having a plurality of synthesizing methods for creating one synthesized image from two left and right images picked up by two imaging optical systems arranged on the right and left, and a switching unit for switching the plurality of synthesizing methods. A compound eye imaging apparatus comprising: 8. The first of the plurality of synthesizing methods is a synthesizing method 1 in which the synthesizing speed is prioritized, and the second of the plurality of synthesizing methods prioritizes the image quality of the synthesized image. 8. The method according to claim 7, wherein the combining method is a combining method 2.
The compound eye imaging device according to claim 1. 9. The combining method 1 combines two captured left and right images by giving a certain amount of overlap, and the combining method 2 combines the two captured left and right images with left and right luminance and color information, respectively. 8. The compound eye imaging apparatus according to claim 7, wherein a difference and a trapezoidal distortion are corrected, and further, an overlap area between the two images is detected and synthesized using an overlap amount calculated from the overlap area. 10. The compound eye imaging apparatus according to claim 7, wherein the switching unit switches to the synthesizing method 1 in the through display mode and to the synthesizing method 2 in the recording and reproducing modes. 11. The compound-eye imaging apparatus according to claim 7, wherein the composite image is a panoramic composite image. 12. The compound eye imaging apparatus according to claim 7, wherein the composite image is a stereoscopic image. 13. A compound-eye imaging method for capturing a pair of images having parallax with each other using two imaging optical systems, comprising a control step of controlling the parallax of a main subject selected from the captured images. A compound eye imaging method, characterized in that: 14. The compound eye imaging method according to claim 13, wherein the control step sets a limit value of a convergence angle of the imaging optical system. 15. The method according to claim 15, wherein the control step adjusts the parallax of the main subject in the image by moving the image to be displayed in parallel when the angle of convergence of the imaging optical system is equal to or greater than a limit value. Item 14. The compound eye imaging method according to Item 13. 16. The parallel movement refers to parallel movement of the imaging optical system, and the control step shortens a base line length of the imaging optical system when the convergence angle of the imaging optical system is equal to or more than a limit value. 16. The compound-eye imaging method according to claim 15, wherein parallax of a main subject in the image is adjusted by capturing the image. 17. The parallel movement refers to parallel movement of an image pickup device in the image pickup optical system, and the control step includes the step of controlling the position of the image pickup optical system when the convergence angle of the image pickup optical system exceeds a limit value. 2. A parallax of a main subject in an image is adjusted by moving an image sensor in parallel away from a center of the two image pickup optical systems and picking up an image.
5. The compound eye imaging method according to 5. 18. The parallel movement refers to parallel movement of the captured left and right images, and the control step includes the step of parallelizing the captured left and right images when the convergence angle of the imaging optical system is equal to or more than a limit value. 16. The compound-eye imaging method according to claim 15, wherein a parallax of a main subject in the image is adjusted by moving and creating a stereoscopically displayed image. 19. A compound-eye imaging method in which a pair of images having parallax is captured by two imaging optical systems, wherein a limit value of a convergence angle of the imaging optical system is set. A compound eye imaging method, comprising a control step of controlling the imaging optical system by parallel movement, and controlling the imaging optical system by the parallel movement when the value is equal to or larger than the limit value. 20. A compound-eye imaging apparatus that captures a pair of images having parallax with each other using two imaging optical systems, the apparatus including a control unit that controls so as to adjust the parallax of a main subject selected from the captured images. A compound eye imaging apparatus characterized by the above-mentioned. 21. The apparatus according to claim 2, wherein the control unit sets a limit value of a convergence angle of the imaging optical system.
0. The compound eye imaging device according to 0. 22. The method according to claim 22, wherein the control means adjusts the parallax of the main subject in the image by moving the image to be displayed in parallel when the angle of convergence of the imaging optical system is equal to or larger than a limit value. Item 21. The compound eye imaging device according to Item 20. 23. The parallel movement refers to parallel movement of the imaging optical system, and the control means shortens the base line length of the imaging optical system when the angle of convergence of the imaging optical system is equal to or larger than a limit value. 23. The compound-eye imaging apparatus according to claim 22, wherein the parallax of the main subject in the image is adjusted by capturing the image. 24. The parallel movement refers to parallel movement of an image pickup device in the image pickup optical system, and the control means controls the image pickup optical system when the angle of convergence of the image pickup optical system is equal to or larger than a limit value. 23. The parallax of a main subject in an image is adjusted by moving the image sensor in parallel away from the center of the two image pickup optical systems and picking up an image.
The compound eye imaging device according to claim 1. 25. The parallel movement refers to parallel movement of the captured left and right images, and the control means parallelizes the captured left and right images when the convergence angle of the imaging optical system is equal to or larger than a limit value. 23. The compound-eye imaging apparatus according to claim 22, wherein a parallax of a main subject in the image is adjusted by moving to create a stereoscopically displayed image. 26. A limit value setting means for setting a limit value of a convergence angle of the image pickup optical system in a compound-eye image pickup apparatus for picking up a set of images having parallax with each other using two image pickup optical systems. Control means for controlling the imaging optical system by the convergence angle or the parallel movement below the limit value set by the means, and controlling the imaging optical system by the parallel movement above the limit value. Compound eye imaging device. 27. The compound eye imaging apparatus according to claim 26, wherein said control means is a user interface. 28. A storage medium for storing a program for controlling a compound-eye imaging device, comprising: a plurality of combining methods for creating one combined image from two left and right images taken by two imaging optical systems arranged on the left and right. A storage medium storing a program having a synthesizing module having: and a switching module for switching between the plurality of synthesizing methods. 29. The first of the plurality of synthesizing methods is a synthesizing method 1 in which the synthesizing speed is prioritized, and the second of the plurality of synthesizing methods prioritizes the image quality of the synthesized image. 29. The storage medium according to claim 28, which is a synthesis method 2 for performing synthesis by combining. 30. The synthesizing method 1 synthesizes two captured left and right images by giving a certain amount of overlap, and the synthesizing method 2 synthesizes the two captured left and right images with left and right luminance and color information, respectively. 30. The storage medium according to claim 29, wherein a difference or a trapezoidal distortion is corrected, and furthermore, an overlap area between two images is detected and synthesized using an overlap amount calculated from the overlap area. 31. The storage medium according to claim 28, wherein the switching module switches to the synthesizing method 1 in the through display mode and to the synthesizing method 2 in the recording and reproducing mode. 32. The storage medium according to claim 28, wherein the composite image is a panoramic composite image. 33. The storage medium according to claim 28, wherein the composite image is a stereoscopic image. 34. A storage medium for storing a program for controlling a compound-eye imaging apparatus that captures a pair of images having parallax by two imaging optical systems, the parallax of a main subject selected from the captured images. A storage medium storing a program having a control module for controlling so as to adjust the storage medium. 35. The storage medium according to claim 34, wherein the control module sets a limit value of a convergence angle of the imaging optical system. 36. The control module according to claim 36, wherein, when the convergence angle of the imaging optical system is equal to or larger than a limit value, the display module adjusts the parallax of the main subject in the image by translating the image to be displayed. Item 34. The storage medium according to Item 34. 37. The parallel movement refers to moving the imaging optical system in parallel, and the control module shortens the base line length of the imaging optical system when the convergence angle of the imaging optical system is equal to or more than a limit value. 37. The storage medium according to claim 36, wherein parallax of a main subject in the image is adjusted by capturing the image. 38. The translation refers to translating an image sensor in the imaging optical system, and the control module determines that the imaging optical system has a convergence angle equal to or greater than a limit value of the convergence angle of the imaging optical system. 37. The storage medium according to claim 36, wherein parallax of a main subject in an image is adjusted by moving an image sensor in parallel away from a center of the two image capturing optical systems to capture an image. 39. The translation refers to translating the captured left and right images, and the control module includes:
When the convergence angle of the imaging optical system is equal to or more than the limit value, the parallax of the main subject in the image is adjusted by moving the captured left and right images in parallel to create a stereoscopically captured display image. The storage medium according to claim 36. 40. A storage medium for storing a program for controlling a compound-eye imaging apparatus that captures a pair of images having parallax by two imaging optical systems, wherein a limit value of a convergence angle of the imaging optical system is set. A limit value setting module,
A control module that controls the imaging optical system by the convergence angle or the translation below the limit value set by the limit value setting module, and controls the imaging optical system by the translation above the limit value. A storage medium characterized by the above-mentioned.