WO2012140919A1

WO2012140919A1 - Stereo image generation device and stereo image generation method

Info

Publication number: WO2012140919A1
Application number: PCT/JP2012/002602
Authority: WO
Inventors: 孝幸有馬; 東郷　仁麿; 和之田中; 井村　康治; 中村　剛; 郁雄渕上
Original assignee: パナソニック株式会社
Priority date: 2011-04-14
Filing date: 2012-04-13
Publication date: 2012-10-18

Abstract

A first image capture unit (10-1) has an optical zoom function and outputs a high-pixel first captured image. A second image capture unit (10-2) has an electronic zoom function and outputs a second captured image which has fewer pixels than the first image capture unit (10-1). When the zoom magnification is less than or equal to a prescribed magnification, an image resizing unit (24) converts the first and second captured images to images having a third number of pixels, and when the zoom magnification exceeds the prescribed magnification, the image resize unit (24) converts the first and second captured images to images having a fourth number of pixels. An image encoding unit (27) encodes each respective first and second captured image converted by the image resize unit (24). A stereo image format conversion unit (28) converts each respective first and second captured image encoded by the image encode unit (27) to a stereo image format and generates a stereo image.

Description

Stereo image generating apparatus and stereo image generating method

The present invention relates to a stereo image generating apparatus and a stereo image generating method suitable for use in a compound-eye camera for capturing a stereo image that captures a stereo image.

As a method of generating a stereo image, a compound eye camera and an image processing method described in Patent Document 1 are known. The compound-eye camera described in Patent Document 1 includes a plurality of imaging means (i.e., cameras) having a maximum number of pixels that can be photographed and an angle of view, and one camera has an angle of view compared to the other camera. Is narrow, and the maximum number of pixels that can be shot is small. When capturing a stereo image, the image captured by a camera with a high number of pixels is reduced or cut out according to the number of pixels of the image captured with a camera with a low number of pixels and the angle of view. The number of pixels of the image and the angle of view are combined to generate a stereo image.

Japanese Unexamined Patent Publication No. 2010-252186

However, in the configuration of the compound-eye camera described in Patent Document 1 described above, one of the cameras with a large number of pixels is used to reduce the number of pixels of an image captured by a camera with a large number of pixels. However, the number of pixels of a stereo image that can be photographed is limited to the number of pixels of a camera with a low number of pixels.

In addition, when a stereo image is taken with a camera with different resolution, such as a low resolution camera and a high resolution camera, there is also a problem that stereoscopic vision can not be performed sufficiently (that is, it is difficult to achieve stereoscopic vision).

The present invention has been made in view of the above circumstances, and a stereo image of high pixels can be taken without using two high pixel number imaging means, and a stereo image that can be sufficiently stereoscopically viewed. An object of the present invention is to provide a generating device and a stereo image generating method.

A stereo image generating device according to the present invention has a first number of pixels, has an optical zoom function, and has a first imaging unit for picking up a first captured image, and a second smaller than the first number of pixels. A second imaging unit configured to capture a second captured image having the number of pixels and having an electronic zoom function, and a resolution adjustment unit configured to adjust a difference in resolution between right and left based on a zoom magnification designated in stereo image shooting; Equipped with

According to the above configuration, two imaging units with different resolutions pick up two images, and adjust the left and right resolution differences based on the specified zoom magnification, so two imaging units with high pixels can be used. Therefore, the cost can be reduced and the mounting area can be reduced.

In the above configuration, the first imaging unit performs an optical zoom operation based on the zoom magnification specified in stereo image shooting, the second imaging unit performs an electronic zoom operation, and the zoom magnification is a predetermined magnification. At the following times, when the first and second captured images are converted to the third pixel number and the predetermined magnification is exceeded, the first and second captured images are processed. An image resizing unit configured to convert the image data into a fourth pixel number smaller than the third pixel number; an image encoding unit configured to encode the first captured image and the second captured image converted by the image resizing unit; A stereo image file for converting the first captured image and the second captured image encoded by the image encoding unit into a stereo image format to generate a stereo image And Tsu door conversion unit, equipped with.

According to the above configuration, the first imaging unit having high pixels and the second imaging unit having pixels lower than the first imaging unit are provided, and when the optical zoom magnification is equal to or less than the predetermined magnification, the first imaging unit is The first captured image captured by the imaging unit and the second captured image captured by the second imaging unit have a third pixel count, specifically, a pixel count higher than the pixel count of the second imaging unit. The first captured image and the enlarged second captured image are encoded, and the encoded first and second captured images are converted into a stereo image format to generate a stereo image.

When the optical zoom magnification exceeds the predetermined magnification, the first captured image captured by the first imaging unit and the second captured image captured by the second imaging unit are used as the third pixel count. The resized and resized first captured image and second captured image to a smaller fourth pixel number, and the encoded first captured image and second captured image are converted to a stereo image format Generate a stereo image.

With such a configuration, when the optical zoom magnification is equal to or less than the predetermined magnification, the number of pixels of the captured image of the low pixel imaging unit is higher than that of the low pixel imaging unit without using two high pixel imaging units. It becomes possible to capture a stereo image.

In addition, even when the optical zoom magnification exceeds the predetermined magnification, the number of pixels of the stereo image generated when the optical zoom magnification is equal to or less than the predetermined magnification is smaller than the first captured image and the second While suppressing the resolution of the captured image to a certain ratio or less, it is possible to capture a stereo image that has a pixel size higher than the number of pixels of the captured image of the low pixel imaging unit and that is easy to view stereoscopically.

Furthermore, since two high-pixel imaging means are not used and two imaging means having an optical zoom function are not used, the cost can be reduced and the mounting area can be reduced.

In the above configuration, the conversion of the first captured image and the conversion of the second captured image, which are performed by the image resizing unit, are enlargement, reduction, or equal-scale conversion.

According to the configuration, it is possible to match the number of pixels of the first captured image captured by the first imaging unit with the number of pixels of the second captured image captured by the second imaging unit.

In the above configuration, the image processing apparatus further includes a zoom magnification threshold storage unit configured to hold a predetermined magnification of the optical zoom magnification as a zoom magnification threshold, and the image resizing section includes the first captured image and the second imaged image based on the zoom magnification threshold. Resize the captured image of.

According to the above configuration, the difference in resolution between the first captured image and the second captured image can be limited to within a certain fixed value.

In the above configuration, the third number of pixels is the first number of pixels.

According to the above configuration, it is possible to match the number of pixels of the second captured image captured by the second imaging unit with the number of pixels of the first captured image captured by the first imaging unit.

In the above configuration, the first imaging unit includes a first angle of view, and the second imaging unit includes a second angle of view narrower than the first angle of view, and the first imaging is performed. An angle of view correction unit that trims the first or second captured image according to the optical zoom magnification of the unit; the image encoding unit is trimmed by the angle of view correction unit; and the image resizing unit The first captured image in which the number of pixels is converted and the second captured image in which the number of pixels is converted in the image resizing unit are encoded.

According to the above configuration, by providing the angle of view correction unit, the first captured image can be made smaller.

In the above configuration, the image capturing apparatus further includes an angle of view information holding unit that holds angle of view correction information for each optical zoom magnification of the first imaging unit, and the angle of view correction unit trims based on the angle of view correction information.

According to the above configuration, by providing the angle-of-view information holding unit that holds the angle-of-view correction information, it is possible to shorten the processing time in trimming.

In the above configuration, a position correction unit that corrects at least one of vertical shift, horizontal shift, rotational shift, or size shift with respect to either the first captured image or the second captured image. The image encoding unit is corrected by the position correction unit, and the first resized image is converted by the image resizing unit to the predetermined number of pixels, and the image correction unit is corrected by the position correction unit. And encode the second captured image converted into the predetermined number of pixels.

According to the above configuration, the position correction unit corrects at least one of vertical deviation, horizontal deviation, rotational deviation, or size deviation with respect to either the first captured image or the second captured image. Thus, it is possible to generate a high-precision stereo image from the first captured image and the second captured image.

In the above configuration, the image pickup apparatus further includes a position correction information holding unit that holds position correction information for each optical zoom magnification of the first imaging unit, and the position correction unit is vertically offset, horizontal offset, or rotational offset based on the position correction information. Or correct at least one of the size deviations.

According to the above configuration, by providing the position correction information holding unit that holds the position correction information, it is possible to shorten the processing time in the position correction.

In the above configuration, a color for performing color correction on the first captured image and / or the second captured image with respect to the difference in color between the first captured image and the second captured image. A correction unit is provided, and the image encoding unit encodes the first captured image whose color is corrected by the color correction unit and the second captured image.

According to the above configuration, color correction of the first captured image and / or the second captured image can be performed by providing the color correction unit.

In the above configuration, when the conversion of the first captured image and the conversion of the second captured image performed by the image resizing unit are enlargement conversions, super-resolution processing is used.

According to the above configuration, by performing the super-resolution processing, it is possible to increase the sense of sharpness even if a blurred portion is generated by enlarging the captured image. That is, high image quality can be achieved.

In the above configuration, an image filter unit that performs averaging processing on peripheral portions of respective images of the first captured image and the second captured image is provided, and the image encoding unit includes pixels in the image resizing unit. After the number conversion is performed, the first captured image subjected to the averaging process by the image filter unit and the pixel number conversion performed by the image resizing unit are averaged by the image filter unit. And encoding the second captured image subjected to the processing.

According to the above configuration, by providing the image filter unit, it is possible to blur the periphery of each of the first captured image and the second captured image, and between the first and second imaging units. It is possible to reduce the deterioration of the image quality of the stereo image due to the difference in lens distortion.

In the above configuration, the magnitude relationship between the first pixel number, the third pixel number, and the fourth pixel number is: first pixel number ≧ third pixel number> fourth pixel number.

A stereo image generating method according to the present invention includes a first imaging step of capturing a first captured image with a first number of pixels in a first imaging unit having an optical zoom function, and a second imaging step having an electronic zoom function. A second imaging step for imaging a second captured image with a second number of pixels smaller than the first number of pixels in the imaging unit, and when the designated zoom magnification is equal to or less than a predetermined magnification, The first captured image and the second captured image are converted to the third pixel number, and when the predetermined magnification is exceeded, the first captured image and the second captured image are converted to the third An image resizing step for converting the number of pixels to a fourth number of pixels smaller than the number of pixels; an image encoding step for encoding the first captured image and the second captured image converted in the image resizing step; Enko And stereo image format conversion step of generating a stereo image by converting the first captured image and the second captured image encoded in de step to the stereo image format, with a.

According to the above method, using the first imaging unit with high pixels and the second imaging unit with pixels lower than the first imaging unit, the first imaged image imaged by the first imaging unit To a predetermined number of pixels, and the second picked-up image picked up by the second image pickup unit to a predetermined number of pixels, and thus the resized first picked-up image and the second picked-up image are encoded And convert the encoded first and second captured images into a stereo image format to generate a stereo image, so that a high-pixel stereo image is captured without using two high-pixel imaging means. It is possible to

In addition, even when the optical zoom magnification exceeds the predetermined magnification, the number of pixels of the stereo image generated when the optical zoom magnification is equal to or less than the predetermined magnification is smaller than the number of pixels of the stereo image. While suppressing the resolution of the image to a certain ratio or less, it is possible to capture a stereo image that is higher in pixels than the number of pixels of the captured image of the low pixel imaging unit and that is easy to view stereoscopically.

In the above configuration, the first imaging unit has both an optical zoom function and an electronic zoom function, and in stereo image shooting, the first imaging unit performs an optical zoom operation and an electronic operation based on a zoom magnification designated. The zoom operation is performed, the second imaging unit performs the electronic zoom operation, and the first imaging unit performs the optical zoom operation when the zoom magnification is equal to or less than a predetermined magnification smaller than the upper limit magnification of the optical zoom function. The second imaging unit performs the electronic zoom operation, and when the predetermined magnification is exceeded, the subsequent zoom operations perform the electronic zoom operation with the first imaging unit and the second imaging unit. Image pickup control unit for controlling the image pickup control unit, an image resizing unit for converting the first picked-up image and the second picked-up image into a predetermined number of pixels, and the first picked-up image converted by the image resizing unit. An image encoding unit encoding the image and the second captured image; and converting the first captured image and the second captured image encoded by the image encoding unit into a stereo image format to generate a stereo image And a stereo image format converter.

According to the above configuration, the first imaging unit having high pixels and the second imaging unit having pixels lower than the first imaging unit are provided, and the zoom magnification is a predetermined magnification smaller than the upper limit magnification of the optical zoom function. In the following cases, the first captured image captured by the first imaging unit by the optical zoom and the second captured image captured by the electronic zoom by the second imaging unit have a predetermined number of pixels. The resized and encoded first captured image and the enlarged second captured image are converted, and the encoded first and second captured images are converted to a stereo image format to generate a stereo image.

When the zoom magnification exceeds a predetermined magnification smaller than the upper limit magnification of the optical zoom function, the first imaging unit optically zooms to a predetermined magnification, and the zoom magnification thereafter is imaged by the electronic zoom Resizing the first captured image and the second captured image captured by the electronic zoom in the second imaging unit to a predetermined number of pixels, and encoding the resized first captured image and the second captured image; The encoded first and second captured images are converted into a stereo image format to generate a stereo image.

With such a configuration, when the optical zoom magnification is equal to or less than a predetermined magnification smaller than the upper limit magnification of the optical zoom function, the imaging of the low pixel imaging portion is performed without using two high pixel imaging means. It is possible to capture a stereo image with a pixel higher than the number of pixels of the image.

In addition, even when the optical zoom magnification exceeds a predetermined magnification, the resolution of the first captured image and the second captured image can be suppressed to a certain ratio or less, and a stereoscopic image which is easy to view stereoscopically is captured. be able to. Moreover, it becomes possible to capture a stereo image of a pixel higher than the number of pixels of a captured image of a low pixel imaging unit, and also possible to perform stereo photography by zooming.

In the above configuration, the zoom magnification threshold storage unit stores the switching magnification of the zoom magnification as a zoom magnification threshold, and the imaging control unit is configured to capture the first imaging unit and the second imaging based on the zoom magnification threshold. Switch control of the zoom operation of the unit.

According to the above configuration, the imaging control unit performs switching control of the zoom operation of the first imaging unit and the second imaging unit based on the zoom magnification threshold value, whereby the first imaging image and the second imaging image are displayed. The difference in resolution can be limited within a certain value.

In the above configuration, the predetermined number of pixels is the first number of pixels.

In the above configuration, the first imaging unit includes a first angle of view, and the second imaging unit includes a second angle of view narrower than the first angle of view, and the first imaging is performed. The image encoding unit includes the first captured image that has been trimmed by the angle of view correction unit and has the number of pixels converted by the image resizing unit, and the image resizing unit. And encoding the second captured image in which the number of pixels has been converted.

In the above configuration, a position correction unit that corrects at least one of vertical shift, horizontal shift, rotational shift, or size shift with respect to either the first captured image or the second captured image. The image encoding unit is corrected by the position correction unit, and the first resized image is converted by the image resizing unit to the predetermined number of pixels, and the image correction unit is corrected by the position correction unit. And encode the second captured image converted into a predetermined number of pixels.

In the above configuration, color correction is performed to correct the color of the first captured image and / or the second captured image with respect to the difference in color between the first captured image and the second captured image. The image encoding unit encodes the first captured image whose color is corrected by the color correction unit and the second captured image.

In the above configuration, an image filter unit that performs averaging processing on peripheral portions of respective images of the first captured image and the second captured image is provided, and the image encoding unit includes pixels in the image resizing unit. The first captured image subjected to averaging processing by the image filter unit after conversion of the number, and the number of pixels converted by the image resizing unit, and the averaging processing performed by the image filter unit 2 and the captured image are encoded.

According to the above configuration, by providing the image filter unit, the peripheral portion of each of the first captured image and the second captured image can be blurred, and a lens between the first and second imaging units It is possible to reduce the deterioration of the image quality of the stereo image due to the difference in distortion.

A stereo image generating method according to the present invention comprises: a first imaging step of imaging a first captured image with a first number of pixels in a first imaging unit having an optical zoom function and an electronic zoom function;
A second imaging step for imaging a second captured image with a second number of pixels smaller than the first number of pixels in a second imaging unit having an electronic zoom function; When the magnification is smaller than the upper limit magnification of the zoom function, the first imaging unit performs the optical zoom operation, and the second imaging unit performs the electronic zoom operation. When the magnification exceeds the predetermined magnification, In the subsequent zoom operation, the first image pickup unit and the second image pickup unit perform a zoom control step to perform electronic zoom operation, the first image pickup image, and the second image pickup image. An image resizing step of converting into a predetermined number of pixels; an image encoding step of encoding the first captured image and the second captured image converted in the image resizing step; And stereo image format conversion step of generating a stereo image by converting the encoded first captured image and the second captured image to the stereo image format in Dosuteppu, with a.

According to the above method, using the first imaging unit with high pixels and the second imaging unit with pixels lower than the first imaging unit, the first imaged image imaged by the first imaging unit To a predetermined number of pixels, and the second captured image captured by the second imaging unit to a predetermined number of pixels, and thus the resized first captured image and the second captured image are encoded. Since the encoded first captured image and the second captured image are converted to a stereo image format to generate a stereo image, a high pixel stereo image is captured without using two high pixel imaging means. It becomes possible.

In addition, even when the zoom magnification exceeds a predetermined magnification, by suppressing the resolution of the first captured image and the second captured image to a certain ratio or less, a high-pixel stereo image that is easy to view stereoscopically is captured. And zoom operation is also possible.

According to the present invention, since two images are captured by two imaging means having different resolutions and the left-right resolution difference is adjusted based on the designated zoom magnification, two high-pixel imaging means are used. Therefore, the cost can be reduced and the mounting area can be reduced.

Block diagram showing a schematic configuration of a stereo image generating apparatus according to Embodiment 1 of the present invention A flowchart showing the entire photographing process of the stereo image generating apparatus of FIG. 1 The flowchart which shows the photographing processing of optical zoom magnification below predetermined magnification of the stereo image generation device of Drawing 1 The figure which shows the process result in each step of step S30-step S32 of imaging processing of FIG. 3, and step S34. A flowchart showing photographing processing when the optical zoom magnification exceeds a predetermined magnification in the stereo image generation device of FIG. 1 The figure which shows the processing result in each step of step S30-step S32 of imaging processing of FIG. 5, and step S40. (A)-(g) The figure which shows an optical axis shift when two imaging parts are arrange | positioned to the stereo image production | generation apparatus of FIG. Block diagram showing a schematic configuration of a stereo image generating apparatus according to Embodiment 2 of the present invention A flowchart showing the entire photographing process of the stereo image generating apparatus of FIG. 8 A flowchart showing the entire photographing process of the stereo image generating apparatus of FIG. 8 A flowchart showing photographing processing of the stereo image generating apparatus of FIG. 8 The figure which shows the process result in each step of step S80-step S82 of the imaging | photography process of FIG. 11, and step S84. The figure which shows the process result in each step of step S80-step S82 of the imaging | photography process of FIG. 11, and step S84.

Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the drawings.

Embodiment 1
FIG. 1 is a block diagram showing a schematic configuration of a stereo image generating apparatus according to Embodiment 1 of the present invention. In FIG. 1, the stereo image generating device 1 according to the first embodiment includes a high pixel imaging unit 10-1 having an optical zoom function, a low pixel imaging unit 10-2 having an electronic zoom function, and zoom control. The unit 11, the autofocus control unit 12, the shutter control unit 13, the exposure control unit 14, the white balance control unit 15, the camera operation unit 16, the control unit 17, and two camera signal processing units 18-1 18-2, the angle of view correction unit 19, the angle of view information holding unit 20, the position correction unit 21, the position correction information holding unit 22, the color correction unit 23, the image resizing unit 24, and super resolution Processing unit 25, image filter unit 26, image encoding unit 27, stereo image format conversion unit 28, image storage unit 29, format conversion unit 30, display control unit 31, display unit 32, imaging control Part 40 , And a zoom magnification threshold storage unit 41.

The imaging units 10-1 and 10-2 each include an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and output an imaging signal. In the first embodiment, an image obtained by imaging by the imaging unit 10-1 is taken as a first captured image, and an image obtained by imaging by the imaging unit 10-2 is taken as a second captured image. In the first embodiment, the imaging unit 10-1 has a first angle of view, and the imaging unit 10-2 has a second image narrower than the first angle of view of the imaging unit 10-1. It has a horn. The imaging unit 10-1 also has an optical zoom function. The imaging unit 10-1 may further include an electronic zoom function. On the other hand, the imaging unit 10-2 does not have an optical zoom function, but only an electronic zoom function.

The zoom control unit 11 performs zoom control of the imaging units 10-1 and 10-2. The autofocus control unit 12 performs autofocus control of the imaging units 10-1 and 10-2. The shutter control unit 13 performs shutter control of the imaging units 10-1 and 10-2. The exposure control unit 14 performs exposure control of the imaging units 10-1 and 10-2. The white balance control unit 15 performs white balance control of the imaging units 10-1 and 10-2.

The camera operation unit 16 is for operating the stereo image generating device 1, and a signal at the time of operation such as a zoom magnification designated by the operator is input to the control unit 17. The control unit 17 outputs an operation signal corresponding to the operation in the camera operation unit 16 to the imaging units 10-1 and 10-2. The camera signal processing unit 18-1 resizes the first captured image captured by the imaging unit 10-1 to the size of the display unit 32 in the preview processing. The camera signal processing unit 18-2 resizes the second captured image captured by the imaging unit 10-2 to the size of the display unit 32 in the preview processing. In the first embodiment, at the time of shooting processing, the camera signal processing unit 18-1 outputs a first captured image (L: left image) of 9.7 megapixels, and the camera signal processing unit 18-2 outputs It is assumed that a second captured image (R: right image) of 2 megapixels is output.

The view angle correction unit 19 reduces the view angle of the first captured image captured by the imaging unit 10-1 based on the view angle correction information stored in the view angle information storage unit 20. For example, the field angle is 5.7 megapixels. The angle of view of the second captured image captured by the imaging unit 10-2 may be corrected. The position correction unit 21 corrects the position of the first captured image captured by the imaging unit 10-1 based on the position correction information stored in the position correction information storage unit 22. In this case, at least one of “vertical shift”, “horizontal shift”, “rotational shift”, or “size shift” is performed as the position correction. The position correction may be performed on the second captured image captured by the imaging unit 10-2. That is, even if the position correction unit 21 performs position correction on either the first captured image captured by the imaging unit 10-1 or the second captured image captured by the imaging unit 10-2. Good.

The color correction unit 23 performs the first color difference between the second captured image captured by the imaging unit 10-2 and the first captured image captured by the imaging unit 10-1 after position correction. Color correction is performed on the captured image and the second captured image (the first captured image or the second captured image). The image resizing unit 24 converts the first captured image captured by the imaging unit 10-1 into a predetermined number of pixels, and converts the second captured image captured by the imaging unit 10-2. The number of pixels is the same as that of the first captured image.

In addition, the image resizing unit 24 converts the first captured image captured by the imaging unit 10-1 into a predetermined number of pixels based on the zoom magnification threshold stored in the zoom magnification threshold storage unit 41, and captures an image. The second captured image captured by the unit 10-2 is also converted into a predetermined number of pixels. Specifically, when the zoom magnification is equal to or less than a predetermined magnification, the first captured image and the second captured image are converted into a third number of pixels, and the predetermined magnification is exceeded. The first captured image and the second captured image are converted into a fourth pixel number. Note that the conversion of the first captured image and the conversion of the second captured image performed by the image resizing unit 24 include “equal magnification” in addition to “enlargement” and “reduction”.

When the conversion of the second captured image is enlargement conversion, the super-resolution processing unit 25 performs enlargement processing by super-resolution processing without performing enlargement processing in the image resizing unit 24. It is possible to select whether to perform enlargement processing by the resize unit 24 or enlargement by super resolution processing. The super-resolution processing is processing for enhancing the sharpness of the blurred portion by enlarging the image (that is, processing for achieving high image quality).

In the first embodiment, the case where enlargement processing is performed by the image resizing unit 24 instead of the super-resolution processing unit 25 will be described. The image filter unit 26 reduces the pixel shift of the left and right images due to the difference in lens distortion of the two imaging units 10-1 and 10-2 so that the images of the first and second captured images are not displayed. Perform averaging processing to blur the periphery. By providing the image filter unit 26, it is possible to reduce the deterioration of the image quality of the stereo image due to the difference in lens distortion between the two imaging units 10-1 and 10-2.

In the image encoding unit 27, conversion of the number of pixels is performed by the image resizing unit 24, and the first captured image subjected to averaging processing by the image filter unit 26 and conversion of the number of pixels by the image resizing unit 24 are performed. Furthermore, the second captured image subjected to the averaging process in the image filter unit 26 is encoded (compressed). In this case, for example, compression is performed in JPEG (Joint Photographic Experts Group) format. The stereo image format conversion unit 28 converts the first captured image and the second captured image encoded by the image encoding unit 27 into a stereo image format to generate a stereo image. In this case, for example, the JPEG format is converted to an MPO (Multi-Picture Format) format. The image storage unit 29 has a flash memory (SD memory or the like), and stores the stereo image generated by the stereo image format conversion unit 28. The format conversion unit 30 converts the first captured image and the second captured image converted into the number of pixels matching the display unit 32 by the image resizing unit 24 into a format for displaying on the display unit 32. The display control unit 31 controls the display unit 32 to display the first captured image and the second captured image converted to a format (for example, pixel by pixel) to be displayed on the display unit 32 by the format conversion unit 30. I do. The display unit 32 is, for example, a liquid crystal display.

Next, the operation of the stereo image generating device 1 of the first embodiment will be described.
In FIG. 2, the user instructs activation of the imaging units 10-1 and 10-2 from the camera operation unit 16 (step S10). Based on this user instruction, the control unit 17 outputs an instruction to activate the imaging units 10-1 and 10-2 to the imaging control unit 40 (step S11). The control unit 17 sets the optical zoom magnification of the imaging unit 10-1 to 1 × for the imaging control unit 40 (step S12).

After the start instruction is output, steps S14 to S21 are repeated until the user instructs the camera operation unit 16 to end shooting. The determination for repeating the process of step S14 to step S21 is performed in step S13 and step S22.

The control unit 17 detects whether the user has performed control of the camera by the camera operation unit 16 (step S14). If not detected, the processes of steps S13, S14 and S22 are repeated. On the other hand, when it is detected, the control unit 17 determines whether it is a photographing instruction (step S15). If it is the shooting instruction, the control unit 17 determines whether the current optical zoom magnification is equal to or less than the zoom magnification threshold read from the zoom magnification threshold storage unit 41 (step S16). If the determination in step S16 is "Yes", the first photographing process is performed (step S18). The details of the shooting process No. 1 will be described later. On the other hand, if the determination in step S16 is "No", the second photographing process is performed (step S19). The details of the shooting process 2 will be described later.

If the determination in step S15 is "No", the control unit 17 determines whether a zoom magnification change instruction has been issued (step S17). If it is the zoom magnification change instruction (that is, if the determination in step S17 is "Yes"), the control unit 17 instructs the imaging control unit 40 to change the zoom magnification, and the imaging control unit 40 receives the imaging unit 10 An instruction to change the optical zoom magnification is issued to -1 (step S20). On the other hand, the imaging control unit 40 designates the trimming position of the image according to the zoom magnification to the camera signal processing unit 18-2. The trimming here is the electronic zoom function. If it is an instruction other than the zoom magnification change (that is, if the determination in step S17 is "No"), control according to that is performed (step S21). For example, the process of changing the size of a captured image is performed.

The above process is repeated until the user instructs the end of shooting by the camera operation unit 16 (step S22). Then, when the user instructs the end of imaging, the control unit 17 stops the operation of each of the imaging units 10-1 and 10-2 (step S23), and then the present process ends.

(Shooting process 1)
Next, the first photographing processing of step S18 when the zoom magnification is equal to or less than the zoom magnification threshold will be described. In the following description, it is assumed that the zoom magnification threshold is 1.8. Here, the case where the optical zoom magnification at the time of shooting of the imaging unit 10-1 is 1.5 will be described as an example.

FIG. 3 is a flowchart for explaining the photographing process 1 in FIG. In FIG. 3, the camera signal processing unit 18-1 resizes the first captured image captured by the imaging unit 10-1 to the image size set by the user, while the camera signal processing unit 18-2 resizes the imaging unit. The second captured image captured by 10-2 is trimmed in accordance with the user-set electronic zoom magnification, and resized in accordance with the user-set image size (step S30).

Next, the field angle correction unit 19 reads out the field angle correction information from the field angle information holding unit 20, and trims the first captured image captured by the imaging unit 10-1 (step S31). Next, the position correction unit 21 reads out the position correction information from the position correction information holding unit 22, and the “vertical shift”, “horizontal shift”, “rotation” with respect to the first captured image captured by the imaging unit 10-1 At least one of the deviations "or the size deviation" is corrected (step S32).

Next, the color correction unit 23 corrects the difference in color of the left and right images (step S33). That is, the difference in color between the second captured image captured by the imaging unit 10-2 and the first captured image captured by the imaging unit 10-1 after position correction is corrected. Next, the control unit 17 compares the current optical zoom magnification with the zoom magnification threshold stored in the zoom magnification threshold storage unit 41, and the current zoom magnification is equal to or less than the zoom magnification threshold. The size of the first captured image after the angle of view correction is indicated as the third number of pixels, which is the number of pixels after resizing, and the image resizing unit 24 is adjusted to the size of the first captured image after the angle of view correction. The second captured image captured by the imaging unit 10-2 is enlarged (step S34). Next, the image encoding unit 27 encodes the left and right images (for example, compresses them into the JPEG format). That is, the second captured image captured by the imaging unit 10-2 and the first captured image captured by the imaging unit 10-1 after position correction are encoded (step S35). Next, the stereo image format conversion unit 28 converts the left and right images encoded by the image encoding unit 27, that is, the first captured image and the second captured image into a stereo image format (step S36). This process ends when the left and right images are converted to the stereo image format.

FIG. 4 is a diagram showing the processing result in each of steps S30 to S32 and step S34 when the specifications of the imaging units 10-1 and 10-2 are determined as follows.
Imaging unit 10-1: 13 megapixel camera, focal length 27 mm equivalent (35 mm film equivalent), 3 × optical zoom function, captured image size 9.7 megapixel (aspect ratio 16: 9)
Imaging unit 10-2: 3 megapixel camera, 35 mm focal length equivalent (35 mm film equivalent), electronic zoom function, 2 megapixel captured image size (aspect ratio 16: 9)
Here, the case where the optical zoom magnification of the imaging unit 10-1 at the time of shooting is 1.5 times will be described as an example.

In step S30, the first captured image is H2336 pixels and W4160 pixels. The second captured image is H1080 pixels and W1920 pixels. In step S31, the first captured image is approximately H1796 pixels and approximately W3200 pixels. Here, due to the difference in the angle of view of the focal lengths of 35 mm and 27 mm, it is possible to shoot at a wide angle of approximately 1.3 times in both the vertical and horizontal directions when shooting with a camera of 27 mm. Therefore, when the camera image of 27 mm is trimmed by the angle of view correction unit 19, the number of pixels becomes about 1 / 1.3. However, even with a camera with the same number of pixels, the image size after trimming varies because of variations in the angle of view.
In the first embodiment, the first captured image is trimmed to match the angle of view of the second captured image, but the pixel is higher than the trimming size according to the zoom magnification of the second captured image. A method of adjusting to the angle of view of the first captured image may be used by trimming with numbers. Specifically, in the case of the zoom magnification of 1.5, the following is obtained with respect to the image of H1080 pixel and W1920 pixel which is the second captured image.
1080 / 1.5 * 1.3 = 936
1920 / 1.5 * 1.3 = 1664
That is, it is possible to match the angle of view with the first captured image by trimming the image of H936 pixels and W1664 pixels for the second captured image. In that case, it is not necessary to trim the first captured image in step S31, and the first captured image remains H2336 pixels and W4160 pixels. In the following description, processing in the case of trimming the first captured image will be described.

In step S31, the second captured image remains H1080 pixels and W1920 pixels. In step S32, the first captured image is H (1796-β) pixels and W (3200-α) pixels. The second captured image remains H1080 pixels and W1920 pixels. In step S34, the first captured image is H (1796-β) pixels and W (3200-α) pixels. The second captured image is H (1796-β) pixels and W (3200-α) pixels, and has the same number of pixels as the first captured image. Here, the meanings of α and β are reduced by correcting the shift of the optical axis when the two cameras are arranged in the stereo image generating device, the shift of the angle of view of each camera itself, and the shift of the optical axis. It shows the number of pixels.

Here, for reference, FIGS. 7A to 7G show the optical axis deviation that occurs when two cameras are arranged in the stereo image generating device 1. FIG.

(Shooting process 2)
Next, the second photographing process of step S19 when the zoom magnification is larger than the zoom magnification threshold will be described. Here, the case where the optical zoom magnification of the imaging unit 10-1 at the time of shooting is doubled will be described as an example.

FIG. 5 is a flowchart for explaining the photographing process 2 in FIG. In FIG. 5, steps S30 to S33 are the same processes as the photographing process 1 described above. Next, the control unit 17 compares the current optical zoom magnification with the zoom magnification threshold, and the current optical zoom magnification is twice and larger than the zoom magnification threshold 1.8 times, so resizing with respect to the image resizing unit 24 The image resizing unit 24 resizes the first captured image to the third number of pixels and resizes the second captured image to the third number of pixels by indicating the third number of pixels, which is the number of subsequent pixels (see FIG. Step S40). Here, as a method of determining the third number of pixels, the third number of pixels is set so that the ratio of the number of pixels during trimming by the electronic zoom in step S30 and the number of pixels after resizing in step S40 is equal to or less than a certain ratio. decide. In this embodiment, as shown in Equation 1 and Equation 2, the ratio of the number of pixels is set to 6.25 times (2.5 times each in the horizontal and vertical length ratios of the image) or less. The number of pixels of was determined.
Horizontal pixel count ratio 2400/960 = 2.5 ... Formula 1
Vertical pixel count ratio 1350/540 = 2.5 ... Formula 2
However,
2400: The number of pixels in the horizontal direction 1350 of the second captured image in step S40: The number of pixels in the vertical direction of the second captured image in step S40 960: With respect to the second captured image in step S30 Pixels in the horizontal direction of the image to be trimmed by the electronic zoom 540: the number of pixels in the vertical direction of the image to be trimmed by the electronic zoom with respect to the second captured image in step S30

Here, the reason for resizing in step S40 so that the ratio of the number of pixels is equal to or less than a certain ratio is that when the number of pixels of the first captured image and the second captured image are trimmed by electronic zoom When the ratio to the number of pixels of the image is increased, the difference in resolution between the left and right images of the generated stereo image is increased, resulting in a stereo image which is difficult to view stereoscopically. The resolution is the fineness of the image. In steps S35 and S36, the same processing as the photographing processing 1 described above is performed, a stereo image is generated, and the present processing ends.

FIG. 6 is a diagram showing the processing result in each of steps S30 to S32 and step S40 when the specifications of the imaging units 10-1 and 10-2 are determined as follows.
Imaging unit 10-1: 13 megapixel camera, focal length 27 mm equivalent (35 mm film equivalent), 3 × optical zoom function, captured image size 9.7 megapixel (aspect ratio 16: 9)
Imaging unit 10-2: 3 megapixel camera, 35 mm focal length equivalent (35 mm film equivalent), electronic zoom function, 2 megapixel captured image size (aspect ratio 16: 9)

In step S30, the first captured image is H2336 pixels and W4160 pixels. The second captured image is H1080 pixels and W1920 pixels. In step S31, the first captured image is approximately H1796 pixels and approximately W3200 pixels. Here, due to the difference in the angle of view of the focal lengths of 35 mm and 27 mm, it is possible to shoot at a wide angle of approximately 1.3 times in both the vertical and horizontal directions when shooting with a camera of 27 mm. Therefore, when the camera image of 27 mm is trimmed by the angle of view correction unit 19, the number of pixels becomes about 1 / 1.3. However, even with a camera with the same number of pixels, the image size after trimming varies because of variations in the angle of view. In the first embodiment, the first captured image is trimmed to match the angle of view of the second captured image, but the area is wider than the trimming size according to the zoom magnification of the second captured image. A method may be made to match the angle of view of the first captured image by trimming the image. Specifically, when the optical zoom magnification is doubled, the image of the second captured image of H1080 pixels and W1920 pixels is as follows.
1080/2 * 1.3 = 702
1920/2 * 1.3 = 1248

That is, by trimming an image of H702 pixels and W1248 pixels, it is possible to match the angle of view with the first captured image. In that case, there is no need to trim the first captured image in step S31, and the pixels remain H2336 pixels and W4160 pixels. In the following description, processing in the case of trimming the first captured image will be described.
In step S31, the second captured image remains H1080 pixels and W1920 pixels. In step S32, the first captured image is H (1796-β) pixels and W (3200-α) pixels. The second captured image remains H1080 pixels and W1920 pixels. In step S40, the first captured image is H1530 pixels and W2400 pixels. The second captured image has H1530 pixels and W2400 pixels, and has the same number of pixels as the first captured image.

Here, the meanings of α and β are reduced by correcting the shift of the optical axis when the two cameras are arranged in the stereo image generating device, the shift of the angle of view of each camera itself, and the shift of the optical axis. It shows the number of pixels.

As described above, according to the stereo image generating device 1 according to the first embodiment, the first imaging unit 10-1 having high optical zoom function and pixels lower than the first imaging unit 10-1 are provided. Then, the second imaging unit 10-2 having the electronic zoom function, the zoom magnification threshold storage unit 41 for storing the zoom magnification threshold, the zoom magnification threshold and the optical zoom magnification at the time of shooting are compared, and the optical zoom magnification is The first captured image and the second captured image are converted to the third pixel number when the magnification is smaller than a predetermined magnification, and the first captured image and the second imaging are extracted when the predetermined magnification is exceeded. An image resizing unit 24 for converting an image into a fourth number of pixels, an image encoding unit 27 for encoding the first captured image and the second captured image converted by the image resizing unit 24, and an image encoding unit 27 First captured image And a stereo image format conversion unit 28 for converting the second captured image into a stereo image format to generate a stereo image, a stereo image of high pixels without using two high pixel imaging means Can be taken, and a high-pixel stereo image that is easy to view stereoscopically is generated by suppressing the ratio of the resolution of the first captured image and the resolution of the second captured image within a certain constant even during zoom operation. be able to. In addition, since two high pixel imaging means are not used, the cost can be reduced and the mounting area can be reduced.

Second Embodiment
FIG. 8 is a block diagram showing a schematic configuration of a stereo image generating apparatus according to Embodiment 2 of the present invention. In FIG. 8, a stereo image generating apparatus 1B according to the second embodiment includes an imaging unit 10-1B of high pixel having an optical zoom function and an electronic zoom function, and an imaging unit 10 of low pixel having an electronic zoom function. 2B, zoom control unit 11, autofocus control unit 12, shutter control unit 13, exposure control unit 14, white balance control unit 15, camera operation unit 16, control unit 17, and two camera signals Processing units 18-1 and 18-2, an angle of view correction unit 19, an angle of view information holding unit 20, a position correction unit 21, a position correction information holding unit 22, a color correction unit 23, and an image resizing unit 24 , Super resolution processing unit 25, image filter unit 26, image encoding unit 27, stereo image format conversion unit 28, image storage unit 29, format conversion unit 30, display control unit 31, It includes a radical 113 32, an imaging control unit 40, and a zoom magnification threshold storage unit 41.

The imaging units 10-1B and 10-2B each include an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and output an imaging signal. In the second embodiment, an image obtained by imaging by the imaging unit 10-1B is taken as a first captured image, and an image obtained by imaging by the imaging unit 10-2B is taken as a second captured image. In the second embodiment, the imaging unit 10-1B has a first angle of view, and the imaging unit 10-2B has a second image narrower than the first angle of view of the imaging unit 10-1B. It has a horn. The imaging unit 10-1B also has an optical zoom function and an electronic zoom function. On the other hand, the imaging unit 10-2B does not have an optical zoom function, but only an electronic zoom function.

The zoom control unit 11 performs zoom control of the imaging units 10-1B and 10-2B. The autofocus control unit 12 performs autofocus control of the imaging units 10-1B and 10-2B. The shutter control unit 13 performs shutter control of the imaging units 10-1B and 10-2B. The exposure control unit 14 performs exposure control of the imaging units 10-1B and 10-2B. The white balance control unit 15 performs white balance control of the imaging units 10-1B and 10-2B.

The camera operation unit 16 is for operating the stereo image generating device 1, and a signal at the time of operation such as a zoom magnification designated by the operator is input to the control unit 17. The control unit 17 outputs an operation signal corresponding to the operation in the camera operation unit 16 to the imaging units 10-1B and 10-2B. The camera signal processing unit 18-1 resizes the first captured image captured by the imaging unit 10-1B to the size of the display unit 32 in the preview processing. The camera signal processing unit 18-2 resizes the second captured image captured by the imaging unit 10-2B to the size of the display unit 32 in the preview processing. In the second embodiment, at the time of shooting processing, the camera signal processing unit 18-1 outputs a first captured image (L: left image) of 9.7 megapixels, and the camera signal processing unit 18-2 outputs It is assumed that a second captured image (R: right image) of 2 megapixels is output.

The view angle correction unit 19 reduces the view angle of the first captured image captured by the imaging unit 10-1B based on the view angle correction information stored in the view angle information storage unit 20. For example, the field angle is 5.7 megapixels. The angle of view of the second captured image captured by the imaging unit 10-2B may be corrected. The position correction unit 21 performs position correction on the first captured image captured by the imaging unit 10-1B based on the position correction information stored in the position correction information storage unit 22. In this case, at least one of “vertical shift”, “horizontal shift”, “rotational shift”, or “size shift” is performed as the position correction. The position correction may be performed on the second captured image captured by the imaging unit 10-2B. That is, even if the position correction unit 21 performs position correction on either the first captured image captured by the imaging unit 10-1B or the second captured image captured by the imaging unit 10-2B. Good.

The color correction unit 23 performs the first color difference between the second captured image captured by the imaging unit 10-2B and the first captured image captured by the imaging unit 10-1B after position correction. Color correction is performed on the captured image and the second captured image (the first captured image or the second captured image). The image resizing unit 24 converts the first captured image captured by the imaging unit 10-1B into a predetermined number of pixels, and converts the second captured image captured by the imaging unit 10-2B, The number of pixels is the same as that of the first captured image.

Further, the image resizing unit 24 converts the first captured image captured by the imaging unit 10-1B into a predetermined number of pixels, and the second captured image captured by the imaging unit 10-2B is also a predetermined pixel. Convert to a number. Specifically, the number of pixels of the second captured image is resized in accordance with the number of pixels of the first captured image. Note that the conversion of the first captured image and the conversion of the second captured image performed by the image resizing unit 24 include “equal magnification” in addition to “enlargement” and “reduction”.

In the second embodiment, the case where enlargement processing is performed by the image resizing unit 24 instead of the super-resolution processing unit 25 will be described. In order to make the pixel shift of the left and right images less noticeable due to the difference in lens distortion of the two imaging units 10-1B and 10-2B, the image filter unit 26 makes each image of the first captured image and the second captured image Perform averaging processing to blur the periphery. By providing this image filter unit 26, it is possible to reduce the deterioration of the image quality of the stereo image due to the difference in lens distortion between the two imaging units 10-1B and 10-2B.

Next, the operation of the stereo image generating device 1B according to the second embodiment will be described using FIGS. 9 and 10. The variables used in FIGS. 9 and 10 are described below.
Current zoom factor: Z_now
User-set zoom factor: Z_next
Zoom switching threshold: Z_th
Digital zoom ratio setting for the image captured by the imaging unit 10-1B: Z_e1
Digital zoom ratio setting for the image captured by the imaging unit 10-2B: Z_e2

In FIGS. 9 and 10, the user instructs activation of the imaging units 10-1B and 10-2B through the camera operation unit 16 (step S50). Based on this user instruction, the control unit 17 outputs an instruction to activate the imaging units 10-1B and 10-2B to the imaging control unit 40 (step S51). The control unit 17 sets the optical zoom magnification of the imaging unit 10-1B to 1 × for the imaging control unit 40, and sets the zoom switching threshold Z_th read from the zoom magnification threshold storage unit 41 in the imaging control unit 40 (see Step S52).

After the start instruction is output, steps S 54 to S 69 are repeated until the user instructs the end of shooting using the camera operation unit 16. The determination for repeating the process of steps S54 to S69 is performed in steps S53 and S70. The control unit 17 detects whether the user has performed control of the camera by the camera operation unit 16 (step S54). If not detected, the processes of steps S53, S54 and S70 are repeated. If it is detected, the control unit 17 determines whether it is a photographing instruction (step S55).

If it is a photographing instruction, the photographing process is performed (step S56). Details of the imaging process will be described later. If the determination in step S55 is "No", the control unit 17 determines whether a zoom magnification change instruction has been issued (step S57). When it is the zoom magnification change instruction (that is, when the determination in step S57 is "Yes"), the control unit 17 passes the information Z_next of the zoom magnification set by the user to the imaging control unit 40, and the imaging control unit 40 The current zoom magnification Z_now is compared with the zoom switching threshold Z_th (step S58).

When the current zoom magnification Z_now is equal to or less than the zoom switching threshold Z_th (that is, when the determination in step S58 is “Yes”), the imaging control unit 40 instructs the user-set zoom magnification Z_next instructed from the control unit 17 to be the zoom switching threshold It is determined whether it is Z_th or less (step S59). If the determination in step S59 is "Yes", the imaging control unit 40 specifies Z_next as the optical zoom magnification to the imaging unit 10-1B (step S61). Next, the imaging control unit 40 sets the electronic zoom magnification Z_e1 = 1 in the camera signal processing unit 18-1, and sets the electronic zoom magnification Z_e2 = Z_next in the camera signal processing unit 18-2 (step S62). If the determination in step S59 is "No", the imaging control unit 40 sets Z_th as an optical zoom magnification to the imaging unit 10-1B (step S63).

Next, the imaging control unit 40 sets the electronic zoom magnification Z_e1 = Z_next-Z_th in the camera signal processing unit 18-1, and sets the electronic zoom magnification Z_e2 = Z_next in the camera signal processing unit 18-2 (step S64). If the determination in step S58 is "No", the imaging control unit 40 compares the user-set zoom magnification Z_next with the zoom switching threshold Z_th (step S60). If the determination in step S60 is "Yes", the imaging control unit 40 specifies Z_next as the optical zoom magnification to the imaging unit 10-1B (step S65).

Next, the imaging control unit 40 sets the electronic zoom magnification Z_e1 = 1 in the camera signal processing unit 18-1, and sets the electronic zoom magnification Z_e2 = Z_next in the camera signal processing unit 18-2 (step S66). If the determination in step S60 is "No", the imaging control unit 40 specifies Z_th as the optical zoom magnification to the imaging unit 10-1B (step S67). Next, the imaging control unit 40 sets the electronic zoom magnification Z_e1 = Z_next-Zth in the camera signal processing unit 18-1, and sets the electronic zoom magnification Z_e2 = Z_next in the camera signal processing unit 18-2 (step S68). If the determination in step S57 is "No", control according to it is performed (step S69). For example, the process of changing the size of a captured image is performed.

The above process is repeated until the user instructs the camera operation unit 16 to end shooting. Then, when the user instructs the end of the imaging, the control unit 17 stops the operation of each of the imaging units 10-1B and 10-2B (step S71), and then the present process ends.

Next, the photographing process of step S56 will be described.
FIG. 11 is a flow chart for explaining the photographing process in FIG. 9 and FIG. In FIG. 11, the camera signal processing unit 18-1 resizes the first captured image captured by the imaging unit 10-1B to the image size set by the user, and trims it to the electronic zoom magnification Z_e1. On the other hand, the camera signal processing unit 18-2 resizes the second captured image captured by the imaging unit 10-2B according to the image size set by the user, and performs trimming processing according to the electronic zoom magnification Z_e2 (step S80). ).

Next, the field angle correction unit 19 reads out the field angle correction information from the field angle information holding unit 20, and trims the first captured image captured by the imaging unit 10-1B (step S81). Next, the position correction unit 21 reads the position correction information from the position correction information holding unit 22, and the “vertical shift”, “horizontal shift”, “rotation” with respect to the first captured image captured by the imaging unit 10-1B. At least one of the "deviation" or "magnitude deviation" is corrected (step S82).

Next, the color correction unit 23 corrects the difference in color of the left and right images (step S83). That is, the color difference between the second captured image captured by the imaging unit 10-2B and the first captured image captured by the imaging unit 10-1B after position correction is corrected. Next, the image resizing unit 24 enlarges the second captured image captured by the imaging unit 10-2B according to the size of the first captured image after the angle of view correction (step S84). Next, the image encoding unit 27 encodes the left and right images (for example, compresses them into the JPEG format). That is, the second captured image captured by the imaging unit 10-2B and the first captured image captured by the imaging unit 10-1B after position correction are encoded (step S85). Next, the stereo image format conversion unit 28 converts the left and right images encoded by the image encoding unit 27, that is, the first captured image and the second captured image into a stereo image format (step S86). This process ends when the left and right images are converted to the stereo image format.

Next, the zoom magnification threshold is set to 1.5 times, and each processing result of the above-mentioned step S56 when the zoom magnification set by the user is 1.5 times and 2 times is shown. FIG. 12 is a diagram showing processing results in each step of steps S80 to S82 and step S84 when the specifications of the imaging units 10-1B and 10-2B are determined as follows.
Imaging unit 10-1B: 13 mega pixel camera, focal length 27 mm equivalent (35 mm film equivalent), 3 × optical zoom function, captured image size 9.7 mega pixel (aspect ratio 16: 9)
Imaging unit 10-2B: 3 megapixel camera, 35 mm focal length equivalent (35 mm film equivalent), electronic zoom function, 2 megapixel captured image size (aspect ratio 16: 9)

FIG. 12 illustrates the case where the zoom magnification set by the user at the time of shooting is 1.5.
In step S80, the first captured image is H2336 pixels and W4160 pixels. The second captured image is H1080 pixels and W1920 pixels. Since the zoom magnification is 1.5 times, it is necessary to perform electronic zoom on the imaging unit 10-2B side. Of the captured image on the imaging unit 10-2B side, H 720 pixels and W 1280 pixels in the central part of the image are trimmed and resized to H 1080 pixels and W 1920 pixels.

In step S81, the first captured image is approximately H1796 pixels and approximately W3200 pixels. Here, due to the difference in the angle of view of the focal lengths of 35 mm and 27 mm, it is possible to shoot at a wide angle of about 1.3 times in both the vertical and horizontal directions when shooting with a 27 mm camera. Therefore, when the camera image of 27 mm is trimmed by the angle of view correction unit 19, the number of pixels becomes about 1 / 1.3. However, even with a camera with the same number of pixels, the image size after trimming varies because of variations in the angle of view.

In the second embodiment, the first captured image is trimmed to match the angle of view of the second captured image, but the pixel is higher than the trimming size according to the zoom magnification of the second captured image. A method of adjusting to the angle of view of the first captured image may be used by trimming with numbers. Specifically, in the case of the zoom magnification of 1.5, the following is obtained with respect to the image of H1080 pixel and W1920 pixel which is the second captured image.
1080 / 1.5 * 1.3 = 936
1920 / 1.5 * 1.3 = 1664

That is, it is possible to match the angle of view with the first captured image by trimming the image of H936 pixels and W1664 pixels with respect to the image center of the second captured image. In that case, it is not necessary to trim the first captured image in step S81, and the first captured image remains H2336 pixels and W4160 pixels. In the following description, processing in the case of trimming the first captured image will be described.

In step S81, the second captured image remains at H1080 pixels and W1920 pixels. In step S82, the first captured image is H (1796-β) pixels and W (3200-α) pixels. The second captured image remains H1080 pixels and W1920 pixels. In step S84, the first captured image is H (1796-β) pixels and W (3200-α) pixels. The second captured image is H (1796-β) pixels and W (3200-α) pixels, and has the same number of pixels as the first captured image. Here, the meanings of α and β are reduced by correcting the shift of the optical axis when the two cameras are arranged in the stereo image generating device, the shift of the angle of view of each camera itself, and the shift of the optical axis. It shows the number of pixels.

Here, for reference, FIGS. 7A to 7G described above show the optical axis deviation that occurs when two cameras are arranged in the stereo image generating device 1B.

Next, the processing result of the above-mentioned step S56 in the case where the zoom magnification set by the user at the time of photographing is 2 is shown. FIG. 13 is a diagram showing processing results in each step of steps S80 to S82 and step S84 when the specifications of the imaging units 10-1B and 10-2B are determined as follows.
Imaging unit 10-1B: 13 mega pixel camera, focal length 27 mm equivalent (35 mm film equivalent), 3 × optical zoom function, captured image size 9.7 mega pixel (aspect ratio 16: 9)
Imaging unit 10-2B: 3 megapixel camera, 35 mm focal length equivalent (35 mm film equivalent), electronic zoom function, 2 megapixel captured image size (aspect ratio 16: 9)

In step S80, the first captured image is H2336 pixels and W4160 pixels. The second captured image is H1080 pixels and W1920 pixels. Here, since the zoom magnification threshold is 1.5 times and the zoom magnification specified by the user is 2 times, the zoom operation by the optical zoom is performed up to 1.5 times, and the zoom operation from there is the electronic zoom For the first captured image of H2336 pixel and W4160 pixel image, the following is performed.
2336 / 1.5 = about 1557 pixels 4160 / 1.5 = about 2773 pixels

That is, the electronic zoom operation is performed by trimming an image of H1557 pixels and W2773 pixels with respect to the image center of the first captured image. Here, the trimmed image of H1557 pixels and W2773 pixels may be enlarged to H2336 pixels and W4160 pixels. Here, the case of not enlarging will be described.

On the other hand, in order to perform the electronic zoom operation twice as large as the second captured image, an image of H540 pixels and W960 pixels is trimmed with respect to the image center of the second captured image, and this is H1080 pixels, Resize to W 1920 pixels. In step S81, the first captured image is approximately H1196 pixels and approximately W2130 pixels. Here, due to the difference in the angle of view of the focal lengths of 35 mm and 27 mm, it is possible to shoot at a wide angle of about 1.3 times in both the vertical and horizontal directions when shooting with a 27 mm camera. Therefore, when the camera image of 27 mm is trimmed by the angle of view correction unit 19, the number of pixels becomes about 1 / 1.3. However, even with a camera with the same number of pixels, the image size after trimming varies because of variations in the angle of view.

In the second embodiment, the first captured image is trimmed to match the angle of view of the second captured image, but the area is wider than the trimming size according to the zoom magnification of the second captured image. A method may be made to match the angle of view of the first captured image by trimming the image. Specifically, when the optical zoom magnification is doubled, the image of the second captured image of H1080 pixels and W1920 pixels is as follows.
1080/2 * 1.3 = 702
1920/2 * 1.3 = 1248

That is, by trimming an image of H702 pixels and W1248 pixels, it is possible to match the angle of view with the first captured image. In that case, there is no need to trim the first captured image in step S81, and the pixels remain H2336 pixels and W4160 pixels. In the following description, processing in the case of trimming the first captured image will be described.

In step S81, the second captured image remains at H1080 pixels and W1920 pixels. In step S82, the first captured image is H (1196-β) pixels and W (2130-α) pixels. The second captured image remains H1080 pixels and W1920 pixels. In step S84, the first captured image is H (1196-β) pixels and W (2130-α) pixels. The second captured image is H (1196-β) pixels and W (2130-α) pixels, and has the same number of pixels as the first captured image. Here, the meanings of α and β are reduced by correcting the shift of the optical axis when the two cameras are arranged in the stereo image generating device, the shift of the angle of view of each camera itself, and the shift of the optical axis. It shows the number of pixels.

As described above, according to the stereo image generating device 1B according to the second embodiment, the first imaging unit 10-1B of high pixel having the optical zoom function and the electronic zoom function, and the first imaging unit 10-1B. A second imaging unit 10-2B having an electronic zoom function, a zoom magnification threshold storage unit 41 that stores a zoom magnification threshold, and a predetermined magnification smaller than the upper limit magnification of the optical zoom function. When the magnification is smaller than the first imaging unit 10-1B performs the optical zoom operation, and when the second imaging unit 10-2B performs the electronic zoom operation and exceeds the predetermined magnification, the zoom operation after that is , An imaging control unit 40 that controls the first imaging unit 10-1B and the second imaging unit 10-2B to perform electronic zoom operations, a first captured image and a second captured image with a predetermined number of pixels Image resize to convert to 24, an image encoding unit 27 encoding the first captured image and the second captured image converted by the image resizing unit 24, and a first captured image and a second captured image encoded by the image encoding unit 27 And the stereo image format conversion unit 28 that generates a stereo image by converting the image data into the stereo image format, so that high-pixel stereo images can be captured without using two high-pixel imaging means. When the zoom magnification is increased so that the ratio of the resolution of the first captured image and the resolution of the second captured image is kept within a certain range even during zoom operation, the imaging unit 10-1B changes from optical zoom to electronic zoom By switching, it is possible to generate a high-pixel stereo image that is easy to view stereoscopically. The resolution is the fineness of the image. In addition, since two high pixel imaging means are not used, the cost can be reduced and the mounting area can be reduced. In addition, the upper limit magnification of the optical zoom function here is the maximum magnification on the specifications of the optical module to be used. Alternatively, in the movable portion of the optical module, the magnification is in a state where the zoom magnification is moved to the limit in the direction in which the zoom magnification is increased.

Although the invention has been described in detail and with reference to specific embodiments, it will be apparent to those skilled in the art that various changes and modifications can be made without departing from the spirit and scope of the invention.

This application is based on the Japanese patent application filed on April 14, 2011 (Japanese Patent Application No. 2011-090395) and the Japanese patent application filed on April 14, 2011 (Japanese Patent Application No. 2011-090396), the contents of which are It is incorporated here as a reference.

The present invention can capture a high-pixel stereo image without using two high-pixel imaging means, and also has a difference in resolution between the first and second captured images when zooming. It has an effect that it is possible to generate a stereo image easy to view stereoscopically, within a fixed value, and it is possible to apply to a compound eye camera or the like for capturing a stereo image that captures a stereo image.

1, 1 B Stereo image generator 10-1, 10-2, 10-1B, 10-2B Imaging unit 11 Zoom control unit 12 Autofocus control unit 13 Shutter control unit 14 Exposure control unit 15 White balance control unit 16 Camera operation unit 17 control unit 18-1 and 18-2 camera signal processing unit 19 angle of view correction unit 20 angle of view information holding unit 21 position correction unit 22 position correction information holding unit 23 color correction unit 24 image resizing unit 25 super resolution processing unit 26 Image filter unit 27 image encoding unit 28 stereo image format conversion unit 29 image storage unit 30 format conversion unit 31 display control unit 32 display unit 40 imaging control unit 41 zoom magnification threshold storage unit

Claims

A first imaging unit configured to capture a first captured image, having a first number of pixels and having an optical zoom function;
A second imaging unit configured to capture a second captured image, having a second number of pixels smaller than the first number of pixels and having an electronic zoom function;
And a resolution adjustment unit configured to adjust a difference in resolution between right and left based on a zoom magnification specified in stereo image shooting.
The stereo image generating apparatus according to claim 1, wherein
The first imaging unit performs an optical zoom operation, and the second imaging unit performs an electronic zoom operation based on a zoom magnification specified in stereo image shooting.
The first captured image and the second captured image are converted into a third number of pixels when the zoom magnification is less than or equal to a predetermined magnification, and the first imaging is performed when the predetermined magnification is exceeded. An image resizing unit configured to convert an image and the second captured image into a fourth number of pixels smaller than the third number of pixels;
An image encoding unit that encodes the first captured image and the second captured image converted by the image resizing unit;
A stereo image format conversion unit that converts the first captured image and the second captured image encoded by the image encoding unit into a stereo image format to generate a stereo image;
Stereo image generator equipped with
The stereo image generating apparatus according to claim 2,
A stereo image generating device, wherein conversion of the first captured image and conversion of the second captured image, which are performed by the image resizing unit, are enlargement, reduction, or equal-scale conversion.
The stereo image generating apparatus according to claim 2 or 3, wherein
A zoom magnification threshold storage unit that holds the predetermined magnification of the optical zoom magnification as a zoom magnification threshold;
The image resizing unit performs resizing of the first captured image and the second captured image based on the zoom magnification threshold.
The stereo image generating apparatus according to any one of claims 2 to 4, wherein
The stereo image generation device whose said 3rd number of pixels is said 1st number of pixels.
The stereo image generating apparatus according to claim 2,
The first imaging unit has a first angle of view,
The second imaging unit includes a second angle of view narrower than the first angle of view.
And an angle-of-view correction unit configured to trim the first or second captured image in accordance with the optical zoom magnification of the first imaging unit.
The image encoding unit is trimmed by the angle of view correction unit, and the first captured image in which the number of pixels is converted in the image resizing unit and the second captured image in which the number of pixels is converted in the image resizing unit And stereo image generator to encode.
The stereo image generating apparatus according to claim 6, wherein
An angle-of-view information holding unit that holds angle-of-view correction information for each optical zoom magnification of the first imaging unit;
The stereo image generating device, wherein the angle of view correction unit performs trimming based on the angle of view correction information.
The stereo image generating apparatus according to claim 2,
A position correction unit configured to correct at least one of vertical deviation, horizontal deviation, rotational deviation, or size deviation with respect to any of the first captured image or the second captured image;
The image encoding unit is corrected by the position correction unit, and the first captured image converted into a predetermined number of pixels by the image resizing unit is corrected by the position correction unit, and the image resizing unit corrects the predetermined image. A stereo image generation device that encodes the second captured image converted into the number of pixels of
A stereo image generating apparatus according to claim 8, wherein
A position correction information holding unit that holds position correction information for each optical zoom magnification of the first imaging unit;
The position correction unit corrects at least one of vertical deviation, horizontal deviation, rotational deviation, or size deviation based on the position correction information.
The stereo image generating apparatus according to claim 2,
A color correction unit that performs color correction on the first captured image and / or the second captured image is provided for a difference in color between the first captured image and the second captured image. ,
The image encoding unit is a stereo image generating device that encodes the first captured image whose color is corrected by the color correction unit and the second captured image.
The stereo image generating apparatus according to claim 3,
A stereo image generation device using super-resolution processing when the conversion of the first captured image and the conversion of the second captured image performed by the image resizing unit are enlargement conversions.
The stereo image generating apparatus according to claim 2,
And an image filter unit that performs averaging processing on peripheral portions of the first captured image and the second captured image.
The image encoding unit performs conversion of the number of pixels in the image resizing unit, and then performs conversion of the number of pixels in the image capturing unit and the first captured image subjected to averaging processing by the image filter unit. A stereo image generating device that encodes the second captured image that has been subjected to averaging processing by the image filter unit after being performed;
A stereo image generating apparatus according to any one of claims 2 to 12, wherein
The stereo image generating device according to the first aspect of the present invention, wherein the first pixel count, the third pixel count, and the fourth pixel count are in the following relationship: first pixel count ≧ third pixel count> fourth pixel count.
A first imaging step of imaging a first captured image with a first number of pixels in a first imaging unit having an optical zoom function;
A second imaging step of imaging a second captured image with a second number of pixels smaller than the first number of pixels in a second imaging unit having an electronic zoom function;
When the specified zoom magnification is equal to or less than a predetermined magnification, the first captured image and the second captured image are converted into a third number of pixels, and when the predetermined magnification is exceeded, An image resizing step of converting the first captured image and the second captured image into a fourth pixel number smaller than the third pixel number;
An image encoding step of encoding the first captured image and the second captured image converted in the image resizing step;
A stereo image format conversion step of converting the first captured image and the second captured image encoded in the image encoding step into a stereo image format to generate a stereo image;
A stereo image generating method comprising:
The stereo image generating apparatus according to claim 1, wherein
The first imaging unit has both an optical zoom function and an electronic zoom function.
In taking a stereo image, the first imaging unit performs an optical zoom operation and an electronic zoom operation based on a designated zoom magnification, and the second imaging unit performs an electronic zoom operation.
When the zoom magnification is equal to or less than a predetermined magnification smaller than the upper limit magnification of the optical zoom function, the first imaging unit performs an optical zoom operation, and the second imaging unit performs an electronic zoom operation, and the predetermined magnification And an imaging control unit that controls so that the first imaging unit and the second imaging unit perform electronic zoom operations in the subsequent zoom operations.
An image resizing unit configured to convert the first captured image and the second captured image into a predetermined number of pixels;
An image encoding unit that encodes the first captured image and the second captured image converted by the image resizing unit;
A stereo image format conversion unit that converts the first captured image and the second captured image encoded by the image encoding unit into a stereo image format to generate a stereo image;
Stereo image generator equipped with
The stereo image generating apparatus according to claim 15.
A stereo image generating device, wherein conversion of the first captured image and conversion of the second captured image, which are performed by the image resizing unit, are enlargement, reduction, or equal-scale conversion.
The stereo image generating apparatus according to claim 15 or 16,
A zoom magnification threshold storage unit configured to store the switching magnification of the zoom magnification as a zoom magnification threshold;
The imaging control unit performs switching control of the zoom operation of the first imaging unit and the second imaging unit based on the zoom magnification threshold.
A stereo image generating apparatus according to any one of claims 15 to 17, wherein
The stereo image generating device, wherein the predetermined number of pixels is the first number of pixels.
The stereo image generating apparatus according to claim 15.
The first imaging unit has a first angle of view,
The second imaging unit includes a second angle of view narrower than the first angle of view.
An angle of view correction unit that trims the first captured image;
The image encoding unit is trimmed by the angle of view correction unit, and the first captured image of which the number of pixels is converted by the image resizing unit, and the second imaging whose number of pixels is converted by the image resizing unit Stereo image generator that encodes images.
20. The stereo image generating device according to claim 19, wherein
A stereo image generating apparatus, comprising: an angle-of-view information holding unit that holds angle-of-view correction information for each optical zoom magnification of the first imaging unit, wherein the angle-of-view correction unit trims based on the angle-of-view correction information.
The stereo image generating apparatus according to claim 15.
A position correction unit configured to correct at least one of vertical deviation, horizontal deviation, rotational deviation, or size deviation with respect to any of the first captured image or the second captured image;
The image encoding unit is corrected by the position correction unit, and the first captured image converted into a predetermined number of pixels by the image resizing unit is corrected by the position correction unit, and the image resizing unit is corrected by the image resizing unit. A stereo image generating device that encodes the second captured image converted into the number of pixels.
22. The stereo image generating device according to claim 21, wherein
A position correction information holding unit that holds position correction information for each optical zoom magnification of the first imaging unit;
The position correction unit corrects at least one of vertical deviation, horizontal deviation, rotational deviation, or size deviation based on the position correction information.
The stereo image generating apparatus according to claim 15.
A color correction unit that corrects the color of the first captured image and / or the second captured image with respect to a difference in color between the first captured image and the second captured image;
The image encoding unit is a stereo image generating device that encodes the first captured image whose color is corrected by the color correction unit and the second captured image.
The stereo image generating apparatus according to claim 16, wherein
A stereo image generation device using super-resolution processing when the conversion of the first captured image and the conversion of the second captured image performed by the image resizing unit are enlargement conversions.
The stereo image generating apparatus according to claim 15.
And an image filter unit that performs averaging processing on peripheral portions of the first captured image and the second captured image.
The image encoding unit converts the number of pixels in the image resizing unit, and then converts the number of pixels in the first resized image subjected to averaging processing by the image filter unit and the image resizing unit, and A stereo image generating device that encodes the second captured image that has been subjected to averaging processing by an image filter unit.
A first imaging step of imaging a first captured image with a first number of pixels in a first imaging unit having an optical zoom function and an electronic zoom function;
A second imaging step of imaging a second captured image with a second number of pixels smaller than the first number of pixels in a second imaging unit having an electronic zoom function;
When the specified zoom magnification is equal to or less than a predetermined magnification smaller than the upper limit magnification of the optical zoom function, the first imaging unit performs an optical zoom operation, and the second imaging unit performs an electronic zoom operation. And a zoom control step of performing control so that the first imaging unit and the second imaging unit perform electronic zoom operations in the subsequent zoom operation,
An image resizing step of converting the first captured image and the second captured image into a predetermined number of pixels;
An image encoding step of encoding the first captured image and the second captured image converted in the image resizing step;
A stereo image format conversion step of converting the first captured image and the second captured image encoded in the image encoding step into a stereo image format to generate a stereo image;
A stereo image generating method comprising: