JP6230266B2 - Imaging apparatus, image processing apparatus, and control method - Google Patents

Description

The present invention relates to an imaging apparatus , an image processing apparatus, and a control method that can perform stereoscopic video processing.

  In recent years, three-dimensional (3D) cinema, 3D display, and other 3D video related devices are rapidly spreading. Although stereoscopic video shooting has been conventionally performed with a film camera or the like, with the widespread use of digital imaging devices, original images for generating stereoscopic video with a digital camera, a digital video camera, and the like have been captured. ing.

  As a mechanism for a user to view a stereoscopic video, left-eye image data and right-eye image data that have a parallax in the left-right direction so as to correspond to an image viewed from the left eye and an image viewed from the right eye are provided. Prepared. A user can view each image stereoscopically by viewing the image with the left eye and the right eye. As such a method, there is a method of dividing an image to be viewed, such as a parallax barrier method or a lenticular method, into a parallax. There is also known a method for presenting different images to the left and right eyes of the user through filters having different characteristics on the left and right.

  On the other hand, Patent Documents 1 and 2 disclose a method for simultaneously capturing images from different viewpoints as a method for capturing an image that can be viewed as a stereoscopic video. Patent Document 1 discloses a solid-state imaging device in which a plurality of microlenses are formed, and one or more pairs of photodiodes are arranged in proximity to each of the microlenses. A first image signal is obtained from the output of one of the photodiode pairs, and a second image signal is obtained from the output of the other photodiode. By using the left-eye image and the right-eye image based on the first and second image signals, the user can view a stereoscopic video.

  Patent Document 2 discloses an output parallax map having an output element having an output value corresponding to a shift to be applied to each pixel of the first image. A second image can be generated based on the output parallax map and the first image.

JP-A-58-24105 Special table 2008-518317

  In order to view a stereoscopic image, it is necessary to prepare a left-eye image for a user to view with a left eye and a right-eye image for a user to view with a right eye. On the other hand, for digital cameras that can select the aspect ratio of the shot image, that is, the aspect ratio, trimming can be performed on the recorded image after image processing, or by performing a trimming process before image processing after imaging according to the aspect ratio. There is a method of processing.

  When a camera capable of outputting an image capable of stereoscopic display has a function of setting a plurality of aspect ratios, the stereoscopic effect of the display image may not be obtained depending on the set aspect ratio. Hereinafter, the reason will be described with reference to FIG.

  FIG. 9 is a diagram schematically showing a photographed image, and illustrates three subjects with different distances from the camera. The arrow Z in FIG. 9C represents the depth direction, and the subject 1105 represents the subject farthest from the camera. A subject 1107 represents a subject closest to the camera, and a subject 1106 represents a subject located at a distance between the subjects 1105 and 1107. The subject 1107 is located on the rightmost side, and the subject 1105 is located on the leftmost side.

  FIG. 9A illustrates a composition in which the subject is photographed within the image frame 1103a with the aspect ratio set to 3: 2, for example. Within the image frame 1103a, the same reference numerals are used for the subject images as the subjects 1105, 1106, and 1107, and they are arranged in the order of the symbols from the top. FIG. 9B illustrates a composition in which the subject is photographed within the image frame 1103b with the aspect ratio set to 1: 1, for example. In other words, since only the image inside the vertical line 1101 and the line 1102 is captured with respect to the image frame 1103a, the image frame 1103b has a narrow lateral imaging range compared to the image frame 1103a. Therefore, the image of the subject 1107 in the image frame 1103a is deleted from the image frame 1103b. When processing is performed on the composition of the image frame 1103b by the same stereoscopic image processing as in the case of the image frame 1103a, the subject 1107 deviates from the angle of view. For this reason, the depth amounts for the subjects 1105 and 1106 are calculated, and there is a possibility that sufficient stereoscopic effect may not be obtained as compared with the case of the image frame 1103a.

An object of the present invention is to provide an image processing line USo location and a control method which does not impair the stereoscopic effect by performing parallax correction in accordance with the setting of the aspect ratio.

In order to solve the above-mentioned problem, an apparatus according to an embodiment of the present invention combines an acquisition unit that acquires image signals of different viewpoints from output signals of a plurality of photoelectric conversion units, and an image signal of the different viewpoints. An image processing unit that generates composite image data, a parallax amount calculating unit that calculates a parallax amount of the subject from the image signals of different viewpoints, and a first aspect ratio related to the captured image, are used for recording or reproducing an image. When there is an image range that is excluded from the image frame due to the change to the two aspect ratio, a parallax correction unit that corrects the parallax amount of the subject in the images of different viewpoints is provided. When the parallax correction unit corrects the parallax amount by a correction parameter determined from the first aspect ratio and the second aspect ratio, the parallax correction unit converts the parallax correction unit to the second aspect ratio among the images having the first aspect ratio. The larger the image range excluded from the image frame due to the change, the larger the parallax amount.

  According to the present invention, by performing parallax correction in accordance with the setting of the aspect ratio, it is possible to perform image processing that does not impair the stereoscopic effect.

FIG. 9 is a diagram schematically illustrating an entire configuration of an image sensor in order to describe an embodiment of the present invention in conjunction with FIGS. 2 to 8. It is a figure which shows schematically the structural example (A) of 1 pixel of an image pick-up element, and arrangement | positioning of a pixel array (B). It is a figure which shows typically the imaging relationship of an object. It is a figure which shows the example of schematic structure of an imaging device. It is the figure which illustrated the stereo image typically. It is the figure which illustrated typically the mode at the time of appreciation with a stereo display apparatus. It is the figure which illustrated conceptually about the parallax correction process. It is a flowchart which shows the process example of an imaging device. It is the figure which illustrated notionally the composition according to the setting of an aspect ratio.

  Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically illustrating a configuration example of an imaging element used in the imaging apparatus of the present embodiment. The image sensor 100 includes a pixel array 101, a vertical selection circuit 102 that selects a row in the pixel array 101, and a horizontal selection circuit 104 that selects a column in the pixel array 101. The readout circuit 103 reads out the signal of the pixel selected by the vertical selection circuit 102 among the pixels in the pixel array 101. The reading circuit 103 includes a memory for accumulating signals, a gain amplifier, an AD converter, and the like for each column.

  The serial interface (SI) unit 105 determines an operation mode of each circuit according to an instruction from an external circuit. The vertical selection circuit 102 sequentially selects a plurality of rows in the pixel array 101 and extracts pixel signals to the readout circuit 103. The horizontal selection circuit 104 sequentially selects a plurality of pixel signals read by the reading circuit 103 for each column. In addition to the components shown in FIG. 1, the imaging device 100 includes a timing generator that provides timing signals to the vertical selection circuit 102, the horizontal selection circuit 104, the readout circuit 103, and the like, a control circuit, and the like. Detailed description thereof will be omitted.

  FIG. 2 is a diagram illustrating a configuration example of pixels of the image sensor 100. FIG. 2A schematically shows the configuration of one pixel. FIG. 2B shows the arrangement of the pixel array 101. A pixel 201 illustrated in FIG. 2A includes a microlens 202 as an optical element and a plurality of photodiodes (hereinafter abbreviated as PD) as light receiving elements.

  FIG. 2A illustrates an example in which two pixels, the left PD 203 and the right PD 204, are provided in one pixel, but three or more (for example, four or nine) PDs may be used. The PD 203 photoelectrically converts the received light beam and outputs a left-eye image signal. The PD 204 photoelectrically converts the received light beam and outputs a right eye image signal. The pixel 201 includes, for example, a pixel amplification amplifier that extracts a PD signal to the readout circuit 103, a row selection switch, a PD signal reset switch, and the like in addition to the illustrated components.

  In order to provide a two-dimensional image, the pixel array 101 is arranged in a two-dimensional array like a large number of pixels 301 to 304 shown in FIG. In the pixels 301 to 304, PDs 301L, 302L, 303L, and 304L correspond to the PD 203 in FIG. PDs 301R, 302R, 303R, and 304R correspond to the PD 204 in FIG. That is, each pixel has a first photoelectric conversion unit (PD 203) that outputs image data for the left eye and a second photoelectric conversion unit (PD 204) that outputs image data for the right eye.

  Next, light reception of the imaging element 100 having the pixel configuration illustrated in FIG. FIG. 3 is a conceptual diagram showing a state where a light beam emitted from the exit pupil of the photographing lens is incident on the image sensor 100.

  The pixel array 101 includes a microlens 202, a color filter 403, and PDs 404 and 405. PD 404 and PD 405 correspond to PD 203 and PD 204 in FIG.

  In FIG. 3, the center of the light beam emitted from the exit pupil 406 of the photographing lens is defined as the optical axis 409 for each microlens 202. The light emitted from the exit pupil 406 enters the image sensor 100 with the optical axis 409 as the center. Partial areas 407 and 408 are areas of the exit pupil 406 of the photographing lens. Light rays 410 and 411 are the outermost light rays of the light passing through the partial region 407. Light rays 412 and 413 are the outermost light rays of the light passing through the partial region 408.

  Among the light beams emitted from the exit pupil 406, with the optical axis 409 as a boundary line, the upper light beam in FIG. 3 is incident on the PD 405, and the lower light beam is incident on the PD 404. That is, PD 404 and PD 405 receive light beams from different regions of the exit pupil of the photographing optical system.

  FIG. 4 is a diagram illustrating a configuration example of the imaging apparatus according to the present embodiment. Hereinafter, an application example to a digital camera will be described with reference to FIG. The lens unit 501 constituting the imaging optical system forms an image of light from the subject on the imaging element 505. The imaging element 505 corresponds to the imaging element 100 illustrated in FIG. 1 and has a pixel configuration illustrated in FIG.

  The lens driving device 502 performs zoom control, focus control, aperture control, and the like. The mechanical shutter 503 is controlled by a shutter driving device 504. The image sensor 505 converts the subject image formed by the lens unit 501 into an image signal. The imaging signal processing circuit 506 performs various processes, corrections, data compression, and the like on the image signal output from the imaging element 505. The timing generator 507 outputs a timing signal necessary for the image sensor 505 and the image signal processing circuit 506.

  A system control unit 509 is a control unit that performs various calculations and controls the entire imaging apparatus, and performs processing by a CPU (central processing unit) (not shown) executing a program. Based on the composite image generated by the image composition circuit 513 and the parallax map generated by the parallax map generation circuit 514, the system control unit 509 uses the left-eye image data and the right-eye image data used for stereoscopic display. Is generated. In addition, the system control unit 509 performs playback control of the left-eye image and the right-eye image, so that the user can view a stereoscopic image on the display screen. Note that the system control unit 509 can also realize phase difference AF (autofocus) by detecting a phase difference based on the left-eye image data and the right-eye image data.

  The storage unit 508 includes a memory that temporarily stores image data. A recording medium control interface unit (hereinafter abbreviated as I / F unit) 510 is provided for recording or reading image data or the like on the recording medium 511. The recording medium 511 that can be attached to and detached from the imaging apparatus is a semiconductor memory or the like. The external I / F unit 512 transmits / receives data to / from an external device. The display unit 518 displays various information and captured images in accordance with the display data from the display control circuit 517.

  The imaging signal processing circuit 506 performs image processing by distributing imaging data output from the imaging element 505 into left-eye image data and right-eye image data. The storage unit 508 stores the output data of the imaging signal processing circuit 506, the synthesized image data generated by the image synthesis circuit 513, and the parallax map data generated by the parallax map generation circuit 514.

  The image composition circuit 513 synthesizes the left-eye image and the right-eye image to generate composite image data. The parallax map generation circuit 514 performs a parallax amount calculation process. The parallax map generation circuit 514 calculates, as a parallax amount, a positional deviation amount of the subject image in the left-eye image / right-eye image based on the position of the subject image in the composite image. Information about the calculated amount of parallax is stored in the storage unit 508 as parallax map data. The parallax correction circuit 515 corrects the parallax amount using the parallax amount calculated by the parallax map generation circuit 514 and the correction parameter. Details of the parallax amount correction processing will be described later. The photometric device 516 acquires a photometric value used for exposure control and outputs it to the system control unit 509.

Next, the operation of the digital camera at the time of shooting in this embodiment will be described with reference to FIG.
When the main power supply is turned on, the power supply of the control system circuit unit is turned on. Further, the power of the imaging processing system circuit such as the imaging signal processing circuit 506 is turned on, and the camera system is in a standby state (S1301). The system control unit 509 determines whether or not the user has operated a release button (not shown) (S1302). If there is an operation instruction to start the shooting operation, the process proceeds to S1303. If there is no operation instruction, the determination process of S1302 is repeated.

  The system control unit 509 performs arithmetic processing related to focus state detection and focus adjustment operations based on output data from the image sensor 505 (S1303). In this calculation process, a focus position or the like corresponding to the distance from the camera to the subject is calculated, and the driving amount of the focus lens is calculated. The system control unit 509 performs drive control of the lens unit 501 via the lens driving device 502 according to the calculation result. Next, it is determined whether or not the calculation process of S1303 has been completed (S1304). If the process has been completed, the shooting operation is started, and the process proceeds to S1305. If the calculation process of S1303 has not ended, the determination process of S1304 is repeated to enter a waiting process.

  When the photographing operation is finished in step S1305, the imaging signal processing circuit 506 performs image processing on the image signal output from the imaging element 505. The system control unit 509 performs control to acquire data of a stereo image (left-eye image and right-eye image) and write the data to the storage unit 508. In step S1306, the system control unit 509 determines whether the acquisition of stereo image data has ended. If the data acquisition has ended, the system control unit 509 proceeds to step S1307. If the data acquisition has not ended, the determination processing in step S1306 is repeated. It is. In step S1307, the system control unit 509 calculates the distance from the camera to the subject (subject distance) based on the calculation result in step S1303. In next step S1308, the system control unit 509 obtains an angle of view at the time of photographing using information such as the subject distance calculated in step S1307 and the focal length of the photographing lens, and calculates a correction parameter corresponding to the angle of view. A method for calculating the correction parameter will be described later.

  Next, the parallax map generation circuit 514 performs a parallax amount detection process and a parallax map generation process (S1309). Based on the amount of parallax detected by the parallax map generation circuit 514 and the correction parameter, the parallax correction circuit 515 corrects the amount of parallax for each of the left-eye image and the right-eye image (S1310). Thereby, the parallax map data is corrected (S1311). In next step S1312, the system control unit 509 determines whether or not to end the shooting. If there is an instruction to end the shooting, the process proceeds to S1313. If the shooting operation is continued, the determination process in S1312 is repeated. In step S <b> 1313, the system control unit 509 stops the power supply to the imaging device and executes a shutdown process that stores necessary information in the storage unit 508.

Next, image processing in this embodiment will be described.
The image sensor 505 outputs all image signals from the PD. In the example shown in FIG. 2B, image signals are sequentially output in the order of PDs, that is, 301L, 301R, 302L, 302R, 303L, 303R, 304L, and 304R. The imaging signal processing circuit 506 acquires image data from the output signal of the imaging element 505 and performs data separation processing into left-eye image data and right-eye image data. The left-eye image data is image data obtained by selectively processing only the outputs of PDs 301L, 302L, 303L, and 304L (left PD output) in FIG. The right-eye image data is image data obtained by selectively processing only the outputs (right PD output) of PDs 301R, 302R, 303R, and 304R in FIG. The left-eye image data and the right-eye image data are stored in the storage unit 508 separately.

  The left-eye image data and right-eye image data stored in the storage unit 508 are sent to the image composition circuit 513 to generate composite image data. The generated composite image data is stored in the storage unit 508. The image processing executed by the image composition circuit 513 is a process for obtaining an addition average value for each pixel for the left-eye image data and the right-eye image data. Even when the image data read from the image sensor 505 is photographed with different shapes of the subject for the left-eye image and the right-eye image, the shape of the subject is interpolated by the image composition processing, and image data with the correct shape is obtained. Generated.

  Next, the parallax map generation circuit 514 generates a parallax map using the left-eye image data and the right-eye image data. The obtained parallax map data is stored in the storage unit 508. Information stored in the parallax map will be described with reference to FIG.

  FIG. 5 is a diagram schematically showing a captured image. FIG. 5A shows images obtained by photographing three subjects 602, 603, and 604 with different distances from the camera in the image frame 601. Small figures similar to each subject exemplify feature points corresponding to face positions and the like. An arrow Z in FIG. 5C represents the depth direction, and a subject 602 represents a subject farthest from the camera. A subject 604 represents a subject closest to the camera, and a subject 603 represents a subject located at a distance between the subjects 602 and 604.

  FIG. 5B shows stereo images obtained when the composition shown by the image frame 601 in FIG. 5A is captured, in the image frames 605 and 606, respectively. The image in the image frame 605 is a left-eye image, and the image in the image frame 606 is a right-eye image. The left-eye image in the image frame 605 includes images 607L, 608L, and 609L corresponding to the subjects 602, 603, and 604, respectively. The right-eye image in the image frame 606 includes images 607R, 608R, and 609R corresponding to the subjects 602, 603, and 604, respectively. A one-dot chain line in the figure represents the positional relationship between the subject images in the image frames 605 and 606, and the shift amount of the subject image between the left-eye image and the right-eye image is defined as a parallax amount. The parallax amount 610 represents the positional deviation amount between the left-eye image and the right-eye image regarding the subject 602, that is, the parallax between the images 607L and 607R. Similarly, the parallax amount 611 represents the positional deviation amount of the images 609L and 609R with respect to the subject 604. The subject 603 shows a state in which the amount of parallax is zero between the left-eye image and the right-eye image.

When obtaining the parallax amount, the parallax map generation circuit 514 detects the position of the subject image included in the left-eye image in the image frame 605 and the right-eye image in the image frame 606, and obtains the shift amount between the subject images. The amount of parallax is calculated. Specifically, when left-eye image data and right-eye image data are input to the parallax map generation circuit 514, parallax amounts 610 and 611 are calculated. Then, the distance to the center of gravity of the subject image in the left-eye image or right-eye image is calculated based on the center point of the subject image in the composite image, that is, the midpoint between the center of gravity in the left-eye image and the center of gravity in the right-eye image. Is required. Data indicating a half amount 612 of the parallax amount 610 and data indicating a half amount 613 of the parallax amount 611 are stored in the storage unit 508 as parallax map information.
A parallax amount 610 illustrated in FIG. 5B corresponds to an apparent depth amount when a stereoscopic image of a subject farthest from the camera is displayed. Further, the parallax amount 611 corresponds to an apparent pop-out amount when a stereoscopic image of the subject closest to the camera is displayed. That is, the parallax amount between the left-eye image and the right-eye image represents the parallax when the subject is viewed with the right eye and the left eye, and determines the apparent depth amount and pop-out amount.

With reference to FIG. 6, a general visual effect that gives a viewer a stereoscopic effect when a left-eye image and a right-eye image are displayed on a stereo display device will be described.
FIG. 6 schematically shows a state of viewing on a stereo display device, and is a view of the stereo display device and an observer viewed from above. The observer 902 observes the screen surface 901 of the stereo display device, and indicates the positions of the left eye 903L and the right eye 903R, respectively. A dotted line schematically shows how each subject image projected on the screen surface 901 is projected onto the right eye 903L and the right eye 903R.

  A left-eye image and a right-eye image are displayed on the screen surface 901, and are projected only on the left eye 903L and the right eye 903R of the observer, respectively. FIG. 6 illustrates feature points 904L and R, 905L and R, and 906L and R of different subject images, which correspond to the observer's gaze point, respectively. That is, the video of the subject feature points 904L, 905L, and 906L in the left-eye image is projected onto the left eye 903L, and the video of the subject feature points 904R, 905R, and 906R in the right-eye image is projected onto the right eye 903R. For the observer, the position 904 of the subject image corresponding to the feature point 904L in the left-eye image and the feature point 904R in the right-eye image appears behind the screen surface 901. In addition, the observer can see the position 905 of the subject image corresponding to the feature point 905L in the left-eye image and the feature point 905R in the right-eye image on the screen surface 901. Further, the observer sees the position 906 of the subject image corresponding to the feature point 906L in the image for the left eye and the feature point 906R in the image for the right eye, which pops out before the screen surface 901. The depth amount and the pop-out amount of these subject images depend on the parallax amount of the subject image in the left-eye image and the right-eye image.

Next, parallax correction processing according to aspect ratio setting information will be described. The aspect ratio setting at the time of image recording or reproduction is automatically set by the system control unit 509 according to the angle of view, composition, or the like, or is set according to a user operation instruction.
For example, the parallax correction circuit 515 performs a parallax correction calculation process using the correction parameters shown in Table 1 below. When it is determined that there is an image range that is excluded from the image frame by trimming when the value of the correction parameter is changed from the first aspect ratio of the captured image to the second aspect ratio at the time of image recording or playback In addition, it is determined to correct the parallax amount.

  The reference table exemplified in Table 1 includes the changed second aspect ratio and parallax information as correction parameters. For example, a plurality of types of the second aspect ratio are described in the column 1001, and the parallax amount corresponding to each aspect ratio is described in the column 1002. The parallax amount is set to a larger value as the second aspect ratio has a larger range in which data is deleted without entering the image frame after the change in the photographed image, that is, the image with the first aspect ratio. The parallax correction circuit 515 corrects the parallax amount based on the correction parameter determined from the first aspect ratio and the second aspect ratio. For example, a row 1003 in Table 1 describes parameter values when the second aspect ratio is 1: 1. Hereinafter, the parallax correction processing based on the value of the correction parameter 1006 will be described with reference to FIG.

  FIG. 7A illustrates a composition obtained by photographing a plurality of subjects shown in the image frame 704. In the image frame 704, subject images 707 and 708 are shown. Small figures similar to each subject illustrate feature points. FIG. 7B shows a left-eye image in an image frame 705, and subject images 707L and 708L correspond to the subject images 707 and 708 in FIG. 7A, respectively. FIG. 7C shows the right-eye image in the image frame 706, and subject images 707R and 708R correspond to the subject images 707 and 708 in FIG. 7A, respectively. However, the parallax amount of the subject image 708 is zero. FIGS. 7D and 7E show image frames 1201 and 1202 when the aspect ratio is set to 1: 1, respectively. In the image frame 1201 in FIG. 7D, the parallax-corrected image for the left eye is shown, and in the image frame 1202 in FIG. 7E, the parallax-corrected image for the right eye is shown.

  To the parallax correction circuit 515, data of the left-eye image in the image frame 705 illustrated in FIG. 7B and data of the right-eye image in the image frame 706 illustrated in FIG. 7C are input. Here, the parallax correction circuit 515 performs a subject image shift process within each image frame in order to correct the parallax amount according to the correction parameter. A case where the parallax for the subject image 707 is corrected will be specifically described. A parallax amount 709 exists between the subject image in the image frame 705 and the subject image in the image frame 706. This amount of parallax 709 corresponds to the position difference between the subject images 707L and 707R within the image frame, and parallax correction is performed based on the correction parameter. The parallax correction circuit 515 refers to the value of the correction parameter 1006 corresponding to the aspect ratio 1: 1 shown in the row 1003 of Table 1 and shifts the subject image as shown in FIGS. 7D and 7E. I do. That is, in FIG. 7D, the subject image 707L is changed to the subject image 1203L by the shift operation for moving the subject image 707L to the left in FIG. 7, and in FIG. 7E, the shift operation for moving the subject image 707R to the right in FIG. Thus, the subject image 1203R is changed. In these drawings, an object image shift operation for further emphasizing the depth amount is shown. Here, the parallax amount 1205 indicates the parallax amount after parallax correction, and is determined based on the value of the correction parameter 1006 in Table 1. For example, as the value of the correction parameter 1006, data indicating the parallax amount 1205 may be used directly, or data indicating the shift ratio may be used. In the case of the shift ratio, a numerical value obtained by dividing the value of the parallax amount 1205 by the value of the parallax amount 709 becomes the value of the correction parameter 1006, and is used as a coefficient for multiplying the parallax amount at the time of parallax correction. Further, the correction parameter 1006 may be a fixed ratio, or may be expressed by a function expression of several orders according to the apparent pop-out amount and depth amount of the video. As long as the correction parameter 1006 is an element that determines the final pop-out amount and depth amount, the calculation form does not matter.

  In the example of FIG. 7, the simple shift operation of the subject image in the left-eye image and the right-eye image has been described as the parallax correction method, but this is an example and the correction method is not limited. For example, it is possible to perform a process of spatially enhancing a stereoscopic effect using affine transformation or the like by the parallax correction circuit 515. The parallax correction circuit 515 performs parallax amount correction related to the left-eye image and the right-eye image based on the correction parameter corresponding to the aspect ratio. Data on the amount of parallax corrected by the parallax correction circuit 515 is stored in the storage unit 508.

  The storage unit 508 stores the combined image data generated by the image combining circuit 513 and the parallax map data generated by the parallax map generating circuit 514 and corrected by the parallax correction circuit 515. The system control unit 509 controls the recording medium control I / F unit 510 to perform processing for recording the composite image data and the parallax map data on a removable recording medium 511 such as a semiconductor memory. Further, image processing can be performed by the external I / F unit 512 outputting data to an external device such as a computer.

  When reproducing the image data, the system control unit 509 controls the recording medium control I / F unit 510 to read out the composite image data and the parallax map data and store them in the storage unit 508. From these data, the left-eye image data and the right-eye image data are read. Image data is generated. The information on the parallax map indicates the amount of parallax between the centroid position of the subject image in the composite image and the centroid position of the subject image in the left-eye image or the right-eye image. Specifically, as described with reference to FIG. 5B, a parallax amount 612 corresponding to one-half of the parallax amount 610 and a parallax amount 613 corresponding to one-half of the parallax amount 611 are used. . The subject image in the composite image is moved to the position of the center of gravity in the left-eye image according to the amount of parallax, and left-eye image data is generated. Similarly, if the sign of the parallax amount indicated by the parallax map is inverted, the parallax amount up to the center of gravity in the right-eye image is obtained. Thus, the subject image in the composite image is moved to the position of the center of gravity in the right-eye image in accordance with the sign-inverted parallax amount, and right-eye image data is generated.

  In this embodiment, the imaging signal processing circuit 506 performs data separation processing for distributing image data read from the image sensor 505 into left-eye image data and right-eye image data, and addition synthesis of the left-eye image data and right-eye image data. The image composition circuit 513 takes charge of the processing. The present invention is not limited to this, and the imaging signal processing circuit 506 may perform a data separation process and a synthesis process. In addition, after the parallax map generation circuit 514 generates the parallax map data, the parallax correction circuit 515 performs the parallax amount correction according to the aspect ratio, but the parallax map generation circuit 514 performs the parallax map generation and the parallax correction processing. But you can.

  As described above, according to the present embodiment, when the image range including the subject image expressing the sense of depth is excluded from the image frame by the trimming process according to the aspect ratio setting, the stereoscopic effect of the video is corrected by correcting the parallax amount. Stereo image processing can be realized so as not to damage the image.

501 Lens unit 505 Imaging device 506 Imaging signal processing circuit 508 Storage unit 509 System control unit 513 Image composition circuit 514 Parallax map generation circuit 515 Parallax correction circuit

Claims (9)

Acquisition means for acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
Image processing means for generating composite image data by combining the image signals of the different viewpoints;
Parallax amount calculating means for calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected comprising a parallax correction means for,
When the parallax correction unit corrects the parallax amount by a correction parameter determined from the first aspect ratio and the second aspect ratio, the parallax correction unit converts the parallax correction unit to the second aspect ratio among the images having the first aspect ratio. An image pickup apparatus , wherein the amount of parallax is increased as the image range excluded from the image frame due to the change is larger . Acquisition means for acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
Image processing means for generating composite image data by combining the image signals of the different viewpoints;
Parallax amount calculating means for calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected comprising a parallax correction means for,
The parallax amount calculating means calculates a positional deviation amount of the subject image in the images of different viewpoints as a parallax amount with reference to the position of the subject image in the composite image data generated by the image processing means, and the parallax amount An imaging apparatus characterized by storing parallax map data including When the parallax correction unit corrects the parallax amount by a correction parameter determined from the first aspect ratio and the second aspect ratio, the parallax correction unit converts the parallax correction unit to the second aspect ratio among the images having the first aspect ratio. The imaging apparatus according to claim 2 , wherein the parallax amount is increased as the image range excluded from the image frame due to the change is larger . The correction parameter indicates a ratio obtained by dividing the parallax amount of the subject at the second aspect ratio by the parallax amount of the subject at the first aspect ratio;
The parallax correction unit performs parallax correction by multiplying the parallax amount calculated by the parallax amount calculation unit by the ratio, and stores parallax map data including the corrected parallax amount in the storage unit. The imaging device according to claim 1 or 3 .   5. The imaging device according to claim 1, further comprising: an imaging unit having a plurality of photoelectric conversion units that receive and photoelectrically convert light beams respectively passing through different areas of the exit pupil of the imaging optical system for each microlens. The imaging apparatus of Claim 1. Acquisition means for acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
Image processing means for generating composite image data by combining the image signals of the different viewpoints;
Parallax amount calculating means for calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected comprising a parallax correction means for,
When the parallax correction unit corrects the parallax amount by a correction parameter determined from the first aspect ratio and the second aspect ratio, the parallax correction unit converts the parallax correction unit to the second aspect ratio among the images having the first aspect ratio. The image processing apparatus , wherein the amount of parallax is increased as the image range excluded from the image frame due to the change is larger . Acquisition means for acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
Image processing means for generating composite image data by combining the image signals of the different viewpoints;
Parallax amount calculating means for calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected comprising a parallax correction means for,
The parallax amount calculating means calculates a positional deviation amount of the subject image in the images of different viewpoints as a parallax amount with reference to the position of the subject image in the composite image data generated by the image processing means, and the parallax amount An image processing apparatus characterized in that parallax map data including is stored in a storage means. A control method executed by an imaging device or an image processing device ,
A data acquisition step of acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
An image synthesizing step of generating synthesized image data form if the image signal of the different viewpoints,
A parallax amount calculating step of calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected have a parallax correction step of,
In the parallax correction step, when the parallax amount is corrected by a correction parameter determined from the first aspect ratio and the second aspect ratio, among the images having the first aspect ratio, the second aspect ratio is corrected. the higher the image area is large are excluded from the image frames, processing to increase the amount of parallax to that control method characterized by being performed by changes. A control method executed by an imaging device or an image processing device ,
A data acquisition step of acquiring image signals of different viewpoints from output signals of a plurality of photoelectric conversion units;
An image synthesizing step of generating synthesized image data form if the image signal of the different viewpoints,
A parallax amount calculating step of calculating the parallax amount of the subject from the image signals of the different viewpoints;
When there is an image range that is excluded from the image frame due to a change from the first aspect ratio related to the captured image to the second aspect ratio related to image recording or playback, the parallax amount of the subject in the images of different viewpoints is corrected have a parallax correction step of,
In the parallax amount calculating step, the positional deviation amount of the subject image in the image of the different viewpoint is calculated as the parallax amount with reference to the position of the subject image in the composite image data generated in the image synthesizing step, and the parallax amount The control method characterized by performing the process which memorize | stores the parallax map data containing this in a memory | storage means.

Images