JP5956732B2 - Imaging system - Google Patents

Description

  The present invention relates to an imaging system for obtaining a stereoscopic image using binocular parallax.

  The demand for 3D images is high, and research and development has been conducted for a long time. The left and right human eyes each acquire a two-dimensional image, but are approximately 6 to 7 cm apart in the horizontal direction, and there is a parallax between the image by the right eye and the image by the left eye depending on the distance to the object. The brain processes these two images to form a three-dimensional stereoscopic image. As a conventional apparatus for photographing a stereoscopic image, there are many systems in which two cameras or two image sensors are prepared, as in the case of human eyes. A sense of depth can be obtained by separately presenting images taken by the left and right cameras.

  By the way, as one of the methods for detecting the focus state of the photographic lens, Patent Document 1 discloses an apparatus that performs pupil division-type focus detection using a two-dimensional sensor in which a microlens is formed in each pixel of the sensor. ing. In the apparatus of Patent Document 1, the photoelectric conversion unit of each pixel constituting the sensor is divided into a plurality of parts, and the divided photoelectric conversion unit is configured to receive different regions of the pupil of the photographing lens through the microlens. Yes. The focus state of the imaging optical system can be detected from the amount of image shift between the left light receiving unit output and the right light receiving unit output under each microlens.

  Patent Document 1 also mentions the realization of stereoscopic video shooting using this apparatus. A first image is created from the output of the left light receiving unit of all the pixels, and a second image is created from the output of the right light receiving unit of all the pixels. A stereoscopic video can be obtained by reproducing the first and second videos with the stereoscopic video reproduction device.

JP-A-58-24105

  By the way, the stereoscopic effect of the subject that the viewer feels becomes thinner as the parallax between the first video and the second video becomes smaller, and the stereoscopic effect can be felt more as the parallax becomes larger.

  However, although the above-described conventional technology can shoot a stereoscopic image, the photographer cannot select a stereoscopic effect. Also, when shooting a 3D image, there are a variety of photographer preferences, such as enhancing only the desired subject to make it appear three-dimensional, and making a close subject look closer and a far subject look farther. It did not correspond. In addition, a technique for three-dimensionally processing an image taken by an application has been developed, but it has been difficult to reflect the intention of the photographer.

  Therefore, an object of the present invention is to provide an imaging system capable of capturing a more effective stereoscopic image even when capturing a stereoscopic image using one imaging element.

An imaging system according to the present invention is an imaging system including an imaging device that captures an image of a subject formed by a photographing lens, and includes a subject selection unit that selects a main subject and a plurality of photoelectric conversion units per unit pixel. An image sensor having a photoelectric conversion unit that selects the photoelectric conversion unit of each unit pixel according to whether or not it is the main subject, and a signal from the selected photoelectric conversion unit Generating means for generating two image signals having parallax, and the plurality of photoelectric conversion units are arranged in a line in a direction in which the parallax is applied, and the photoelectric conversion unit selection unit is arranged on the main subject. For the corresponding pixels, the combination of the photoelectric conversion units is selected so that the parallax is given in the first direction and the parallax is maximized, and the pixels other than the pixels corresponding to the main subject are selected. Selecting the combination of the photoelectric conversion units so as to give a parallax in the first direction and to be inside the combination of the photoelectric conversion units selected for the pixel corresponding to the main subject. It is characterized by.
An imaging system according to the present invention is an imaging system including an imaging device that captures an image of a subject formed by a photographing lens, and includes a subject selection unit that selects a main subject and a plurality of photoelectric conversions per unit pixel. An image sensor having a unit, a photoelectric conversion unit selecting unit that selects the photoelectric conversion unit of each of the unit pixels according to whether or not it is the main subject, and a signal from the selected photoelectric conversion unit Generating means for generating two image signals having parallax, and the plurality of photoelectric conversion units are arranged in a line in a direction in which the parallax is applied, and the photoelectric conversion unit selection unit includes the main conversion unit. For the pixel corresponding to the subject, the combination of the photoelectric conversion units is selected so as to give a parallax in the first direction, and for the pixels other than the pixel corresponding to the main subject, the reverse of the first direction First As direction put parallax, and selects a combination of the photoelectric conversion unit.
An imaging system according to the present invention is an imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens, the subject selecting means for selecting a main subject, and the main subject. Depending on whether or not, an imaging device selection unit that selects two imaging devices from the plurality of imaging devices, and generates two image signals having parallax based on signals from the selected two imaging devices And the plurality of imaging devices are arranged in a line in a direction in which parallax is applied, and the imaging device selection unit performs parallax in a first direction for an area corresponding to the main subject. The combination of the imaging devices is selected so that the parallax is maximized, and the areas other than the area corresponding to the main subject are parallaxed in the first direction, and So also the inside of a combination of the imaging device is selected for a region corresponding to the main object, and selects the combination of the imaging device.
An imaging system according to the present invention is an imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens, the subject selecting means for selecting a main subject, and the main subject. Depending on whether or not, an imaging device selection unit that selects two imaging devices from the plurality of imaging devices, and generates two image signals having parallax based on signals from the selected two imaging devices And the plurality of imaging devices are arranged in a line in a direction in which parallax is applied, and the imaging device selection unit performs parallax in a first direction for an area corresponding to the main subject. The combination of the imaging devices is selected so as to apply a parallax, and the region other than the region corresponding to the main subject is subjected to the parallax in the second direction opposite to the first direction. Combination And selects allowed.
The imaging system according to the present invention is an imaging system including an imaging device that captures an image of a subject formed by a photographing lens, and includes two detection units for detecting a distance to the subject and two per unit pixel. An image sensor having the above photoelectric conversion unit, a photoelectric conversion unit selection unit that selects the photoelectric conversion unit of each unit pixel according to the distance to the subject, and a signal from the selected photoelectric conversion unit Generating means for generating two image signals having a parallax based on the photoelectric conversion unit, wherein the photoelectric conversion units are arranged in a line in a direction in which the parallax is applied, and the photoelectric conversion unit selection unit includes the detection unit For the pixel corresponding to the subject whose distance detected by the step is less than the predetermined distance, the combination of the photoelectric conversion units is set so that the parallax is given in the first direction and the parallax is maximized. The pixel corresponding to the subject whose distance detected by the detection means is equal to or greater than the predetermined distance is applied to the subject having a parallax in the first direction and less than the predetermined distance. The combination of the photoelectric conversion units is selected so as to be inside the combination of the photoelectric conversion units selected for the corresponding pixel.
The imaging system according to the present invention is an imaging system including an imaging device that captures an image of a subject formed by a photographing lens, and includes two detection units for detecting a distance to the subject and two per unit pixel. An image sensor having the above photoelectric conversion unit, a photoelectric conversion unit selection unit that selects the photoelectric conversion unit of each unit pixel according to the distance to the subject, and a signal from the selected photoelectric conversion unit Generating means for generating two image signals having a parallax based on the photoelectric conversion unit, wherein the photoelectric conversion units are arranged in a line in a direction in which the parallax is applied, and the photoelectric conversion unit selection unit includes the detection unit For the pixel corresponding to the subject whose distance detected by the step is less than the predetermined distance, the combination of the photoelectric conversion units is selected so as to give a parallax in the first direction, and is detected by the detection unit. For a pixel corresponding to a subject whose distance is equal to or greater than the predetermined distance, the combination of the photoelectric conversion units is selected so as to give a parallax in a second direction opposite to the first direction. To do.
An imaging system according to the present invention is an imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens, and includes detection means for detecting a distance to the subject, An imaging device selection unit that selects two imaging devices from the plurality of imaging devices according to a distance, and generation that generates two image signals having parallax based on signals from the selected two imaging devices And the plurality of imaging devices are arranged in a line in the direction of parallax, and the imaging device selection unit corresponds to a subject whose distance detected by the detection unit is less than a predetermined distance For the region to be selected, the combination of the imaging devices is selected so that the parallax is given in the first direction and the parallax is maximized, and the distance detected by the detection unit is the predetermined distance The region corresponding to the subject that is greater than or equal to the distance is inside the combination of the imaging devices selected so as to add parallax in the first direction and the region corresponding to the subject that is less than the predetermined distance. Thus, the combination of the imaging devices is selected.
An imaging system according to the present invention is an imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens, and includes detection means for detecting a distance to the subject, An imaging device selection unit that selects two imaging devices from the plurality of imaging devices according to a distance, and generation that generates two image signals having parallax based on signals from the selected two imaging devices And the plurality of imaging devices are arranged in a line in the direction of parallax, and the imaging device selection unit corresponds to a subject whose distance detected by the detection unit is less than a predetermined distance For the area to be selected, the combination of the imaging devices is selected so as to give parallax in the first direction, and the area corresponding to the subject whose distance detected by the detection means is equal to or greater than the predetermined distance. For, wherein the first direction so as to give a parallax in a second direction opposite, and selects a combination of the imaging device.

  According to the present invention, it is possible to provide an imaging system capable of capturing a more effective stereoscopic image even when capturing a stereoscopic image using one imaging element.

1 is a block diagram of a solid-state imaging device according to a first embodiment of the present invention. FIG. 3 is a diagram for explaining a configuration of an image sensor according to the first embodiment of the present invention. FIG. 3 is a diagram for explaining a configuration of a unit pixel of the image sensor according to the first embodiment of the present invention. 1 is a configuration diagram of an image sensor according to a first embodiment of the present invention. FIG. 1 is a configuration diagram of an image sensor according to a first embodiment of the present invention. FIG. 1 is a configuration diagram of an image sensor according to a first embodiment of the present invention. FIG. 2 is a drive timing chart of the image sensor according to the first embodiment of the present invention. FIG. 3 is a diagram for explaining the principle of stereoscopic image shooting using the image sensor of the first embodiment of the present invention. FIG. 3 is a diagram for explaining the principle of stereoscopic image shooting using the image sensor of the first embodiment of the present invention. The flowchart for demonstrating operation | movement of the 1st Embodiment of this invention. The composition example for demonstrating stereoscopic image photography. An image example obtained by stereoscopic image shooting. The figure which shows the photoelectric conversion part selection switch used in the modification of 1st Embodiment. The figure which shows the structure of an image pick-up element at the time of using the photoelectric conversion part selection switch of FIG. The block diagram of the imaging system which concerns on the 2nd Embodiment of this invention. The flowchart for demonstrating operation | movement of the 2nd Embodiment of this invention. The block diagram of the solid-state imaging device which concerns on the 3rd Embodiment of this invention. The flowchart for demonstrating operation | movement of the 3rd Embodiment of this invention. The composition example for demonstrating the stereo image imaging | photography of 3rd Embodiment. An example of a distance map of an image obtained by stereoscopic image shooting. An image example obtained by stereoscopic image shooting. The flowchart for demonstrating the operation | movement of the 4th Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing the overall configuration of the imaging apparatus according to the first embodiment of the present invention. In FIG. 1, an image sensor 101 is a CMOS type image sensor, and captures an image formed by a photographing lens (not shown). An AFE (Analog Front End) 102 is a signal processing circuit that performs amplification of the signal from the image sensor, black level adjustment (OB clamp), and the like. The AFE 102 receives the OB clamp timing, the OB clamp target level, and the like from the timing generation circuit 110, and performs processing according to the OB clamp timing and the OB clamp target level. Then, the processed analog signal is converted into a digital signal. A DFE (Digital Front End) 103 receives a digital signal of each pixel converted by the AFE 102 and performs digital processing such as image signal correction and pixel rearrangement. The image processing apparatus 105 performs processing such as performing development processing to display an image on the display circuit 108 and recording the image on the recording medium 109 via the control circuit 106. In addition, the control circuit 106 also performs control such as sending an instruction to the timing generation circuit 110 in response to an instruction from the operation unit 107. The recording medium 109 includes a compact flash (registered trademark) memory. The memory circuit (storage unit) 104 is used as a working memory at the development stage of the image processing apparatus 105. Further, it is also used as a buffer memory when imaging is continued and development is not in time. The operation unit 107 includes a power switch for activating the digital camera, a shooting preparation operation start such as photometry processing and distance measurement processing, and a signal read from the image sensor 101 by driving a mirror and a shutter to process a recording medium. A shutter switch for instructing the start of a series of imaging operations written to 109 is included.

  2 and 3 are diagrams for explaining the configuration of the image sensor 101. FIG. 2 illustrates some pixels of the image sensor 101. In FIG. 2, only the pixels of 6 rows × 6 columns are shown, but in actuality, the number of pixels is several thousand rows × thousand columns.

  The image sensor 101 includes R pixels, G pixels, and B pixels provided with color filters that transmit each wavelength band of R (red), G (green), and B (blue) on the photoelectric conversion unit (PD). Have. R, G, and B are arranged in a Bayer shape, and one microlens (ML) is arranged for each pixel. Each pixel is divided into four in one pixel in the horizontal direction as indicated by a broken line.

  FIG. 3 is a diagram illustrating the structure of one pixel (unit pixel) of the image sensor 101. In the unit pixel 30 of the image sensor 101, four photoelectric conversion units (PD) 301 to 304 are arranged along the horizontal direction as shown in the figure. In the present embodiment, the number of divisions per pixel is described as four, but this is not a limitation. It may be divided not only in the horizontal direction but also in the vertical direction.

  FIG. 4 is a diagram illustrating an example of a circuit of the image sensor 101 according to the present embodiment. The circuit of FIG. 4 is a circuit of the unit pixel 30 shown in FIG. 3, and is composed of four photoelectric conversion units. The PD (photodiode) 401 receives the light image formed by the photographing lens, generates electric charge, and accumulates it. The transfer switch 402 is composed of a MOS transistor. The charge accumulated in the PD 401 is transferred to an FD (floating diffusion unit) 404 via the transfer switch 402, and the charge is converted into a voltage, which is output from the source follower amplifier 405. The selection switch 406 collectively outputs the pixel signals for one row to the vertical output line 407. The reset switch 403 resets the potential of the FD 404 and the potential of the PD 401 to VDD via the transfer switch 402. Four circuits having such a configuration are arranged in the horizontal direction to constitute a unit pixel.

  FIG. 5 is a block diagram illustrating a configuration example of the image sensor. In FIG. 5, only 2 × 2 pixels are shown, but in actuality, it is composed of several thousand × several thousand pixels. The vertical shift register 501 outputs signals such as row selection lines Pres1, Ptx1, and Psel1 to the pixel region 508. The pixel region 508 has a configuration in which unit pixels 509 are arranged in a grid pattern, and the unit pixel 509 includes four photoelectric conversion units 510. The circuit configuration of the unit pixel 509 is as shown in FIG.

  A current source 507 is connected to each vertical output line 407. The readout circuit 502 inputs the pixel signal on the vertical output line and outputs the pixel signal to the differential amplifier 505 via the n-channel MOS transistor 503. Further, the noise signal is output to the differential amplifier 505 via the n-channel MOS transistor 504. The horizontal shift register 506 controls on / off of the transistors 503 and 504, and the differential amplifier 505 outputs a difference between the pixel signal and the noise signal.

  The gate of the transfer switch 402 in FIG. 4 is connected to a first row selection line Ptx1 (FIG. 5) arranged extending in the horizontal direction. The gates of other similar transfer switches 402 arranged in the same row are also commonly connected to the first row selection line Ptx1. The gate of the reset switch 403 in FIG. 4 is connected to a second row selection line Pres1 (FIG. 5) that is arranged extending in the horizontal direction. The gates of other similar reset switches 403 arranged in the same row are also commonly connected to the second row selection line Pres1. The gate of the selection switch 406 in FIG. 4 is connected to a third row selection line Psel1 (FIG. 5) arranged extending in the horizontal direction. The gates of other similar selection switches 406 arranged in the same row are also commonly connected to the third row selection line Psel1. These first to third row selection lines Ptx1, Pres1, and Psel1 are connected to and driven by the vertical shift register 501. In the remaining rows shown in FIG. 5, pixels having the same configuration and row selection lines are provided. These row selection lines are row selection lines Ptx2, Pres2, Psel2, etc. connected to the vertical shift register 501.

  The source of the selection switch 406 is connected to a terminal Vout of a vertical output line that extends in the vertical direction. The sources of similar selection switches 406 of the photoelectric conversion units arranged in the same column are also connected to the terminal Vout of the vertical output line. In FIG. 5, the terminal Vout of the vertical output line is connected to a constant current source 507 which is a load means.

  FIG. 6 is a diagram showing a circuit example for one column of the read circuit 502 shown in FIG. There are a portion surrounded by a broken line as many as the number of columns (the number of photoelectric conversion units in the horizontal direction), and a terminal Vout is connected to each vertical output line.

  FIG. 7 is a timing chart showing an operation example of a CMOS type image sensor. Prior to the reading of the optical signal charge from the photodiode 401, the gate line Pres1 of the reset switch 403 becomes high level. As a result, the gate of the amplification MOS transistor is reset to the reset power supply voltage. After the gate line Pres1 of the reset switch 403 returns to the low level and the gate line Pc0r (FIG. 6) of the clamp switch becomes high level, the gate line Psel1 of the selection switch 406 becomes high level. As a result, the reset signal (noise signal) on which the reset noise is superimposed is read to the vertical output line Vout and clamped to the clamp capacitor C0 of each column. Next, after the gate line Pc0r of the clamp switch returns to the low level, the gate line Pctn of the noise signal side transfer switch becomes the high level, and the reset signal is held in the noise holding capacitor Ctn provided in each column. Next, after the gate line Pcts of the pixel signal side transfer switch is set to the high level, the gate line Ptx1 of the transfer switch 402 is set to the high level, and the optical signal charge of the photodiode 401 is transferred to the gate of the amplifier 405 at the same time. The optical signal is read out to the vertical output line Vout. Next, after the gate line Ptx1 of the transfer switch 402 returns to the low level, the gate line Pcts of the pixel signal side transfer switch becomes the low level. As a result, the amount of change (optical signal) from the reset signal is read out to the signal holding capacitors Cts provided in each column. With the operation so far, the optical signal of the photoelectric conversion unit 510 connected to the first row is held in the signal holding capacitors Ctn and Cts connected to the respective columns.

  Thereafter, the horizontal transfer switch gates in each column are sequentially set to the high level by the signal Ph supplied from the horizontal shift register 506. The voltages held in the signal holding capacitors Ctn and Cts are sequentially read out to the horizontal output lines Chn and Chs, subjected to differential processing by the output amplifier, and sequentially output to the output terminal OUT. The horizontal output lines Chn and Chs are reset to the reset voltages VCHRN and VCHRS by a reset switch between signal readings of each column. Thus, reading of the pixels connected to the first row is completed.

  Similarly, the signals from the photoelectric conversion units connected in the second and subsequent rows are sequentially read by signals from the vertical shift register 501, and the reading of all the pixels (the signals accumulated in all the photoelectric conversion units) is completed. . Note that the image sensor of the present embodiment has a configuration in which four photoelectric conversion units are arranged in the horizontal direction per pixel. The charges accumulated in each photoelectric conversion unit are read out, and four signal data are held per pixel. This signal data is processed by the image processing device 105 and displayed as an image on the display circuit 108 or written to the recording medium 109. Of course, all the signal data may be written in the recording medium 109 and image processing may be performed by a PC or the like. In addition, in the case of the two-dimensional imaging mode, an image may be generated with an addition signal of four signal data as one pixel.

In the first embodiment of the present invention described with reference to FIGS. 4 to 7, one vertical output line is provided for one photoelectric conversion unit. However, the present invention is not limited to this, and a plurality of photoelectric conversions are performed. Alternatively, the vertical output lines may be shared by the units. In this case, the signals of the plurality of photoelectric conversion units sharing the vertical output line are sequentially read out at different times.

Next, the principle of stereoscopic image shooting using the image sensor 101 having the pixel structure as shown in FIG. 2 will be described. FIGS. 8 and 9 are diagrams for explaining the principle of stereoscopic image shooting using the image sensor 101. A unit pixel 30 in FIG. 8 represents one pixel of the image sensor 101 as in FIG. 3, and includes four photoelectric conversion units 301 to 304. The parallax 803 is the distance between the pupil centroid 801 of the photoelectric conversion unit 302 and the pupil centroid 802 of the photoelectric conversion unit 303 in the photographing lens 804. As shown in FIG. 8, the light beam that has passed through the exit pupil 801 passes through the microlens (ML) and the color filter (CF) and forms an image on the photoelectric conversion unit 302. On the other hand, the light beam that has passed through the exit pupil 802 passes through the microlens (ML) and the color filter (CF) and forms an image on the photoelectric conversion unit 303. The obtained information has a shift corresponding to the parallax 803 between the photoelectric conversion unit 302 and the photoelectric conversion unit 303. Note that the closer the object is to the front, the larger the parallax 803 becomes.

  In contrast to FIG. 8, FIG. 9 schematically illustrates the case where the photoelectric conversion unit 301 and the photoelectric conversion unit 304 are used. The parallax 903 is a distance between the pupil centroid 901 of the photoelectric conversion unit 301 and the pupil centroid 902 of the photoelectric conversion unit 304 in the photographing lens 804. The light beam that has passed through the exit pupil 901 passes through the microlens (ML) and the color filter (CF) and forms an image on the photoelectric conversion unit 301. On the other hand, the light beam that has passed through the exit pupil 902 passes through the micro lens (ML) and the color filter (CF) and forms an image on the photoelectric conversion unit 304. The amount of deviation (parallax 903) is greater when the outer photoelectric conversion units 301 and 304 are used than when the inner photoelectric conversion units 302 and 303 are used. That is, when a subject at the same position is photographed and stereoscopic reproduction is performed in the same manner, the photoelectric conversion unit 301 and the photoelectric conversion are more effective than the stereoscopic image obtained from the signals obtained from the photoelectric conversion unit 302 and the photoelectric conversion unit 303. The stereoscopic image obtained from the signal obtained from the unit 304 seems to be positioned in front.

  FIG. 10 is a flowchart illustrating the flow of stereoscopic image acquisition performed by the imaging system of the present embodiment. First, photographing is performed using the imaging apparatus of FIG. 1 (step S1001). The operation of the operation unit 107 activates the imaging apparatus, performs photometry / ranging processing on a desired subject, and presses the shutter to shoot. Note that the photographer may select a subject position in the screen during the photographing operation.

  Subsequently, the electric charge accumulated in the photoelectric conversion unit (PD) of the image sensor 101 is read according to the above-described reading method (step S1002). Here, the charges of all four PDs of each pixel are read out. The read signal is stored in the memory circuit 104.

  The main subject is extracted (subject selection) from the read signal information (step S1003). For the extraction of the main subject, the photographer may select a position on the screen at the time of shooting, or the distance information is calculated from the read signal and the subject at the shortest distance is set as the main subject. Also good. There are various methods for calculating the distance to the subject, but the image sensor 101 according to the present embodiment has a feature that a signal having parallax can be obtained at each coordinate, and triangulation method using parallax is used. Is preferred. For example, there is a method in which the screen is divided into predetermined blocks composed of a plurality of pixels, and the distance is calculated for each block. The extraction of the desired main subject can be obtained from the calculated distance information. For example, there is a method in which the coordinates of the main subject are the center, the distance between the surrounding areas and the distance in the center coordinates of the main subject are compared, and if they are similar, the surrounding area is selected as the main subject. This process is repeated, and if there is a difference equal to or greater than a certain threshold between the distance in the central coordinates and the distance between the surrounding areas, the area is not a main subject and the extraction of the main subject is terminated. There is also a method of detecting an edge forming a selected main subject outline in a photographed image and extracting the inner region as a main subject region. The main subject can also be extracted by other known techniques.

  FIG. 11 is an example of a captured image. For example, it is assumed that the main subject 1101 has been selected at the time of shooting in step S1001. First, the distance dc of the selected coordinates (Xc, Yc) is calculated from the captured data. If the distance between the surrounding areas is compared with the distance dc and coincides with dc, the pixel of the coordinates is extracted as the main subject area. The determination as to whether or not the distance to the main subject coincides with the distance dc may be made by setting a threshold value such as coincidence if the distance is within a range of 0.90 × dc to 1.10 × dc. Further, the distance distribution may be subjected to cluster analysis and statistically determined. The above processing is performed, and coordinate information corresponding to the area extracted as the main subject is stored in the memory circuit 104 (step S1004).

  From step S1005, left eye image and right eye image creation processing is performed. In order to create image data, pixel data is sequentially read out. The processing from step S1005 to step S1009 is repeated until the end of all pixels. The pixel data is read by scanning, for example, starting from the upper left of the image, reading in the horizontal direction, and reading the next row when one row is completed.

  It is determined whether or not the read pixel (X, Y) is the main subject region (step S1006). The main subject area information processed in steps S1003 and S1004 is collated. If it is the main subject area, the process proceeds to step S1007, and if not, the process proceeds to step S1008.

  In the case of the area of the main subject, image data (having parallax in the first direction) is obtained using a signal from the outside PD. Specifically, referring to FIG. 9, the data of the PD 304 is read for the left eye image, and the data of the PD 301 is read for the right eye image (step S1007). On the other hand, when the region is not the main subject region (a pixel other than the pixel corresponding to the main subject), the image data is obtained using a signal from the inner PD. That is, the data of the PD 303 is read for the left eye image, and the data of the PD 302 is read for the right eye image (step S1008).

  The PD data selected according to whether or not the subject is the main subject is written as the data of the pixel (step S1009). In addition, since it consists of four photoelectric conversion parts per pixel, the quantity of light incident on one photoelectric conversion part is 1/4, and the signal output obtained here is an optical signal of almost 1/4. Considering the division of the photoelectric conversion unit, the image processing apparatus 105 performs gain correction on the obtained signal output. Steps S1005 to S1009 are repeated until the end of all pixels (step S1010).

  As described above, a series of processing from image capturing to image data creation in the imaging system of the present embodiment is completed. The written left-eye image data and right-eye image data are developed by the image processing apparatus 105 and stored in the recording medium 109. FIG. 12 shows an example of the left-eye image 1201 and the right-eye image 1202 that are stored. The main subject 1203 of the image for the left eye consists of signals read from the PD 304, and the other area 1205 consists of signals read from the PD 303. On the other hand, the main subject 1204 of the right-eye image is composed of signals read from the PD 301, and the other area 1206 is composed of signals read from the PD 302. If the left-eye image 1201 and the right-eye image 1202 obtained in this way and saved by the recording medium 109 are reproduced by a stereoscopic image reproduction device, the main subject can be seen more stereoscopically.

  In the present embodiment, the charge data accumulated in the image sensor 101 is read for all PDs. However, when the charge is read from the image sensor 101, the main subject is extracted and the PD to be used is selected. good. In that case, for example, it can be implemented by providing a photoelectric conversion unit selection switch 130 configured as shown in FIG. A photoelectric conversion selection signal controlled by the photoelectric conversion selection means is input to a terminal 1301 in FIG. The terminal 1302 is connected to the vertical output line, and charges from the photoelectric conversion unit are input. And the electric charge from the photoelectric conversion part selected by control of the photoelectric conversion selection signal can be read. Further, a current source 1303 is connected to the output terminal Vout.

  FIG. 14 is a diagram illustrating a configuration of an image sensor when the photoelectric conversion unit selection switch 130 of FIG. 13 is used. Each configuration from the vertical output line 407, the vertical shift register 501 to the horizontal shift register 506, and the pixel region 508 to the photoelectric conversion unit 510 is the same as that in FIG. The current source 507 installed on each vertical output line in FIG. 5 is replaced with the current source 1303 in the photoelectric conversion unit selection switch 130, and is not necessary in the configuration of FIG. A photoelectric conversion unit selection switch 130 is disposed between the vertical output line 407 and the readout circuit 60. Note that the readout circuit 60 in FIG. 14 is a part of the readout circuit in FIG. In the case of FIG. 14, four PDs are arranged horizontally per unit pixel, one of the two left PDs is selected and read, and one of the two right PDs is selected and read. Since only the selected PD is read out, the number of readout circuits can be reduced. Also, the number of current sources can be reduced similarly.

  In this embodiment, the number of PDs per pixel is four, but the number of divisions may be more than that. In particular, the larger the number of divisions in the horizontal direction, the greater the difference in parallax. For example, the PD to be selected can be changed depending on the subject, or the combination of PDs to be used can be selected according to the photographer's preference.

  In addition, for pixels in a region other than the main subject, the PD selects PD 302 or 301 for the left eye and PD 303 or 304 for the right eye, and adds a parallax in the reverse direction (second direction) different from the above-described parallax. Image data may be created. By selecting the PD in this way, the contrast of the parallax information can be given to the main subject and the region that is not so, and a stereoscopic image can be created more effectively.

  In the present embodiment, only still image shooting has been described, but it is also suitable for moving image shooting. In the case of a moving image, the main subject can be selected in the previous frame, and the processing can be speeded up by tracking it.

  Note that it is possible to perform both two-dimensional image photographing and three-dimensional image photographing using the image sensor of the present embodiment. In the case of two-dimensional image shooting, charges from all PDs in one pixel are added to obtain an output of one pixel. The addition process may be performed by an image processing apparatus or the like after reading the charges of all PDs. In the case of a pixel having four PDs per pixel as shown in FIG. 2, the output is quadrupled. On the other hand, in the case of three-dimensional image shooting, since the output is obtained from the charge of one PD per pixel for both the left-eye image and the right-eye image, the output is about ¼ for the two-dimensional image. Therefore, when only one divided pixel is read for a three-dimensional image, gain correction is performed. By applying the gain correction, the brightness of the images can be made uniform in the two-dimensional image photographing and the three-dimensional image photographing.

  In addition, by providing an adder circuit in the image sensor, an output signal for one pixel can be calculated and read out in the case of two-dimensional image shooting. As an addition method, there are a method of adding by FD, a method of adding by a readout circuit, and the like. Also in this case, it is desirable to perform gain correction so as to eliminate the output level difference caused by addition (two-dimensional image shooting) and non-addition (three-dimensional image shooting).

(Second Embodiment)
An imaging system according to a second embodiment of the present invention will be described with reference to the drawings. FIG. 15 is a diagram illustrating a configuration of an imaging system according to the second embodiment of the present invention. The second embodiment includes a plurality of imaging devices 1501 to 1504. Each of the imaging devices 1501 to 1504 has one imaging element. Note that this image sensor does not have to be pupil-divided. The electric charges accumulated in the imaging elements respectively included in the imaging devices 1501 to 1504 are processed by the image processing device 1505, and the image data is stored in the recording medium 1507. This imaging system also includes a distance measuring system 1508, and has a function of dividing the screen into predetermined blocks made up of a plurality of pixels and detecting the distance for each block in the screen.

  FIG. 16 is a flowchart illustrating a flow of stereoscopic image acquisition performed by the imaging system of the second embodiment. First, photographing is performed (step S1601). Here, although not shown, the control circuit controls the imaging devices 1501 to 1504 to perform imaging so as to perform imaging. In addition, the photographer may select the main subject during this photographing operation.

  Subsequently, the electric charge accumulated in the image sensor of each imaging device is read and stored in the memory circuit 1506 (step S1602). By reading the charge of each imaging device, four pieces of charge data having different parallax information for each pixel are stored in the memory circuit 1506.

  The main subject is extracted from the read signal information and the main subject area data is stored (steps S1603 and S1604). The main subject extraction method can be performed in the same manner as in the first embodiment. The extracted main subject area information is stored in the memory circuit 1506.

  From step S1605, left-eye image and right-eye image creation processing is performed. In order to create image data, the pixel data stored in the memory circuit 1506 is sequentially read out. The processing from step S1605 to step S1609 is repeated until the end of all pixels. The pixel data is read by scanning, for example, starting from the upper left of the image, reading in the horizontal direction, and reading the next row when one row is completed.

  It is determined whether or not the read pixel (X, Y) is the main subject region (step S1606). The main subject area information processed in steps S1603 and S1604 is compared, and if it is the main subject area, the process proceeds to step S1607, and if not, the process proceeds to step S1608.

  In the case of the area of the main subject, image data is obtained using the signal from the outer camera (imaging device) (imaging device selection). That is, the data of the imaging device 1504 is read for the left eye image, and the data of the imaging device 1501 is read for the right eye image (step S1607). On the other hand, if it is not the main subject area, it is set as image data using a signal from the inner camera. That is, the data of the imaging device 1503 is read for the left eye image, and the data of the imaging device 1502 is read for the right eye image (step S1608). Data of the imaging device selected according to whether or not it is the main subject is written as data of the pixel (step S1609). Steps S1605 to S1609 are repeated until the end of all pixels (step S1610).

  As described above, a series of processing from image capturing to image data creation in the imaging system of the present embodiment is completed. The written left-eye image data and right-eye image data are developed by the image processing device 1505 and stored in the recording medium 1507.

  When the left-eye image and the right-eye image obtained in this way and stored on the recording medium 1507 are reproduced by the stereoscopic image reproducing device, the main subject can be seen more stereoscopically. Further, for pixels in a region other than the main subject, image data may be created by selecting the imaging devices 1502 and 1501 for the left eye and the imaging devices 1503 and 1504 for the right eye. By selecting the imaging device in this way, the contrast of the parallax information can be given to the main subject and the region that is not so, and a stereoscopic image can be created more effectively.

(Third embodiment)
Hereinafter, a third embodiment will be described. The configuration of the imaging system of the third embodiment shown in FIG. 17 is almost the same as the configuration of the imaging system of FIG. 1 showing the first embodiment, except that it has a distance measuring system 111. The contents described in the first embodiment with reference to FIGS. 2 to 9 are the same in the third embodiment.

  FIG. 17 is a diagram illustrating a configuration of an imaging system according to the third embodiment. In FIG. 17, a ranging system 111 is a system that detects a distance to a subject, and detects a distance for all subject information in the viewfinder. The detected distance information is signal-processed as a distance map, and is stored as a distance map. As a distance measurement method, a dedicated external measurement AF module may be used. The image sensor 101 used in the present embodiment has a structure in which the photoelectric conversion unit of each pixel is divided into a plurality of parts, and can obtain a signal having parallax at each coordinate. The distance information may be obtained by triangulation using this parallax information. The distance map will be described in detail later.

  Further, as described above, the photoelectric conversion unit of each pixel constituting the imaging device is divided into a plurality of parts, and the divided photoelectric conversion unit is configured to receive different areas of the pupil of the photographing lens through the microlens. ing. The focus state of the imaging optical system can be detected (focus detection) from the amount of image shift between the left light receiving unit output and the right light receiving unit output under each microlens.

  By the way, the parallax becomes zero in the in-focus position, and occurs in the direction in which the image jumps out at the front pin and in the direction in which the image retreats at the rear pin. This phenomenon is emphasized when the outer photoelectric conversion units (PDs 301 and 304 in FIG. 3) are used as compared with the case where the inner photoelectric conversion units (PDs 302 and 303 in FIG. 3) are used.

  FIG. 18 is a flowchart illustrating the flow of obtaining a stereoscopic image performed by the imaging system of the present embodiment. First, photographing is performed using the imaging apparatus of FIG. 17 (step S1801). The operation of the operation unit 107 activates the imaging apparatus, performs photometry / ranging processing on a desired subject, and presses the shutter to shoot.

  Subsequently, the charges accumulated in the photoelectric conversion unit (PD) of the image sensor 101 are read according to the above-described reading method (step S1802). Here, the charges of all four PDs of each pixel are read out. The read signal is stored in the memory circuit 104.

  A subject distance is calculated from the read signal information, and a distance map is created (step S1803). There are various methods for calculating the distance to the subject, but the image sensor 101 according to the present embodiment has a feature that a signal having parallax can be obtained at each coordinate, and triangulation method using parallax is used. Is preferred. It is preferable to obtain the distance by a known technique such as performing correlation calculation using the signals of the outermost PD 301 and PD 304 among the charge signals of the four PDs of each pixel read out in step S1802. The distance information of each pixel calculated in this way is stored in the memory circuit 104 as a distance map. As the distance information of each pixel, the calculated distance may be stored as it is, or may be stored after being quantized. For example, the distance from the nearest to infinity is divided into four, quantized as 0 for the distance to d1, 1 for d1 to d2, 2 for d2 to d3, 2 for d3 to infinity, and so on. If stored, the memory capacity can be reduced. Here, the distance dn is an arbitrary value and has a relationship of d1 <d2 <d3 (n = 1, 2, 3,..., K). Alternatively, all the subjects in the finder may be extracted by image processing, and the subjects at similar distances may be grouped and quantized.

  FIG. 19 is an example of a captured image. The distance information of each pixel is calculated from the charge signal of each pixel read out in step S1802. A distance map is created based on the distance calculation result. For example, if the subject 1901 is closest to the distance da and 0 ≦ da <d1, the pixel distance information corresponding to the subject 1901 is stored as zero. A subject 1902 that is farther than the subject 1901 and closer to the background is a distance db. If d1 ≦ db <d2, the distance information of the pixel corresponding to the subject 1902 is stored as 1. If the distance of the subject 1903 (background) is d3 to infinity, the distance information of the corresponding pixel stores 3.

  As shown by the dotted line in FIG. 19, the image is divided into blocks, and distance information is acquired in each block. In this embodiment, although divided into a total of 28 blocks of 7 divisions in the horizontal direction and 4 divisions in the vertical direction, this is not restrictive. Further finely, distance information may be provided for each unit pixel. FIG. 20 shows a distance map obtained by calculating the distance to the subject for each block. The block corresponding to the subject 1901 in FIG. 19 stores distance information 0, the block corresponding to the subject 1902 stores distance information 1, and the block corresponding to the subject 1903 stores distance information 3.

  Here, depending on the block, there are cases where subjects located at different distances are mixed in the same block. In that case, when a PD to be described later is selected, an area in which a PD different from the PD to be actually selected is generated. That is, it is possible that different PDs are selected depending on the region even for the same subject. Therefore, it is desirable to perform subject extraction and store the same distance information for subjects at the same distance. For example, it is preferable to perform subject extraction based on the image information as shown in FIG. 19, expand the distance map for each block in FIG. 20, obtain the distance for each unit pixel, and create a distance map for each unit pixel. It is. For example, the pixel at the coordinates (Xk, Yk) in FIG. 19 is in the subject 1902 and the distance information should be originally stored as 1. However, the distance map for each block in FIG. . Such an inconvenience can be solved by extracting the subject by storing the same distance information, that is, 1 for the same subject. Processing as described above is performed, and distance information of the captured image is stored in the memory circuit 104 as a distance map (step S1804).

  From step S1805, left eye image and right eye image creation processing is performed. In order to create image data, pixel data is sequentially read out. The processing from step S1805 to step S1809 is repeated until the end of all pixels. The pixel data is read by scanning, for example, starting from the upper left of the image, reading in the horizontal direction, and reading the next row when one row is completed.

  A PD to be selected is determined for the read pixel (X, Y) (step S1806). The distance information corresponding to the pixel (X, Y) is read from the distance map created in step S1804, and the PD pair is selected according to the value. For example, when the distance information is 0, it is set as image data using a signal from the outside PD. Specifically, referring to FIG. 9, the data of the PD 304 is read for the left eye image, and the data of the PD 301 is read for the right eye image. If the distance information is 1, image data is obtained using a signal from the inner PD. That is, the data of the PD 303 is read for the left eye image, and the data of the PD 302 is read for the right eye image. When the distance information is 2, the signal from the inner PD is used, but the left and right PDs are switched and read. That is, the PD 302 data is read for the left-eye image, and the PD 303 data is read for the right-eye image. When the distance information is 3, the left and right PDs are switched and the signal from the outer PD is read. That is, the PD 301 data is read for the left eye image, and the PD 304 data is read for the right eye image.

  The PD data selected according to the distance of the subject is written as the pixel data (step S1807). In addition, since it consists of four photoelectric conversion parts per pixel, the signal output obtained here becomes a substantially 1/4 optical signal. Considering the division of the photoelectric conversion unit, the image processing apparatus 105 performs gain correction on the obtained signal output. Steps S1805 to S1807 are repeated until all pixels are completed (step S1808).

  As described above, a series of processing from image capturing to image data creation in the imaging system of the present embodiment is completed. The written left-eye image data and right-eye image data are developed by the image processing apparatus 105 and stored in the recording medium 109. FIG. 21 shows an example of the stored left-eye image 2101 and right-eye image 2102. Here, the subject 1901 at the closest position corresponds to the subject 2103 in the left-eye image 2101 and the subject 2104 in the right-eye image 2102. Similarly, the subject 1902 corresponds to the subject 2105 and the subject 2106, and the subject (background) 1903 corresponds to the subject 2107 and the subject 2108. The subject 2103 of the left-eye image 2101 is composed of signals read from the PD 304, the subject 2105 is composed of signals read from the PD 303, and the subject 2107 is composed of signals read from the PD 301. On the other hand, the subject 2104 of the right-eye image 2102 is composed of signals read from the PD 301, the subject 2106 is composed of signals read from the PD 302, and the subject 2108 is composed of signals read from the PD 304. If the left-eye image 2101 and the right-eye image 2102 obtained in this way and saved by the recording medium 109 are played back by the stereoscopic image playback device, the subject 1901 appears closer and the subject 1903 appears farther away. 3D images can be seen.

  In the present embodiment, the charge data accumulated in the image sensor 101 is read for all PDs. However, if a distance map is separately acquired using a dedicated external measurement AF module in advance, when reading out charges from the image sensor 101, a PD to be used is selected according to the distance of the subject and the necessary charges are obtained. May be read out. The configuration and operation in that case are the same as those in the first embodiment described with reference to FIGS. 13 and 14.

  In this embodiment, the number of PDs per pixel is four, but the number of divisions may be more than that. In particular, the larger the number of divisions in the horizontal direction, the greater the difference in parallax. In that case, it is preferable to keep the distance information of the distance map finely according to the number of divisions. That is, in this embodiment, the distance information held by the distance map has been described with four types of 0, 1, 2, and 3, but may be increased beyond that. Alternatively, a combination of PDs to be used depending on the distance may be selected according to the photographer's preference.

(Fourth embodiment)
Since the configuration of the imaging system of the fourth embodiment is the same as the configuration of the imaging system of the second embodiment shown in FIG. 15, the description of the configuration is omitted, and only the operation will be described.

  FIG. 22 is a flowchart for describing a flow of stereoscopic image acquisition performed by the imaging system according to the fourth embodiment of the present invention. First, photographing is performed (step S2201). Here, although not shown, the control circuit controls the imaging devices 1501 to 1504 to perform imaging so as to perform imaging.

  Subsequently, the electric charge accumulated in the image sensor of each imaging device is read and stored in the memory circuit 1506 (step S2202). By reading the charge of each imaging device, four pieces of charge data having different parallax information for each pixel are stored in the memory circuit 1506.

  A subject distance is calculated from the read signal information, and a distance map is created (steps S2203 and S2204). The subject distance information acquisition method can be performed in the same manner as in the third embodiment. The created distance map is stored in the memory circuit 1506.

  From step S2205, left eye image and right eye image creation processing is performed. In order to create image data, the pixel data stored in the memory circuit 1506 is sequentially read out. The processing from step S2205 to step S2208 is repeated until the end of all pixels. The pixel data is read by scanning, for example, starting from the upper left of the image, reading in the horizontal direction, and reading the next row when one row is completed.

  A PD to be selected is determined for the read pixel (X, Y) (step S2206). The distance information corresponding to the pixel (X, Y) is read from the distance map created in step S2204, and the camera pair that uses the data is selected according to the value. For example, when the distance information is 0, it is set as image data using a signal from the outer camera (imaging device). Specifically, the data of the imaging device 1504 is read for the left eye image, and the data of the imaging device 1501 is read for the right eye image. If the distance information is 1, image data is obtained using a signal from the inner camera (imaging device). That is, the data of the imaging device 1503 is read for the left eye image, and the data of the imaging device 1502 is read for the right eye image. When the distance information is 2, the signal from the inner imaging device is used, but the left and right imaging devices are switched and read. That is, the data of the imaging device 1502 is read for the left eye image, and the data of the imaging device 1503 is read for the right eye image. When the distance information is 3, the left and right imaging devices are switched and the signal from the outer imaging device is read. That is, the data of the imaging device 1501 is read for the left eye image, and the data of the imaging device 1504 is read for the right eye image.

  The PD data selected according to the distance of the subject is written as the pixel data (step S2207). Steps S2205 to S2207 are repeated until the end of all pixels (step S2208).

  As described above, a series of processing from image capturing to image data creation in the imaging system of the present embodiment is completed. The written left-eye image data and right-eye image data are developed by the image processing device 1505 and stored in the recording medium 1507. When the left-eye image and the right-eye image obtained in this way and saved by the recording medium 1507 are reproduced by a stereoscopic image reproducing device, an effective stereoscopic object is closer to a closer subject and a farther subject is closer. You can see the image.

(Fifth embodiment)
Since the imaging device of the fifth embodiment has the same configuration as that of the third embodiment, description thereof is omitted. In the third embodiment, the PD to be selected is changed according to the distance of the subject, but in the present embodiment, the PD to be selected is changed depending on whether the subject is in focus. If the AF function of the imaging device is used at the time of shooting, it can be determined which subject is in focus. In addition, the photographer may input a focused subject. Information on whether or not the subject is in focus is given to the distance map. If the subject is in focus, the inner PD is selected. This is because when the image is in focus, the same signal is input to all the divided pixels. That is, no parallax occurs. However, since the outer PD where light enters more obliquely receives less light, when the charge from the outer PD is read, it is darker than when the image is acquired by reading the charge from the inner PD. Become an image. Therefore, it is preferable to select the inner PD as the subject in focus.

  For other subjects that are not in focus, the PD may be selected in the same manner as in the third embodiment. However, based on the distance of the subject in focus, it is preferable to select the outer PD for subjects closer to it, and to select the PD by switching the left and right for subjects farther than the subject in focus. For example, a case where a scene as shown in FIG. 19 is photographed will be described. When the subject 1901 is focused, for the subject 1901, the data of the PD 303 is read out for the image for the left eye, and the data of the PD 302 is read out for the image for the right eye. A subject 1902 farther than the subject 1901 selects the inner PD by switching left and right, and reads PD302 data for the left-eye image and PD303 data for the right-eye image. The farther subject 1903 reads the PD 301 data for the left eye image and the PD 304 data for the right eye image. When the subject 1902 is focused, PD 303 data is read out for the left eye image and PD 302 data is read out for the right eye image. For a subject 1901 that is closer than the subject 1902, PD304 data is read out for the left-eye image and PD301 data is read out for the right-eye image. For a subject 1903 at a long distance, the PD 301 data is read out as the left eye image and the PD 304 data is read out as the right eye image.

  Thus, by selecting the PD in consideration of the focus information of the subject, a brighter image can be obtained and an effective stereoscopic image can be obtained. Also, when the aperture is stopped at the time of shooting, the outside PD does not get much light. Therefore, when photographing with a narrow aperture, a brighter image can be obtained without using the outermost PD. In this way, the PD to be used may be changed according to the aperture value at the time of shooting. Alternatively, when the outermost PD is used, correction may be performed by applying a gain according to the aperture value at that time.

Claims (13)

An imaging system including an imaging device that captures an image of a subject formed by a photographing lens,
Subject selection means for selecting a main subject;
An image sensor having a plurality of photoelectric conversion units per unit pixel;
Photoelectric conversion unit selection means for selecting the photoelectric conversion unit of each of the unit pixels according to whether or not it is the main subject;
Generating means for generating two image signals having parallax based on a signal from the selected photoelectric conversion unit;
Have
The plurality of photoelectric conversion units are arranged in a line in a direction in which parallax is applied, and the photoelectric conversion unit selection unit applies parallax in a first direction for pixels corresponding to the main subject, and The combination of the photoelectric conversion units is selected so that the parallax is maximized, and pixels other than the pixel corresponding to the main subject are set to have parallax in the first direction and correspond to the main subject. An imaging system, wherein a combination of the photoelectric conversion units is selected so as to be inside a combination of the photoelectric conversion units selected for a pixel. An imaging system including an imaging device that captures an image of a subject formed by a photographing lens,
Subject selection means for selecting a main subject;
An image sensor having a plurality of photoelectric conversion units per unit pixel;
Photoelectric conversion unit selection means for selecting the photoelectric conversion unit of each of the unit pixels according to whether or not it is the main subject;
Generating means for generating two image signals having parallax based on a signal from the selected photoelectric conversion unit;
Have
The plurality of photoelectric conversion units are arranged in a line in a direction in which parallax is applied, and the photoelectric conversion unit selection unit performs the photoelectric conversion so that parallax is applied in a first direction for pixels corresponding to the main subject. Selecting the combination of the photoelectric conversion units so that the pixels other than the pixel corresponding to the main subject have a parallax in the second direction opposite to the first direction. An imaging system characterized by the above.   3. The imaging system according to claim 1, wherein each unit pixel of the imaging element includes two or more photoelectric conversion units for the same microlens.   4. The gain correction is performed on an output signal of the photoelectric conversion unit according to the number of the photoelectric conversion units that read out a charge signal per unit pixel. 5. Imaging system. An imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens,
Subject selection means for selecting a main subject;
An imaging device selection means for selecting two imaging devices from the plurality of imaging devices according to whether or not the subject is the main subject;
Generating means for generating two image signals having parallax based on signals from the two selected imaging devices;
Have
The plurality of imaging devices are arranged in a line in a direction in which parallax is applied, and the imaging device selection unit is configured to apply parallax in a first direction for a region corresponding to the main subject, and the parallax is The combination of the imaging devices is selected so as to be the largest, and the region other than the region corresponding to the main subject is selected so as to add parallax in the first direction and the region corresponding to the main subject. An imaging system, wherein the combination of the imaging devices is selected so as to be inside the combination of the imaging devices . An imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens,
Subject selection means for selecting a main subject;
An imaging device selection means for selecting two imaging devices from the plurality of imaging devices according to whether or not the subject is the main subject;
Generating means for generating two image signals having parallax based on signals from the two selected imaging devices;
Have
The plurality of imaging devices are arranged in a line in a direction in which parallax is applied, and the imaging device selection unit is configured to combine the imaging devices so as to add parallax in a first direction for an area corresponding to the main subject. And a combination of the imaging devices is selected so that parallax is applied in a second direction opposite to the first direction for a region other than the region corresponding to the main subject. Imaging system. An imaging system including an imaging device that captures an image of a subject formed by a photographing lens,
Detection means for detecting the distance to the subject;
An image sensor having two or more photoelectric conversion units per unit pixel;
Photoelectric conversion unit selection means for selecting the photoelectric conversion unit of each unit pixel according to the distance to the subject;
Generating means for generating two image signals having parallax based on a signal from the selected photoelectric conversion unit;
Have
The photoelectric conversion units are arranged in a line in a direction in which parallax is applied, and the photoelectric conversion unit selection unit selects a first pixel corresponding to a subject whose distance detected by the detection unit is less than a predetermined distance. The pixel corresponding to the subject whose distance detected by the detection means is equal to or larger than the predetermined distance is selected so that the parallax is given in the direction of the image and the parallax is maximized. Of the photoelectric conversion unit so as to add parallax in the first direction and to be inside the combination of the photoelectric conversion units selected for the pixels corresponding to the subject that is less than the predetermined distance. An imaging system characterized by selecting a combination. An imaging system including an imaging device that captures an image of a subject formed by a photographing lens,
Detection means for detecting the distance to the subject;
An image sensor having two or more photoelectric conversion units per unit pixel;
Photoelectric conversion unit selection means for selecting the photoelectric conversion unit of each unit pixel according to the distance to the subject;
Generating means for generating two image signals having parallax based on a signal from the selected photoelectric conversion unit;
Have
The photoelectric conversion units are arranged in a line in a direction in which parallax is applied, and the photoelectric conversion unit selection unit selects a first pixel corresponding to a subject whose distance detected by the detection unit is less than a predetermined distance. For the pixel corresponding to the subject whose distance detected by the detection means is equal to or greater than the predetermined distance, the combination of the photoelectric conversion units is selected so as to give a parallax in the direction of the direction opposite to the first direction. An imaging system, wherein a combination of the photoelectric conversion units is selected so as to give parallax in the second direction.   9. The imaging system according to claim 7, wherein each unit pixel of the imaging element has two or more photoelectric conversion units for the same microlens.   10. The imaging system according to claim 7, wherein distance information that is information on a distance to the subject is stored for each block obtained by dividing the screen of the imaging element into a plurality of regions. .   The apparatus further includes focus detection means for detecting whether or not the subject is in focus, and the photoelectric conversion units are arranged in a line in a parallax direction, and the photoelectric conversion unit selection means is the focus detection means. 11. The pixel corresponding to the subject determined to be in focus by selecting a combination of the photoelectric conversion units located inside of the photoelectric conversion units. The imaging system according to claim 1. An imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens,
Detection means for detecting the distance to the subject;
An imaging device selection means for selecting two imaging devices from the plurality of imaging devices according to a distance to the subject;
Generating means for generating two image signals having parallax based on signals from the two selected imaging devices;
Have
The plurality of imaging devices are arranged in a line in a parallax direction, and the imaging device selection unit is configured to select a first region corresponding to a subject whose distance detected by the detection unit is less than a predetermined distance. For a region corresponding to a subject whose distance detected by the detection means is equal to or greater than the predetermined distance, the combination of the imaging devices is selected so as to add parallax in the direction of The combination of the imaging devices is selected so as to add parallax in the first direction and to be inside the combination of the imaging devices selected for the area corresponding to the subject that is less than the predetermined distance. An imaging system characterized by that. An imaging system including a plurality of imaging devices that capture an image of a subject formed by a photographing lens,
Detection means for detecting the distance to the subject;
An imaging device selection means for selecting two imaging devices from the plurality of imaging devices according to a distance to the subject;
Generating means for generating two image signals having parallax based on signals from the two selected imaging devices;
Have
The plurality of imaging devices are arranged in a line in a parallax direction, and the imaging device selection unit is configured to select a first region corresponding to a subject whose distance detected by the detection unit is less than a predetermined distance. The combination of the imaging devices is selected so as to give a parallax in the direction of and the region corresponding to the subject whose distance detected by the detection means is equal to or greater than the predetermined distance is opposite to the first direction. An imaging system, wherein a combination of the imaging devices is selected so as to add parallax in a second direction.