JP2015177258A - image input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a high-definition and high-dynamic-range image by using a compound-eye camera.SOLUTION: An array camera 411 comprises an array of plural cameras. A parallax detector 420 calculates the parallax of a subject by using images captured by the respective cameras on exposure conditions for detecting the parallax. An image synthesizer 430 generates a sheet of high-dynamic-range image by synthesizing the images captured by the respective cameras for an exposure time for obtaining a high-dynamic-range image, by deviating the images by the parallax calculated by the parallax detector 420.

Description

  The present invention relates to an image input device that generates a high dynamic range image.

  Cameras used in surveillance applications, industrial applications, in-vehicle applications, and the like are required to have a wide gradation characteristic (high dynamic range) that can be distinguished from low-illuminance to high-luminance subjects. There is a compound eye camera as one means for realizing this.

  For example, in Patent Document 1, each single-lens camera constituting a compound-eye camera is equipped with an optical filter having different transmittance, and each single-eye camera simultaneously captures images with different imaging conditions. A technique for obtaining a range image is disclosed.

JP 2002-281361 A

  By the way, in a compound eye camera, since each single-eye camera is arrange | positioned spatially, the position of the to-be-photographed object in the image imaged by each single-eye camera is image | photographed by shifting according to the distance. Therefore, if the images obtained by the single-eye cameras are synthesized as they are, there is a problem that a high-definition high dynamic range image cannot be obtained due to this shift.

  However, the compound eye camera described in Patent Document 1 does not take into account image shifts obtained with a single-eye camera.

  Here, it is also conceivable to obtain the parallax between the images by searching for corresponding points and to correct the shift between the images. However, in Patent Document 1, when the parallax of an object is obtained from an image obtained from a single-eye camera, the images captured by the single-eye camera are respectively subjected to different exposure conditions using color filters having different transmittances. Since it is a captured image, an accurate parallax cannot be obtained.

  That is, in order to obtain the parallax, it is necessary to obtain the same part (corresponding point) of the same subject between images, but it is difficult to search for the corresponding point because the brightness of the subject differs under different exposure conditions. . It is impossible to find a corresponding point especially when the subject is saturated or is captured in black.

  An object of the present invention is to provide an image input device that generates a high-definition high dynamic range image using a compound eye camera.

  An image input apparatus according to an aspect of the present invention includes an array camera unit in which a plurality of cameras are integrally formed, an exposure control unit that can individually set exposure conditions of each camera, and each camera by the exposure control unit. An image synthesis unit that is set as an exposure condition for obtaining a high dynamic range image and synthesizes an image captured by each camera into a single image.

  In this case, the exposure condition of each camera is set to the exposure condition for obtaining a high dynamic range image, and the subject is imaged. Therefore, a high-definition high dynamic range image is obtained.

  In the above aspect, the exposure control unit further includes a parallax detection unit that is set to an exposure condition for each camera to detect parallax, and detects a parallax of a subject using an image captured by each camera, The image synthesizing unit may align the subject using the detected parallax, and synthesize the images captured by the cameras into a single image.

  In this case, the exposure condition of each camera is set to the exposure condition for detecting parallax, the subject is imaged, and the parallax of the subject is detected from the image captured by each camera. The images captured by the cameras are combined with the subject aligned. Therefore, a high-definition high dynamic range image can be obtained.

  In the above aspect, an array lens in which a plurality of lenses are integrally arranged to face each camera may be further provided.

  In this case, since an array lens in which the lenses are integrally arranged is used, it is easy to accurately align the lens with the camera when creating the imaging unit.

  In the above aspect, the array camera may be configured such that a plurality of light receiving units corresponding to the plurality of cameras are integrally formed on the same substrate.

  In this case, an array camera can be configured using an image sensor made of one substrate, and the array camera can be created at low cost.

  In the above aspect, the exposure control unit may set the exposure time of each camera independently.

  In this case, the exposure condition of each camera can be changed by changing the exposure time of each camera.

  In the above aspect, the exposure control unit may set the exposure conditions of the cameras to different exposure conditions in order to obtain the high dynamic range image.

  In this case, a high dynamic range image is obtained by synthesizing images captured by cameras set with different exposure conditions.

  In the above aspect, the exposure control unit may set the exposure conditions of the cameras so that the exposure times of the cameras are the same in order to detect the parallax.

  In this case, the parallax can be accurately detected because the parallax is detected using images captured by a plurality of cameras having the same exposure time.

  In the above aspect, the parallax detection unit detects the parallax using a corresponding point search method, and exposure is set to detect the parallax using a corresponding point search result by the parallax detection unit. Further comprising a reliability estimation unit for estimating the reliability of the condition, the exposure control unit causes each camera to change the exposure condition until the reliability exceeds a specified value, the parallax detection unit, The parallax may be detected using an image captured under an exposure condition in which the reliability exceeds a specified value.

  In this case, since the parallax is detected using an image captured under an exposure condition whose reliability exceeds a specified value, the parallax can be detected with high accuracy.

  In the above aspect, the parallax detection unit detects the parallax using a corresponding point search method, and the exposure control unit switches a plurality of exposure conditions to detect each parallax in order to detect the parallax. The parallax detection further includes a reliability estimation unit that images the subject a plurality of times, and uses the search result of the corresponding points by the parallax detection unit to estimate the reliability of the exposure condition set to detect the parallax. The unit may detect the parallax using an image captured under an exposure condition that maximizes the reliability.

  In this case, since the subject is imaged under a plurality of predetermined exposure conditions and the parallax is detected using an image captured under the exposure condition with the maximum reliability, the parallax can be detected with high accuracy.

  Further, the camera further includes an exposure evaluation value calculation unit that calculates an exposure evaluation value of each camera from an image captured by each camera, and the exposure control unit captures an image with each camera when the reliability is equal to or less than a specified value. The exposure evaluation value of the obtained image is calculated by the exposure evaluation value calculation unit, and for a camera that captures an image in which the calculated exposure evaluation value is a predetermined value or more away from a predetermined appropriate range, the exposure condition is set as follows. You may change to optimal exposure conditions.

  In this case, the exposure condition is set to the optimum exposure condition for the camera that cannot capture the subject well, and the subject is imaged again by each camera, so that the parallax can be detected with high accuracy.

  In the above aspect, the image composition unit may identify a predetermined number of subjects from the images captured by the cameras, and synthesize the images captured by the cameras so that the specified subjects are aligned.

  In this case, since a predetermined number of main subjects are aligned, it is possible to prevent the subjects from being blurred.

  According to the present invention, a high-definition high dynamic range image can be obtained.

It is the figure which showed an example of the structure of the array lens and imaging part which comprise the image input device by one embodiment of this invention. It is the figure which showed another example of the structure of the array lens and imaging part which comprise the image input device by one embodiment of this invention. It is the figure which showed another example of the structure of the array lens and imaging part which comprise the image input device by one embodiment of this invention. It is a block diagram which shows an example of a structure of the image input device by one embodiment of this invention. It is a block diagram which shows an example of a structure of an imaging part when the imaging part shown in FIG. 2 is employ | adopted. It is a block diagram which shows an example of a structure of an imaging part when the imaging part shown in FIG. 1 is employ | adopted. FIG. 3 is a diagram illustrating a distribution of light receiving blocks in an array camera unit when the imaging unit of FIG. 2 is employed. It is a figure which shows the other example of the light reception block formed in an array camera part when the imaging part of FIG. 3 is employ | adopted. It is a block diagram which shows an example of a structure of the image pick-up element at the time of employ | adopting the image pick-up part shown in FIG. It is a sequence diagram which shows an example of the read-out sequence of the image pick-up element shown in FIG. It is a flowchart which shows an example of the process by the image input device of this Embodiment. It is a flowchart which shows an example of the parallax detection process of a 1st pattern. It is a flowchart which shows an example of the parallax detection process of a 2nd pattern. It is a figure which shows an example of the image imaged when detecting parallax. It is a figure which shows an example of the image imaged when obtaining a high dynamic range image.

  FIG. 1 is a diagram showing an example of the configuration of an array lens 11 and an imaging unit 12 constituting an image input device according to an embodiment of the present invention. The imaging unit 12 includes an imaging element 121 and an imaging substrate 122. The imaging elements 121 are arranged in an array of predetermined rows × predetermined columns on the imaging substrate 122. In the example of FIG. 1, the imaging elements 121 are arranged in 4 rows × 4 columns, but this is only an example, and a plurality of imaging elements may be included in the imaging unit 12.

  Each image sensor 121 includes one light receiving portion 121a. In the light receiving unit 121a, one photodiode corresponding to one pixel is arranged in an array of predetermined rows × predetermined columns. Therefore, in the example of FIG. 1, a configuration in which the light receiving unit 121a and the lens 111 are combined corresponds to one camera.

  The imaging substrate 122 is a support member such as silicon, and includes a control circuit that controls the imaging element 121 and the like.

  The array lens 11 includes a lens substrate 112 arranged in an array of predetermined rows × predetermined columns corresponding to the imaging elements 121. In the example of FIG. 1, since the image sensors 121 are arranged in 4 rows × 4 columns, the lens substrates 112 are arranged in 4 rows × 4 columns. However, this is only an example, and a plurality of lens substrates 112 are provided in accordance with the number of image sensors 121. Each lens substrate 112 includes one lens unit 111. That is, in the example of FIG. 1, the array lens 11 is configured by arranging single lens substrates 112 in an array.

  In the imaging unit 12 of FIG. 1, an optical image of a subject in front of the array lens 11 is individually formed on each light receiving unit 121 a through each lens unit 111. Then, an optical image individually formed on each light receiving unit 121a is read as an image picked up by one camera. Here, the optical image formed on each light receiving unit 121a captures the same subject, but since each lens unit 111 is spatially shifted, there is a shift in the imaging position of the subject between the optical images. Occur.

  FIG. 2 is a diagram illustrating another example of the configuration of the array lens 11 and the imaging unit 12 that constitute the image input device according to the embodiment of the present invention. In the example of FIG. 2, one imaging element 121 is arranged on the imaging substrate 122. In the imaging device 121, a plurality of light receiving portions 121b are arranged in an array in a predetermined row × predetermined column. In the light receiving unit 121b, one photodiode corresponding to one pixel is arranged in an array of predetermined rows × predetermined columns. That is, in FIG. 2, the plurality of light receiving portions 121b are defined by dividing the surface of one image sensor 121 into a plurality of regions. Therefore, in the example of FIG. 2, a configuration in which one light receiving unit 121b and one lens 111 are combined corresponds to one camera.

  In the example of FIG. 2, the array lens 11 is configured by a single lens substrate 112 in which a plurality of lens portions 111 are arranged in a predetermined row × predetermined array corresponding to each light receiving portion 121b. That is, in the example of FIG. 2, the array lens 11 includes a plurality of lens portions 111 integrally formed.

  In the imaging unit 12 of FIG. 2, an optical image of a subject in front of the array lens 11 is individually formed on each light receiving unit 121 b through each lens unit 111. Then, an optical image individually formed on each light receiving unit 121b is read as an image captured by one camera. Here, the optical image formed on each light receiving unit 121b captures the same subject, but each lens unit 111 is spatially displaced. A shift occurs in the image position.

  FIG. 3 is a diagram showing another example of the configuration of the array lens 11 and the imaging unit 12 constituting the image input device according to the embodiment of the present invention. 3, the configuration of the array lens 11 is the same as that of FIG. 2, but the configuration of the imaging unit 12 is different from that of FIG. 3 is the same as FIG. 2 in that one image pickup device 121 is arranged on the image pickup substrate 122. However, the imaging device 121 of FIG. 3 is different from FIG. 2 in that it includes a single light receiving unit 121c. That is, in the imaging unit 12 of FIG. 3, an optical image of a subject in front of the array lens 11 is individually formed on one light receiving unit 1 c through each lens unit 111. Then, each of a plurality of optical images individually formed on the light receiving unit 1c is read as an image captured by one camera. Also in FIG. 3, since each lens unit 111 is spatially displaced, as in FIGS. 1 and 2, a deviation occurs in the imaging position of the subject between the optical images. FIG. 4 is a block diagram showing an example of the configuration of the image input apparatus according to the embodiment of the present invention. The image input device includes an imaging unit 12, a parallax detection unit 420, an image composition unit 430, an exposure evaluation value calculation unit 440, a reliability estimation unit 450, and an exposure control unit 460.

  The imaging unit 12 includes an array camera unit 411 and an exposure time setting unit 412. The array camera unit 411 includes a plurality of cameras arranged in an array. When the imaging unit 12 of FIG. 1 is employed, one light receiving unit 121a and one lens 111 included in each of the plurality of imaging elements 121 arranged on the imaging substrate 122 are combined into one camera of the array camera unit 411. It corresponds to. When the imaging unit 12 of FIG. 2 is employed, one light receiving unit 121b formed on one imaging element 121 disposed on the imaging substrate 122 and one lens 11 facing the light receiving unit 121b are arranged in an array. This corresponds to one camera of the camera unit 411. When the imaging unit 12 of FIG. 3 is employed, one region on the light receiving unit 121c where an optical image of one lens unit 111 is formed and the lens 111 facing the region are one of the array camera unit 411. Equivalent to two cameras.

  The exposure time setting unit 412 includes, for example, a circuit module mounted on the imaging substrate 122. The exposure time setting unit 412 receives an instruction regarding the setting of the exposure time for each camera from the exposure control unit 460, and instructs the exposure time (charge accumulation time). ) To expose each camera and accumulate charges.

  The parallax detection unit 420 calculates the parallax of the subject using an image captured by each camera under an exposure condition for detecting the parallax. Here, the parallax detection unit 420 uses, for example, an image captured by one of a plurality of cameras as a reference image, and an image captured by other remaining cameras as a reference image. Then, the parallax detection unit 420 may search the reference image for points corresponding to the respective points of the standard image, and calculate the parallax of the subject using the obtained corresponding points. Here, as a corresponding point search method, for example, SAD (Sum of Absolute Differences) or POC (Phase-Only Correlation) can be employed.

  If there are a plurality of reference images, the parallax detection unit 420 may calculate the parallax for the base image and each reference image. For example, if there are three reference images, the parallax detection unit 420 may calculate the parallax by individually applying the corresponding point search process for one reference image to the three reference images. Note that the parallax detection unit 420 may set all points of the reference image as search targets for corresponding points, or set points arranged at certain intervals in the reference image as search targets for corresponding points. May be. In the latter case, the processing cost can be reduced.

  The image combining unit 430 combines images captured by the cameras under the exposure conditions for obtaining a high dynamic range image by shifting the parallax calculated by the parallax detection unit 420 for each subject and combining the images. Generate an image.

  The exposure evaluation value calculation unit 440 calculates an exposure evaluation value of an image captured by each camera. Here, as the exposure evaluation value, for example, the average luminance of a partial region (for example, a rectangular region located at the center of the image) can be employed in each of the images captured by each camera. Alternatively, an image obtained by dividing an image into a plurality of regions and weighting and adding the average luminance values of the divided regions may be adopted as the exposure evaluation value.

  The exposure control unit 460 outputs an instruction for setting the exposure time of each camera to the exposure time setting unit 412, and sets the exposure time of each camera individually. Here, when detecting the parallax, for example, the exposure control unit 460 sets the exposure time of each camera to be the same, and causes each camera to image the subject. Further, when obtaining a high dynamic range image, for example, the exposure time setting unit 412 sets the exposure time of each camera to a different value and causes each camera to image the subject.

  Further, when performing optimal exposure control, the exposure control unit 460 calculates the optimal exposure time of each camera using the exposure evaluation value of each camera calculated by the exposure evaluation value calculation unit 440, and sets it to each camera. In this case, for example, the exposure control unit 460 may set the optimum exposure time so that the luminance value of the image from the exposure control target camera falls within a predetermined appropriate range.

  The reliability estimation unit 450 estimates the reliability of the exposure condition set when the parallax is detected, using the corresponding point search result by the parallax detection unit 420. Here, the value of reliability increases as more corresponding points can be searched correctly in the corresponding point search process. The setting of exposure conditions using reliability will be described later.

  In FIG. 4, the parallax detection unit 420, the image synthesis unit 430, the exposure evaluation value calculation unit 440, the reliability estimation unit 450, and the exposure control unit 460 may be realized, for example, by a CPU executing a program. Alternatively, it may be configured by a dedicated hardware circuit such as an image processor.

  FIG. 5 is a block diagram illustrating an example of the configuration of the imaging unit 12 when the imaging unit 12 illustrated in FIG. 2 is employed. FIG. 5 illustrates an example of the imaging unit 12 in which the light receiving units 121b are arranged in 3 rows × 3 columns.

  The imaging unit 12 includes nine light receiving units 121b, one output unit 123, and one exposure time setting unit 412 arranged in 3 rows × 3 columns. The output unit 123 individually receives images from each of the nine light receiving units 121b. Then, the output unit 123 outputs the image input from each of the light receiving units 121b to the parallax detection unit 420, the image synthesis unit 430, and the exposure evaluation value calculation unit 440 as an image captured by one camera.

  The exposure time setting unit 412 writes a register value for setting the exposure time of each camera according to the register control signal output from the exposure control unit 460. Then, the exposure time setting unit 412 generates a drive signal for each camera according to the clock signal and synchronization signal output from the exposure control unit 460 and the register value, and individually sets the exposure time for each camera.

  FIG. 6 is a block diagram illustrating an example of the configuration of the imaging unit 12 when the imaging unit 12 illustrated in FIG. 1 is employed. FIG. 6 illustrates an example of the imaging unit 12 in which the light receiving units 121a are arranged in 3 rows × 3 columns. In the imaging unit 12 illustrated in FIG. 1, one imaging element 121 is assigned to each camera. Therefore, each image sensor 121 includes an output unit 123, a light receiving unit 121a, and an exposure time setting unit 412. In this case, one image output from one output unit 123 is output to the parallax detection unit 420, the image synthesis unit 430, and the exposure evaluation value calculation unit 440 as an image captured by one camera.

  FIG. 7 is a diagram illustrating the distribution of the light receiving blocks BL1 to BL16 in the array camera unit 411 when the imaging unit 12 of FIG. 2 is employed. In FIG. 7, since the light receiving portions 121 b are arranged in 4 rows × 4 columns, 16 light receiving blocks BL <b> 1 to BL <b> 16 arranged in 4 rows × 4 columns are arranged on the image sensor 121. Here, the exposure time setting unit 412 is individually connected to each light receiving unit 121b. Therefore, the exposure time setting unit 412 can set the exposure time for each light receiving unit 121b. That is, the exposure time setting unit 412 can set the exposure time for each of the light receiving blocks BL1 to BL16.

  In addition, when the imaging unit 12 of FIG. 1 is employed, as shown in FIG. 6, an imaging element 121 is provided corresponding to each light receiving unit 121a. Therefore, even when the imaging unit 12 of FIG. 1 is employed, the exposure time can be set for each of the light receiving blocks BL1 to BL16.

  FIG. 8 is a diagram illustrating another example of the light receiving blocks formed in the array camera unit 411 when the imaging unit 12 of FIG. 3 is employed. In the imaging unit 12 of FIG. 3, an optical image of the subject is arranged in 4 rows × 4 columns on the light receiving unit 121c by the lens unit 111 of 4 rows × 4 columns. Therefore, when the imaging unit 12 of FIG. 3 is employed, the light receiving unit 121c is partitioned into light receiving blocks BL1 to BL16 of 4 rows × 4 columns as shown in FIG.

  However, since the imaging unit 12 shown in FIG. 3 has one imaging element 121, there is also one V shift register (see FIG. 9) that vertically scans each pixel. Therefore, it is easy to set the exposure time for each row. Therefore, when the image pickup unit 12 shown in FIG. 3 is adopted, the image pickup device 121 has four light receiving blocks included in each row as one light receiving block group as shown in FIG. 8, and exposure is performed for each light receiving block group. Time is set. Specifically, the light receiving blocks BL1 to BL4 constitute one light receiving block group D1, the light receiving blocks BL5 to BL8 constitute one light receiving block group D2, and the light receiving blocks BL9 to BL12 constitute one light receiving block group D3. The light receiving blocks BL13 to BL16 constitute one light receiving block group D4.

  FIG. 9 is a block diagram illustrating an example of a detailed configuration of the imaging element 121 when the imaging unit 12 illustrated in FIG. 3 is employed. As shown in FIG. 9, the image sensor 121 includes a V shift register 901, a column A / D 902, an H shift register 903, and a pixel array unit 904.

  The pixel array unit 904 includes 24 pixels in the horizontal direction and 16 pixels in the vertical direction, for a total of 24 × 16 pixels arranged in an array. In this example, the pixel array unit 904 is divided into four light receiving blocks with block numbers BH1 to BH4 in the horizontal direction and four light receiving blocks with block numbers BV1 to BV4 in the vertical direction, for a total of 16 light receiving blocks. It is partitioned into BL.

  The V shift register 901 sets the four light receiving blocks BL defined by the block number BV1 as one light receiving block group D1, and the four light receiving blocks BL defined by the block number BV2 as one light receiving block group D2. The four light receiving blocks BL specified by the block number BV3 are set in one light receiving block group D3, and the four light receiving blocks BL specified by the block number BV4 are set in one light receiving block group D4. The V shift register 901 sets an exposure time for each of the light receiving block groups D1 to D4. In addition, the V shift register 901 sequentially selects from the first row to the 16th row of the pixel array unit 904, and outputs a pixel signal from the pixels in the selected row.

  The column A / D 902 is connected to each column of the pixel array unit 904 via a signal line L1. Twenty-four signal lines L1 are provided corresponding to each column of the pixel array unit 904. The column A / D 902 reads a pixel signal from the pixel in the row selected by the V shift register 901, A / D converts this pixel signal into a digital pixel signal having a predetermined number of bits, and holds it in a register (not shown).

  The H shift register 903 horizontally scans the column A / D every time the A / D conversion of the pixel signal for one row by the column A / D 902 is completed. Accordingly, the column A / D 902 sequentially outputs A / D converted digital pixel signals.

  FIG. 10 is a sequence diagram illustrating an example of a reading sequence of the image sensor 121 illustrated in FIG. 9. In the example of FIG. 10, the exposure times of the light receiving block groups D1 to D4 are set to H, 2H, 4H, and 8H, respectively. The readout periods of the pixels belonging to the 1st to 16th rows are assigned to horizontal scanning periods H (t1) to H (t16), respectively. H represents one horizontal scanning period. Further, in FIG. 10, a band elongated in the horizontal direction indicates the exposure time of each row. In FIG. 10, numbers 1 to 16 attached in a vertical column are indexes that define the rows of the pixel array unit 904.

  Here, since the readout period of each row is fixed, the exposure start timing of each row is determined based on the readout period. For example, in the first row belonging to the light receiving block group D1, since the readout period is assigned to the horizontal scanning period H (t1), the start timing of the horizontal scanning period H (t0) is set to set the exposure time to H. The exposure start timing is set.

  In the fifth row belonging to the light receiving block group D2, since the readout period is assigned to the horizontal scanning period H (t5), the start timing of the horizontal scanning period H (t3) is set to set the exposure time to 2H. The exposure start timing is set.

  As described above, in the sequence shown in FIG. 10, the pixel signals in the first to fourth rows in which the exposure time is set to H are sequentially applied to the column A / D 902 in each of the horizontal scanning periods H (t1) to H (t4). Is read out. Further, in each of the horizontal scanning periods H (t5) to H (t8), the pixel signals in the 5th to 8th rows in which the exposure time is set to 2H are sequentially read out to the column A / D902. Further, in each of the horizontal scanning periods H (t9) to H (t12), the pixel signals in the 9th to 12th rows with the exposure time set to 4H are sequentially read out to the column A / D 902. Further, in each of the horizontal scanning periods H (t13) to H (t16), the pixel signals on the 13th to 16th rows with the exposure time set to 8H are sequentially read out to the column A / D 902.

  One vertical scanning period V is defined by the horizontal scanning periods H (t1) to H (t16). Then, the vertical scanning period V is repeated, and four images captured by the light receiving block groups D1 to D4 are obtained for each vertical scanning period V.

  In FIG. 9, the number of rows and the number of columns in the pixel array unit 904 are examples, and values other than the number of rows and the number of columns shown in FIG. 9 may be employed. Further, in the pixel array unit 904, four rows are set as one light receiving block group, but this is an example, and one light receiving block group may be configured by a number of rows other than four rows. In FIG. 9, there are four light receiving block groups. However, this is an example, and light receiving block groups other than four may be set.

  FIG. 11 is a flowchart illustrating an example of processing performed by the image input apparatus according to the present embodiment. First, parallax detection processing is executed (S1101). Here, the parallax detection process is a process of capturing a subject in order to detect parallax and generating parallax information for each image from the obtained image, which will be described in detail later.

  When the parallax detection process ends, the exposure control unit 460 instructs the exposure time setting unit 412 to set the exposure time of each camera to the exposure time for obtaining a high dynamic range image (S1102). In this case, for example, the exposure control unit 460 obtains the optimum exposure time for the reference camera that captures the reference image to capture the main subject (for example, the subject located at the center of the image). Then, the exposure control unit 460 may set the exposure time of 100/2 (plus or minus n power)% (n is an integer of 1 or more) of the optimum exposure time as the exposure time of the remaining cameras other than the reference camera. .

  For example, if there are three cameras (CAM1 to CAM3) other than the reference camera, and the optimal exposure time of the reference camera is Tx, the exposure control unit 460 has an exposure time T_CAM1 = Tx × 2 of CAM1, and an exposure time T_CAM2 of CAM2. = Tx / 2, CAM3 exposure time T_CAM3 = Tx × 4, and the like.

  The method of setting the exposure time of each camera other than the reference camera at 100/2 (plus / minus n)% of the optimum exposure time is merely an example. For example, the exposure time of each camera other than the reference camera may be set at 100/10 (plus / minus n)% of the optimal exposure time. In short, the exposure time of each camera other than the reference camera may be set so as to vary up and down on the basis of the optimum exposure time of the reference camera. At least 3 to 4 types of exposure time are set for each camera in order to obtain a high dynamic range image. Therefore, if the number of cameras is five or more, the exposure control unit 460 may set different exposure times for all cameras, or cyclically set three to four types of exposure times for all cameras. May be.

  An ND (Neutral Density) filter may be attached to the array lens 11. In this case, the exposure time setting unit 412 may set the exposure time of each camera by adding a predetermined offset value to the exposure time so that the dimming by the ND filter is canceled.

  Next, each camera constituting the array camera unit 411 captures an image of the subject with the set exposure time (S1103).

  Next, the image synthesis unit 430 synthesizes images captured by the cameras (S1104). Here, when a plurality of subjects are included in an image captured by each camera, the image composition unit 430 identifies one representative subject, and shifts the image captured by each camera by the parallax of the identified subject. What is necessary is just to synthesize. As a typical one subject, for example, a subject imaged at the center of each camera can be adopted. Here, the image composition unit 430 uses the disparity information generated by the disparity detection unit 420 to extract a group of points with the same or constant disparity as one subject, and out of the extracted subjects The subject located in the center may be specified as one representative subject. Then, the image composition unit 430 may calculate the representative parallax value of the specified one subject as the parallax of the subject. Here, as the representative value of the parallax, for example, the average value of the parallaxes of the respective points constituting the subject or the parallax of the point located at the center of gravity of the subject can be adopted. The disparity information is information in which coordinates in the reference image of one or more points to be searched for corresponding points are associated with the disparity for each point.

  For example, when there are three reference images S1 to S3, the image composition unit 430 calculates three parallaxes k1 to k3 from each of the three parallax information K1 to K3 corresponding to the reference images S1 to S3 for one subject. To do. Then, the image synthesis unit 430 synthesizes the reference images R1 with the reference images S1 to S3 shifted by parallax k1 to k3, respectively. As a result, a high dynamic range image without deviation can be obtained for one subject.

  Alternatively, when a plurality of subjects are included in an image captured by each camera, the image composition unit 430 may synthesize the images captured by each camera by shifting the plurality of subjects. For example, the image compositing unit 430 specifies a group of points having the same or constant parallax as one subject in each of the parallax information K1 to K3. Then, the image composition unit 430 obtains the parallax of each subject for each of the parallax information K1 to K3. For example, if there are three subjects OB1 to OB3, the image composition unit 430 obtains the parallax k11 to k13 from the parallax information K1 to K3 for the subject OB1, and obtains the parallax information K1 to K3 for the subject OB2. The parallax k21 to k23 are obtained, and the parallax k31 to k33 are obtained from the parallax information K1 to K3 for the subject OB3.

  Then, the image synthesis unit 430 synthesizes the area corresponding to the subject OB1 of the reference images S1 to S3 with respect to the standard image R1 by shifting them by the parallax k11 to k13, respectively, to the subject OB2 of the reference images S1 to S3 The corresponding regions may be combined by shifting the parallax k21 to k23, respectively, and the regions corresponding to the subject OB3 of the reference images S1 to S3 may be combined by shifting the parallax k31 to k33, respectively.

  When correcting the parallax, the parallax may be obtained for each pixel, and the parallax may be corrected based on the parallax.

  Thereby, one high dynamic range image is generated. The generated high dynamic range image is displayed on a display device (not shown) or recorded in an image memory (not shown), for example.

  Next, the parallax detection process shown in S1101 will be described. In the present embodiment, two patterns are prepared as parallax detection processing. FIG. 12 is a flowchart illustrating an example of parallax detection processing of the first pattern. First, the exposure control unit 460 sets a default exposure condition common to each camera to each camera, and causes each camera to image a subject (S1301). Here, the optimum exposure time can be adopted as a default exposure condition common to each camera.

  Next, the parallax detection unit 420 detects parallax from an image captured by each camera (S1302), and generates parallax information.

  Next, the reliability estimation unit 450 estimates the reliability with respect to the exposure condition when the parallax is detected using the parallax information (S1303). For example, it is assumed that there are three reference images S1 to S3 and that there are three points on the reference image to be searched for corresponding points. Here, in order to simplify the explanation, the number of points on the reference image is three, but in reality, far more points are adopted as the search target points. In this case, if the corresponding point of the reference image S1 is correctly searched for only the point P1 among the three points P1 to P3, the score of the reference image S1 is, for example, one point. Further, assuming that the corresponding points can be correctly searched for only the points P1 and P2 for the reference image S2, the score for the reference image S2 is, for example, two points. Further, regarding the reference image S3, if the corresponding points can be correctly searched at all of the points P1 to P3, the score of the reference image S3 is, for example, three points. In this case, the reliability of the currently set exposure condition is 1 + 2 + 3 = 6 points. In this way, the reliability estimation unit 450 estimates the reliability of the exposure conditions.

  Next, if the estimated reliability is greater than the specified value (YES in S1304), the exposure control unit 460 determines the disparity information generated in S1302 as the disparity information to be output (S1305). Here, when the product of the total number of points to be searched for corresponding points and the total number of reference images is a perfect score, for example, values such as 60%, 70%, 80%, and 90% of the full score are given. Can be adopted.

  On the other hand, if the estimated reliability is equal to or less than the specified value (NO in S1304), exposure control unit 460 changes the exposure condition (S1306).

  Here, the exposure control unit 460 causes the exposure evaluation value calculation unit 440 to calculate the exposure evaluation value of each image captured by each camera, and captures an image in which the exposure evaluation value is a predetermined value or more away from a predetermined appropriate range. Identify the camera that did. Then, the exposure control unit 460 may set the exposure time for setting the exposure evaluation value within the appropriate range for the specified camera as the exposure condition of the specified camera. Further, the exposure control unit 460 may maintain the previously set exposure condition for a camera whose exposure evaluation value is not more than a certain value from the appropriate range.

  Then, the exposure control unit 460 causes each camera to capture an image of the subject again (S1307), and returns the process to S1302. In this case, the exposure conditions of each camera are set to two or more types, and parallax is detected.

  Note that the above-described method for changing the exposure conditions is merely an example. For example, the exposure control unit 460 may divide an image captured under the default exposure condition into a plurality of areas according to the luminance, and determine an optimal exposure condition for each of the divided areas. Here, for example, the exposure control unit 460 creates a histogram representing the relationship between the pixel value and the frequency from the reference image, and uses the group of pixels based on the peak of the histogram as one region, and the reference image as a plurality of regions. What is necessary is just to divide into. Then, the exposure control unit 460 may assign the determined optimal exposure condition for each region to each camera, cause the camera to capture the subject again, and cause the parallax detection unit 420 to detect the parallax.

  Here, for example, the exposure control unit 460 divides the reference image into a plurality of regions according to the luminance, and causes the exposure evaluation value calculation unit 440 to calculate an exposure evaluation value for each divided region. Then, the exposure control unit 460 determines optimum exposure conditions that bring the calculated exposure evaluation values within a predetermined appropriate range. For example, assuming that two regions RD1 and RD2 are extracted, an optimum exposure time T1 for setting the exposure evaluation value of the region RD1 within an appropriate range and an optimum exposure time T2 for setting the exposure evaluation value of the region RD2 within an appropriate range are determined. Is done.

  Then, the exposure control unit 460 may set the optimum exposure time T1 and the optimum exposure time T2 in a checkered pattern for each camera and change the exposure conditions.

  In addition, as a method for changing the exposure condition, the following method may be employed. For example, the exposure control unit 460 may increase or decrease the exposure time of each camera uniformly, and repeatedly perform imaging by each camera until the reliability becomes a specified value or more. In this case, the exposure control unit 460 causes the exposure evaluation value calculation unit 440 to calculate the exposure evaluation value of the image captured by each camera. If the average value of the exposure evaluation values is less than or equal to the appropriate range, for example, the exposure time If it is greater than the appropriate range, the exposure time may be decreased. Moreover, the exposure control part 460 should just increase / decrease exposure time in the direction in which reliability approaches a regulation value, if the average value of exposure evaluation value is in an appropriate range.

  Thus, in the parallax detection process of the first pattern, since the exposure conditions of each camera are set so that the reliability is greater than the specified value, the detection accuracy of the parallax is increased, and the subject shift between images is reduced. It can be detected accurately.

  Note that the exposure control unit 460 sets the default exposure conditions for each camera in S1301, but for example, during continuous imaging, the exposure control unit 460 sets each camera when the reliability exceeds a specified value in the previous parallax detection process. The exposure conditions that have been used may be employed.

  FIG. 13 is a flowchart illustrating an example of the second-pattern parallax detection process. The parallax detection process of the second pattern is characterized in that a plurality of exposure conditions prepared in advance are set in each camera and the exposure condition with the highest reliability is adopted.

  First, the exposure control unit 460 sets the first exposure condition for each camera and images the subject (S1401). S1402 and S1403 are the same as S1302 and S1303 in FIG. Next, if the imaging under all exposure conditions has been completed (YES in S1404), the exposure controller 460 outputs the parallax information obtained from the image captured under the exposure condition with the maximum reliability. Information is determined (S1405).

  On the other hand, if the imaging under all exposure conditions has not been completed (NO in S1404), the exposure control unit 460 sets the exposure condition of each camera to the next exposure condition (S1406), and images the subject in each camera. (S1407), and the process returns to S1402. As described above, in the parallax detection process of the second pattern, since the parallax is detected using the image captured under the exposure condition with the maximum reliability from the plurality of exposure conditions, it is possible to accurately obtain the deviation of the subject. Can do.

  In the examples of FIGS. 12 and 13, the exposure conditions are set in consideration of the reliability, but the present invention is not limited to this. For example, the parallax detection process may be performed by determining the optimal exposure time for each region according to the luminance value described in S1306 and assigning the determined optimal exposure time to each camera in a checkered pattern.

  FIG. 14 is a diagram illustrating an example of an image captured when parallax is detected. In the example of FIG. 14, since a subject is imaged by a camera of 4 rows × 3 columns, small rectangular images are arranged in 4 rows × 3 columns. Each camera images two stuffed animals placed in front of the window in a room with a window. In the example of FIG. 14, two types of exposure time, that is, a first exposure time and a second exposure time shorter than the first exposure time are employed as exposure conditions. Here, the optimal exposure time in the room is adopted as the first exposure time. As the second exposure time, the optimum exposure time outside the room is adopted.

  The first exposure time and the second exposure time are assigned to each camera in a checkered pattern. Therefore, in the example of FIG. 14, an image in which the subject appears bright and an image in which the subject appears dark are alternately arranged.

  The parallax detection unit 420 uses one of these images as a standard image and the rest as a reference image, and executes corresponding point search processing. And the parallax detection part 420 detects the parallax of the two stuffed animals which are main subjects about each reference image.

  Next, the exposure control unit 460 sets the exposure time of each camera in order to obtain a high dynamic range image. FIG. 15 is a diagram illustrating an example of an image captured when a high dynamic range image is obtained. In FIG. 15, the same subject as in FIG. 14 is captured by a 4 × 3 camera. Therefore, small rectangular images are arranged in 4 rows × 3 columns.

  The exposure time of each camera is set to a different time such as 1/10 times, 1/100 times, 10 times, 100 times,... This is an example, and the exposure time of each camera may be set to a power of 2 with respect to the reference exposure time. As the reference exposure time, for example, the exposure time of the camera that captures the reference image is employed. In the example of FIG. 15, the exposure time is set to be longer toward the upper row, and two types of exposure times are alternately assigned to each row. Therefore, the image in the upper row is brighter.

  The image composition unit 430 aligns the twelve images shown in FIG. 15 using the parallax detected by the parallax detection unit 420 and synthesizes the image into one image. Here, twelve images are aligned so that the two stuffed animals match. As a result, a high dynamic range image in which the two stuffed animals, which are the main subjects, are not displaced and the surrounding images are clearly displayed can be obtained.

  In the above description, the exposure time is mainly adopted as the exposure condition. However, this is only an example, and the aperture amount may be adopted as the exposure condition, or a combination of the exposure time and the aperture amount is used as the exposure condition. It may be adopted.

410 Imaging unit 411 Array camera unit 412 Exposure time setting unit 420 Parallax detection unit 430 Image composition unit 440 Exposure evaluation value calculation unit 450 Reliability estimation unit 460 Exposure control unit

Claims (11)

An array camera unit in which a plurality of cameras are integrally formed;
An exposure control unit capable of individually setting the exposure conditions of each camera;
An image input device comprising: an image composition unit configured to synthesize an image captured by each camera, which is set to an exposure condition for each camera to obtain a high dynamic range image by the exposure control unit. A parallax detection unit configured to detect parallax of a subject using an image captured by each camera, set to an exposure condition for each camera to detect parallax by the exposure control unit;
The image input device according to claim 1, wherein the image combining unit aligns the subject using the detected parallax and combines the images captured by the cameras into a single image.   The image input device according to claim 1, further comprising an array lens in which a plurality of lenses are integrally arranged facing each camera.   The image input device according to claim 1, wherein the array camera includes a plurality of light receiving units corresponding to the plurality of cameras integrally formed on the same substrate.   The image input apparatus according to claim 1, wherein the exposure control unit sets an exposure time of each camera independently.   The image input device according to claim 1, wherein the exposure control unit sets exposure conditions of the cameras to different exposure conditions in order to obtain the high dynamic range image.   The image input apparatus according to claim 2, wherein the exposure control unit sets an exposure condition of each camera so that an exposure time of each camera becomes the same in order to detect the parallax. The parallax detection unit detects the parallax using a corresponding point search method,
Using a search result of corresponding points by the parallax detection unit, further comprising a reliability estimation unit that estimates the reliability of the exposure condition set to detect the parallax,
The exposure control unit changes the exposure conditions until the reliability exceeds a specified value, and causes each camera to take an image.
The image input device according to claim 2, wherein the parallax detection unit detects the parallax using an image captured under an exposure condition in which the reliability exceeds a specified value. The parallax detection unit detects the parallax using a corresponding point search method,
The exposure control unit switches the plurality of exposure conditions to detect the parallax and causes each camera to image the subject multiple times.
Using a search result of corresponding points by the parallax detection unit, further comprising a reliability estimation unit that estimates the reliability of the exposure condition set to detect the parallax,
The image input device according to claim 2, wherein the parallax detection unit detects the parallax using an image captured under an exposure condition that maximizes the reliability. An exposure evaluation value calculation unit that calculates an exposure evaluation value of each camera from an image captured by each camera;
The exposure control unit causes the exposure evaluation value calculation unit to calculate an exposure evaluation value of an image captured by each camera when the reliability is equal to or less than a predetermined value, and the calculated exposure evaluation value is a predetermined appropriate value. The image input apparatus according to claim 8, wherein the exposure condition is changed to an optimum exposure condition for a camera that captures an image that is separated from a range by a certain value or more.   The said image synthetic | combination part specifies the predetermined number of to-be-photographed object from the image imaged by the said camera, and synthesize | combines the image imaged by each camera so that the specified to-be-photographed object may be aligned. The image input device described in 1.