JP2015207861A - Imaging device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide imaging device and method that are flexibly adaptable to user's requirement for ranging precision and dynamic range width of HDR images by using one compound-eye camera.SOLUTION: An imaging device has plural imaging optical systems, a compound-eye camera having plural imaging units which are adaptable to the plural imaging optical systems and image optical images of a target object which are formed by the plural imaging optical systems, an optical filter unit having plural filters which correspond to the plural imaging optical systems and transmit incident light while attenuating the incident light with preset different attenuation factors, a sensitivity changing unit for changing the sensitivity of the compound-eye camera, an image creator for creating one image from images picked up by the plural imaging units, and a controller which has plural modes of different brightness ranges, makes the sensitivity changing unit change the sensitivities of the plural imaging units according to the mode, and makes the image creator create one image within the brightness range with images picked up by the plural imaging units.

Description

  The present invention relates to an imaging apparatus and an imaging method for generating a high dynamic range image.

  In recent years, with the progress of various digital technologies, for example, a digital image can be obtained relatively easily by using a solid-state imaging device such as a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor. This digital image is used in various ways. For example, there are technologies for measuring the distance from a plurality of digital images to a predetermined subject and technologies for creating a high dynamic range image (hereinafter referred to as “HDR image”).

  As a technique for creating an HDR image from a plurality of digital images, there is a technique for imaging the same subject by changing the sensitivity (exposure) of the camera and synthesizing the obtained plurality of images to create an HDR image. When a subject is imaged while changing sensitivity with a single camera, a beautiful HDR image may not be obtained if the subject moves. Therefore, a technique using a compound eye camera that includes a plurality of single-eye cameras and can simultaneously capture a plurality of images of the same subject has been proposed. For example, a technique is disclosed in which an object is imaged by a compound eye camera with different sensitivity of a single-eye camera, and a plurality of obtained images are combined to create an HDR image (see Patent Document 1). In addition, since the parallax (position shift | offset | difference) has arisen in the some image image | photographed using such a compound eye camera, the technique which performs parallax correction at the time of a synthesis | combination is disclosed (refer patent document 3). In addition, a technique is disclosed in which incident light from a subject is input to each eye of a multi-lens optical system by a beam splitter or the like, and an image without parallax is captured (see Patent Document 3).

  Compound eye cameras are also used in technology for measuring the distance from a plurality of digital images to a predetermined subject. For example, the corresponding point search is executed in units of pixels for a pair of images captured by a compound eye camera. Then, the parallax between the pair of single-eye cameras is obtained in units of pixels, and the distance to the subject is obtained based on the so-called triangulation principle based on the obtained parallax.

  In this way, an HDR image can be created by a single compound camera, and the distance to the subject can be measured.

JP 2002-171430 A JP 2011-254265 A JP 2012-199805 A

  However, in order to obtain an HDR image with a wide dynamic range, a plurality of images captured with a plurality of types of sensitivity are required according to the dynamic range width. Then, the wider the dynamic range of the desired HDR image, the more images that are captured with different types of sensitivity are required.

  In addition, the accuracy of distance measurement, that is, the accuracy of obtaining parallax increases as the distance (baseline length) between single-eye cameras that capture images for searching for corresponding points increases. Further, the accuracy of distance measurement increases as the number of single-eye cameras increases, that is, as the number of image pairs for which corresponding point search is performed increases. However, in order to obtain the parallax with high accuracy in the corresponding point search, it is desirable that the pair images for which the corresponding point search is performed are images captured with the same sensitivity.

  Therefore, since the number of single-eye cameras normally included in a compound-eye camera is limited, the number of image pairs for which the corresponding point search is performed decreases as the type of sensitivity increases. That is, if an HDR image having a wide dynamic range is obtained, the accuracy of distance measurement is lowered. Further, if the number of image pairs for which corresponding point search is performed in order to increase the accuracy of distance measurement, the type of sensitivity is reduced. That is, when the accuracy of distance measurement is increased, the dynamic range width of the HDR image is narrowed.

  Different users have different requirements for ranging accuracy and the dynamic range of HDR images. Therefore, it is not practical to create a compound eye camera corresponding to a user's request for each user's request, resulting in an increase in cost.

  Accordingly, an object of the present invention is to flexibly meet a user's request regarding the accuracy of distance measurement and the dynamic range of an HDR image with a single compound eye camera.

  As a result of various studies, the present inventor has found that the above object is achieved by the present invention described below. That is, an imaging apparatus according to an aspect of the present invention corresponds to a plurality of imaging optical systems arranged in an integrated manner, and the plurality of imaging optical systems, and targets formed by the plurality of imaging optical systems. A compound-eye camera having a plurality of imaging units that capture an optical image, and a plurality of filters respectively corresponding to the plurality of imaging optical systems, and dimming incident light at different preset predetermined attenuation rates. And at least two of the plurality of filters have the sensitivity of each of the optical filter unit having the same predetermined attenuation rate and the plurality of imaging units. A sensitivity changing unit to be changed; a corresponding point searching unit that calculates parallax between the plurality of images by searching for corresponding points using images captured by the plurality of imaging units; and the preset predetermined number The imaging unit corresponding to the plurality of filters having the same attenuation rate, and using the plurality of images captured by the imaging unit changed to the same sensitivity by the sensitivity changing unit, parallax is applied to the corresponding point search unit. The brightness range is different from that of the image creation unit that performs calculation, aligns the target using the calculated parallax, and synthesizes the images captured by the plurality of imaging units to create one image. According to the mode, the sensitivity changing unit changes the sensitivity of the plurality of imaging units, and the images created by the plurality of imaging units are used for the image creation unit. And a control unit that creates an image in a luminance range according to the mode.

  Further, an imaging method according to another aspect includes a plurality of imaging optical systems arranged in an integrated manner, and an optical of an object imaged by each of the plurality of imaging optical systems, corresponding to the plurality of imaging optical systems. A compound-eye camera having a plurality of imaging units that capture images, and a plurality of filters respectively corresponding to the plurality of imaging optical systems, wherein incident light is dimmed at different preset predetermined attenuation rates An imaging method that is used in an imaging apparatus that includes a plurality of filters that transmit, and wherein at least two of the plurality of filters include the optical filter unit that has the same predetermined predetermined attenuation rate. The parallax between the plurality of images is calculated by a sensitivity changing step for changing the sensitivity of each of the plurality of imaging units and a corresponding point search using images captured by the plurality of imaging units. A plurality of images picked up by the image pickup unit corresponding to the plurality of filters having the same predetermined predetermined attenuation rate and changed to the same sensitivity by the sensitivity changing step. Using the image of the corresponding point, the parallax is calculated in the corresponding point search step, the target is aligned using the calculated parallax, and the images captured by the plurality of imaging units are combined to obtain one image A plurality of modes having different brightness ranges, and the sensitivity changing step changes the sensitivity of the plurality of imaging units according to the mode, and imaging is performed by the plurality of imaging units. And a control step of creating an image of a luminance range corresponding to each mode in the image creation step using the image that has been obtained.

  In such an imaging apparatus and imaging method, an object is imaged through a plurality of filters that attenuate and transmit incident light at different predetermined preset attenuation rates, and further improve the sensitivity of the imaging unit. Since it can be changed, a plurality of images having different relative luminance ranges can be acquired. Accordingly, it is possible to create an image with a wide luminance range (high dynamic range) according to the mode. In addition, a plurality of images having the same relative luminance range can be taken, and based on these images, an image shift (parallax) can be calculated. It is possible to create a single image. That is, the imaging apparatus can create images with various luminance ranges (dynamic ranges) using one compound eye camera.

  In another aspect, in the above-described imaging device, the predetermined attenuation rate that is set in advance is n types (n ≧ 2), and the luminance range of the image captured by the imaging unit is on the horizontal axis. In the graph with the reliability of the luminance in the luminance range as the vertical axis, the portion of the first graph of the image captured by the first imaging unit corresponding to the filter having the first attenuation rate is a low-luminance portion, and A part of the bottom part, which is a part where the reliability is gradually decreased, corresponds to a filter having a second attenuation factor that is the second lowest attenuation factor after the first attenuation factor, and has the same sensitivity as the first imaging unit. The first attenuation factor and the second attenuation factor are set so as to overlap the second graph of the image captured by the set second imaging unit, and the plurality of filters are filters of n types of attenuation factors. Each has a plurality of filters with n attenuation factors In the plurality of modes, the imaging unit corresponding to the filters of the first group and the second group is changed to the same sensitivity by the sensitivity changing unit. The imaging unit corresponding to the first group filter is changed to the first sensitivity by the first mode and the sensitivity changing unit, and the imaging unit corresponding to the second group filter is higher in sensitivity than the first sensitivity. The first sensitivity and the second sensitivity are changed to the second sensitivity, and the first sensitivity and the second sensitivity of the graph of the image captured by the imaging unit corresponding to the filter having the lowest attenuation rate among the imaging units changed to the first sensitivity The second mode has a sensitivity such that all or part of the sensitivity is overlapped with the graph of the image captured by the imaging unit corresponding to the filter having the highest attenuation rate among the imaging units changed to the second sensitivity. Then, the imaging unit corresponding to the first group of filters is changed to the third sensitivity by the sensitivity changing unit, and the imaging unit corresponding to the second group of filters is higher in sensitivity than the third sensitivity. The third sensitivity and the fourth sensitivity are the skirt portions of the graph of the image captured by the imaging unit corresponding to the filter having the highest attenuation rate among the imaging units changed to the third sensitivity. Are all in the third mode in which the sensitivity of the image captured by the image capturing unit corresponding to the filter having the highest attenuation rate overlaps the graph of the image capturing unit changed to the fourth sensitivity. And

  In such an imaging apparatus, a plurality of images having a continuous relative luminance range can be captured, and the luminance range in the entire plurality of images and the dynamic range in which the reliability of luminance is changed are wide according to the mode. An image can be created. Specifically, in the first mode, the imaging unit having the same sensitivity captures an image through a plurality of attenuation rate filters such that the relative luminance ranges of the captured images are continuous. Can create an image of a luminance range (dynamic range) according to the attenuation rate of the filter. In the second mode, the range of the relative luminance of the images captured by the imaging units having different sensitivities, which are captured through such a plurality of attenuation factor filters, is wider than that in the first mode. An image with a luminance range (dynamic range) can be created. In the third mode, a portion with a low reliability in the range of the relative luminance of the image captured through the plurality of filters with the attenuation factors is the relative luminance of the image captured by the imaging unit having a different sensitivity. Since an image that overlaps the range is captured, it is possible to create an image with a higher luminance reliability than other modes, that is, an image with a wide luminance range (dynamic range) with good image quality.

  In another aspect, in the above-described imaging device, a distance for obtaining a distance to the object based on an image captured by an imaging unit corresponding to at least two imaging optical systems among the plurality of imaging optical systems. An imaging unit is further provided, and the distance calculation unit is an imaging unit corresponding to the plurality of filters having the same predetermined predetermined attenuation rate, and the imaging is changed to the same sensitivity by the sensitivity changing unit. The corresponding point search unit calculates parallax using a plurality of images captured by the unit, and calculates the distance to the object based on the calculated parallax.

  In such an imaging apparatus, since it is possible to capture a plurality of images having the same relative luminance range, the parallax of the object in the image can be calculated based on these images, and the distance to the object can be calculated. It becomes possible to do.

  In another aspect, the imaging apparatus further includes setting input means for setting the mode, and the control unit sends the sensitivity changing unit to the sensitivity changing unit according to the mode set by the setting input means. It is characterized in that the sensitivities of a plurality of imaging units are changed.

  In such an imaging apparatus, the user can set a mode of a desired luminance range (dynamic range), so that an image with a desired dynamic range can be easily obtained.

  According to another aspect, in the above-described imaging device, the control unit determines a mode based on images captured by the plurality of imaging units, and the sensitivity changing unit determines the mode according to the determined mode. It is characterized in that the sensitivities of a plurality of imaging units are changed.

  In such an imaging apparatus, the mode is determined based on the actually captured image, and the sensitivity of the imaging unit is set. Therefore, it is possible to easily capture a high dynamic range image.

  According to another aspect, in the above-described imaging device, the imaging unit includes a solid-state imaging element, and the sensitivity changing unit changes a storage time of the solid-state imaging element included in each of the plurality of imaging units. To change the sensitivity of the plurality of imaging units.

  In such an imaging apparatus, since the sensitivity of the imaging unit is changed by changing the accumulation time of the solid-state imaging device, it is possible to easily perform imaging by changing the sensitivity of the compound eye camera.

  The imaging apparatus and the imaging method according to the present invention can flexibly meet the user's request regarding the accuracy of distance measurement and the wide dynamic range of the HDR image with a single compound eye camera.

It is a block diagram which shows the structure of the imaging device of embodiment. It is a block diagram which shows the structure of the image acquisition part in the imaging device of embodiment. (A) is a figure which shows the arrangement position of the array camera and optical filter in the image acquisition part of embodiment, (b) is a figure which shows the structure of the array camera in the image acquisition part of embodiment, (c) ) Is a diagram showing a configuration of an optical filter unit. It is a figure for demonstrating the sensitivity range (latitude) of an image pick-up element. It is a figure for demonstrating the range of the relative luminance of the image obtained by the image imaging part of embodiment. It is a figure for demonstrating the sensitivity of the optical filter part and array imaging part of embodiment. It is a figure for demonstrating the corresponding point search in the normal HDR mode of embodiment. It is a figure for demonstrating the relative-luminance range in normal HDR mode of embodiment. It is a figure for demonstrating the corresponding point search in the super HDR mode of embodiment. It is a figure for demonstrating the relative-luminance range in the super HDR mode of embodiment. It is a figure for demonstrating the relative-luminance range in the modification of the super HDR mode of embodiment. It is a figure for demonstrating the corresponding point search in the high quality HDR mode of embodiment. It is a figure for demonstrating the relative-luminance range in the high image quality HDR mode of embodiment. It is a figure for demonstrating the ranging method by the stereo method in the imaging device of embodiment. It is a figure for demonstrating the parallax of the image obtained by the array camera of embodiment. It is a figure for demonstrating the parallax of the image obtained by the array camera of embodiment. It is a flowchart which shows the imaging process in the imaging device of embodiment.

Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted suitably. Further, in this specification, when referring generically, it is indicated by a reference symbol without a suffix, and when referring to an individual configuration, it is indicated by a reference symbol with a suffix.
<Embodiment>
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to the embodiment. FIG. 2 is a block diagram illustrating a configuration of an image acquisition unit in the imaging apparatus according to the embodiment. FIG. 3A is a perspective view showing an arrangement position of the array camera and the optical filter in the image acquisition unit, and FIG. 3B is a perspective view showing a configuration of the array camera in the image acquisition unit. FIG. 3C shows the configuration of the optical filter unit 32.

  The imaging apparatus according to the embodiment creates an HDR image and calculates a distance to a subject using an image captured by changing the sensitivity (sensitivity of an imaging element) of a plurality of single-lens cameras constituting an array camera. It is. The imaging apparatus D of the embodiment provides a plurality of imaging modes in which the width of the dynamic range of the HDR image that can be acquired and the distance measurement accuracy are different from each other.

  For example, as illustrated in FIG. 1, the imaging device D according to the embodiment includes a control processing unit 1, a storage unit 2, an image acquisition unit 3, an input unit 4, and an output unit 5.

<Image acquisition unit>
The image acquisition unit 3 is connected to the control processing unit 1 and acquires a plurality of images obtained by imaging the same measurement target (object, subject). Such an image acquisition unit 3 includes, for example, an array camera 31 and an optical filter unit 32 as shown in FIG.

  As shown in FIG. 3A, the array camera 31 includes an array lens unit 311 having a plurality of single-lens lenses (imaging optical systems) and an optical image of an object imaged by each of the plurality of single-lens lenses. And an array imaging unit 312 having a plurality of single-eye imaging units (imaging units) that respectively capture images. The optical filter unit 32 includes a plurality of types of ND filters and is disposed on the object side of the array lens unit 311.

  As illustrated in FIG. 3B, the array lens unit 311 includes a plurality of single-eye lenses 3111. One single lens 3111 includes one or more optical lenses along the optical axis. In the example shown in FIG. 3B, the plurality of single-lens lenses 3111 are arranged so that the optical axes are substantially parallel to each other, and in two linearly independent directions, more specifically, the X direction and the Y direction orthogonal to each other. They are arranged in a two-dimensional matrix in two directions. In the example shown in FIG. 3B, the plurality of single-lens lenses 3111 are shown as twelve single-lens lenses 3111-11 to 3111-34 arranged in a two-dimensional matrix in three rows and four columns.

  The array imaging unit 312 includes a plurality of single-eye imaging units 3121. One single-eye imaging unit 3121 includes a plurality of photoelectric conversion elements (a plurality of pixels) arranged in a two-dimensional matrix, and each photoelectric conversion element converts electricity according to the amount of received light. The signal is output as data of each pixel in the image. The plurality of single-eye imaging units 3121 correspond to the plurality of single-eye lenses 3111 in the array lens unit 311 and are arranged so that the respective imaging surfaces are on the same plane. In the example shown in FIG. 3B, the plurality of single-eye imaging units 3121 are arranged in a two-dimensional matrix in two linearly independent directions, more specifically, two directions of X and Y directions orthogonal to each other. Yes. In the example shown in FIG. 3B, the plurality of single-eye imaging units 3121 are shown as twelve single-eye imaging units 3121-11 to 3121-34 arranged in a two-dimensional matrix in three rows and four columns. Yes. The plurality of single-eye imaging units 3121 may be configured to include a plurality of the solid-state imaging elements arranged in a two-dimensional matrix on the same substrate, but in the example illustrated in FIG. The solid-state imaging device is provided. In this case, the effective pixel area in one solid-state imaging device is divided into a plurality of areas arranged in a two-dimensional matrix so as to correspond to each single-eye imaging unit 3121, and each of these areas is divided into each single-eye imaging unit. Assume that 312 is used.

  The optical filter unit 32 is an optical element that transmits incident light so as to be optically different from region to region. For example, the optical filter unit 32 includes a plurality of sub optical filter regions that transmit incident light so as to be optically different from each other. In the embodiment, as shown in FIG. 3C, the optical filter unit 32 includes eight ND filters 321-1 to 332-34 arranged in 3 rows and 4 columns. The ND filter is a filter that transmits incident light after dimming it with a predetermined attenuation rate. These ND filters 321 may be individual optical elements, or may be optical elements formed integrally.

  ND filter 321-11 arranged in 1 row and 1 column of optical filter section 32, ND filter 321-13 arranged in 1 row and 3 column, ND filter 321-32 arranged in 3 row and 2 column, and The ND filters 321-34 arranged in 3 rows and 4 columns have an optical density (OD (Optical Density) value) of “2.8”. Also, ND filter 321-22 arranged in 2 rows and 2 columns, ND filter 321-24 arranged in 2 rows and 4 columns, ND filter 321-31 arranged in 3 rows and 1 column, and 3 rows and 3 columns The optical density of the ND filters 321 to 33 arranged in the range is “1.4”. ND filter 321-12 arranged in 1 row and 2 columns, ND filter 321-14 arranged in 1 row and 4 columns, ND filter 321-21 arranged in 2 rows and 1 column, and arranged in 2 rows and 3 columns The ND filter 231-34 to be used has an optical density of “0.0”. The optical density “0.0” indicates that the transmittance of the filter is “100%”, the optical density “1.4” indicates that the transmittance is about “4%”, and the optical density “2.8” indicates that The transmittance is about “0.15%”.

  Each of the plurality of ND filters 321 of the optical filter unit 32 is disposed on each object side of each individual lens 3111 corresponding to each of the plurality of individual lenses 3111 provided in the array lens unit 311. Therefore, the light flux from the subject incident through each ND filter 321 is incident through each individual lens 3111 so as to form an optical image of the object on each light receiving surface of each individual image pickup unit 3121. Each photoelectric conversion element of the plurality of single-eye imaging units 3121 photoelectrically converts the received light into an electrical signal corresponding to the amount of light, and outputs the electrical signal to the control processing unit 1 as data of each pixel in the image. To do. The plurality of single-eye imaging units 3121 outputs image data that is shifted by a substantially parallax that represents the same subject.

  The optical filter unit 32 is provided on the object side of the array lens unit 311, but may be provided between the array lens unit 311 and the single-eye imaging unit 3121.

  Here, the sensitivity range of the image sensor (single-eye camera) will be described with reference to FIG. Hereinafter, a combination of the single-eye imaging unit 3121 and the single-eye lens 3111 facing each other is referred to as a “single-eye camera”.

FIG. 4 is a graph schematically showing the relationship between the output of the image sensor and its reliability (hereinafter referred to as “reliability graph”). The horizontal axis represents the relative luminance based on the output of the image sensor, and the vertical axis represents the reliability of the pixel value (output value). The reliability is assumed to increase as it goes in the direction of the arrow on the vertical axis. The reliability refers to gradation expression power, S / N ratio, and the like. “2 ⁶ 2 ⁴ 2 ² 2 ⁰ ” on the horizontal axis indicates that the range of relative luminance “10 ⁰ ” to “10 ⁻² ” is represented by 6 bits.

As shown in the reliability graph of FIG. 4, the reliability of the output value of the image sensor is not constant within the sensitivity range (relative luminance 10 ⁻² to 10 ¹ ) of the image sensor, and the image becomes dark in a relatively dark region. There is a region (hereinafter referred to as “bottom part”) in which the reliability gradually decreases as shown (see arrow 41). It should be noted that the range of brightness perceived by humans is considerably wider than the sensitivity range that the image sensor can capture at a time, so it is desirable to cover the range of brightness that humans can see at once. .

Next, FIG. 5 shows a reliability graph of the image sensor (single-eye camera) when receiving light through the ND filter 321. A graph G1 is a reliability graph when light having passed through the ND filter 321 having an optical density of “2.8”, and a graph G2 represents light having passed through the ND filter 321 having an optical density of “1.4”. The graph is a reliability graph when light is received, and the graph G3 is a reliability graph when light is received through the ND filter 321 having an optical density of “0.0”. That is, since the incident light is attenuated by the ND filter at a predetermined attenuation rate and is incident on the image sensor, the relative luminance ranges of the images output from the single-eye camera are different. In FIG. 5, if the optical density of the ND filter is different by “1.4”, the range of relative luminance of the image output from the single-eye camera is different by “10 ^1.4 ”. That is, since (10 ^1.4 ≈2 ⁴ ), a single-eye camera equipped with an ND filter having a different optical density can cover a relative luminance range shifted by 4 bits.

  Therefore, by providing the three types of ND filters 321, the ND filter 321 is not provided in the range of relative luminance of the entire plurality of images output from the array camera 31 (hereinafter referred to as “relative luminance range of the array camera 31”). Compared to the case, the relative luminance range is wide.

<Input section>
Returning to FIG. 1, the input unit 4 is connected to the control processing unit 1, and is necessary for executing various types of commands such as a command for instructing the start of imaging, and imaging processing such as setting of an imaging mode, for example. It is a device that inputs various data to the imaging device D. For example, an operation unit attached to the main body of the imaging apparatus D, such as a button or a slide switch. Further, it may be a keyboard such as a personal computer or a mouse connected so as to be communicable.

<Output unit>
The output unit 5 is connected to the control processing unit 1 and outputs commands and data input from the input unit 4, an HDR image created by the imaging device D, a distance to a measured object, and the like. For example, a display device such as a CRT display, an LCD, and an organic EL display.

  A touch panel may be configured from the input unit 4 and the output unit 5. In the case of configuring this touch panel, the input unit 4 is a position input device that detects and inputs an operation position such as a resistive film method or a capacitance method, and the output unit 5 is a display device. In this touch panel, a position input device is provided on the display surface of the display device, one or more input content candidates that can be input to the display device are displayed, and the user touches the display position where the input content to be input is displayed. Then, the position is detected by the position input device, and the display content displayed at the detected position is input to the imaging device D as the operation input content of the user. With such a touch panel, the user can easily understand the input operation intuitively, and thus an imaging device D that is easy for the user to handle is provided.

<Storage unit>
The storage unit 2 is a device that is connected to the control processing unit 1 and stores various programs and various data necessary for executing the imaging process. For example, the storage unit 2 is a ROM (Read Only Memory) or an EEPROM (Electrically Erasable Programmable Read Only). Non-volatile memory elements such as Memory), volatile memory elements such as RAM (Random Access Memory), and peripheral circuits thereof. The storage unit 2 may include a large-capacity storage device such as a hard disk.

  The storage unit 2 stores image data, and is used as a work area for performing imaging processing described later on the image data by the control processing unit 1.

<Control processing unit>
The control processing unit 1 controls each part of the imaging device D according to the function of each part so as to execute an imaging process described later based on a predetermined imaging process program. The predetermined imaging processing program is a program for imaging a subject according to an imaging mode set by a user, creating an HDR image, and measuring a distance to the subject. The control processing unit 1 includes, for example, a CPU (Central Processing Unit) and its peripheral circuits. By executing a predetermined imaging processing program, the control processing unit 1 functionally includes the control unit 11, A sensitivity adjustment unit 12 and a calculation unit 13 are configured.

  The control unit 11 controls each unit of the imaging apparatus D according to the function of each unit in order to execute a predetermined imaging process.

  The sensitivity adjustment unit 12 changes the photon accumulation time (integration time) of the image sensor of the single-eye imaging unit 3121. That is, the sensitivity of the single-eye camera is changed by changing the light reception time of the photoelectric conversion element of the single-eye imaging unit 3121. Hereinafter, the combination of the single-eye imaging unit 3121-11 and the single-eye lens 3111-11 is referred to as “single-eye camera 11”, and the combination of the single-eye imaging unit 3121-12 and single-eye lens 3111-12 is referred to as “single-eye camera 12”. ".

  FIG. 6 shows the optical filter unit 32, where the rectangle indicates the ND filter 321, and the number enclosed by a circle in the rectangle indicates the optical density of the ND filter 321. “1” indicates an optical density “2.8”, “2” indicates an optical density “1.4”, and “3” indicates an optical density “0.0”. That is, the ND filter 321 with the same numeral is the ND filter 321 having the same optical density. Hereinafter, the optical density “2.8” is referred to as “optical density (0)”, the optical density “1.4” is referred to as “optical density (2)”, and the optical density “0.0” is referred to as “optical density (3)”. There is a case.

  In the embodiment, as shown in FIG. 6, the sensitivity adjustment unit 12 divides the ND filter 321 into groups, that is, the 12 single-eye cameras of the array camera 31 are divided into two groups (group A and group B). Separately, change the sensitivity for each group. That is, the group A includes the single camera 11, the single camera 12, the single camera 21, the single camera 22, the single camera 31, and the single camera 32, and the group B includes the single camera 13, The camera is composed of a single camera 14, a single camera 23, a single camera 24, a single camera 33, and a single camera 34.

  The sensitivity adjustment unit 12 changes the sensitivities of the group A and the group B according to the imaging mode set by the user. The processing according to the imaging mode will be described in the section <imaging mode>.

  For example, as illustrated in FIG. 1, the calculation unit 13 includes an HDR image generation unit 131, a corresponding point search unit 132, and a distance calculation unit 133.

  The HDR image generation unit 131 combines the images picked up by the array camera 31 by shifting the parallax, and generates one HDR image. That is, the HDR image generation unit 131 performs alignment of the subject in each image and synthesizes each image. The corresponding point search unit 132 calculates the parallax between the images. In addition, the HDR image generation unit 131 determines a dynamic range (luminance range) based on output values from the imaging elements constituting each of the plurality of single-eye imaging units 3121 of the array imaging unit 312, and images are taken by the array camera 31. The synthesized images are combined to create one image of the determined dynamic range.

  Here, the image shift will be described with reference to FIGS. The direction in which the shift of each image appears is based on the position of the single lens of the single camera that captured each image. FIG. 15A shows a composite image obtained by superimposing two images captured by two single-eye cameras, and FIG. 15B shows an array camera array that captures the two images of FIG. An imaging part is shown. The array imaging unit of this array camera is composed of nine single-eye imaging units arranged in a two-dimensional matrix of 3 rows and 3 columns. A rectangle in which numerical values are represented indicates a single-eye imaging unit. Here, images captured by the single-eye imaging unit are represented as images 1 and 2 using numerical values in the rectangle. And The composite image shown in FIG. 15A is a composite image of image 1 and image 2 shown in FIG. The difference between the image 1 and the image 2 is the intersection of the optical axis of the single lens that captured the image 1 and the imaging surface of the single image capturing unit, and the optical axis of the single lens that captured the image 2 and the single image capturing unit. Appear in the direction connecting the intersection with the imaging surface (see arrow 152). The direction of the arrow 151 on the composite screen in FIG.

  Similarly, FIG. 16A shows a composite image in which two images taken by two single-eye cameras are superimposed, and FIG. 16B shows an array in which the two images in FIG. 16A are taken. The array imaging part of a camera is shown. The composite image shown in FIG. 16A is a composite image of the image 2 and the image 5 shown in FIG. The deviation between the image 2 and the image 5 is the intersection of the optical axis of the single-lens lens that captured the image 2 and the imaging surface of the single-eye imaging unit, the optical axis of the single-lens lens that captured the image 5, and the single-eye imaging unit. Appear in the direction connecting the intersection with the imaging surface (see arrow 162). The direction of the arrow 161 on the composite screen in FIG. The direction of this shift is the so-called epipole line direction. This is because the epipole line is a line obtained by projecting the line of sight of the camera on the base image side onto the reference image. When single-cameras are arranged in a two-dimensional matrix in two directions of X and Y directions orthogonal to each other as shown in FIG. 3, and the baseline lengths between single-cameras are equal, as shown in FIG. Further, two images in the X-axis direction, for example, the shift amount between the images 1 and 2, and two images in the Y-axis direction, for example, the shift amount between the images 2 and 5, as shown in FIG. That's fine. This is because the amount of deviation in the X-axis direction between images is the same between the images adjacent to the left and right, and the amount of deviation in the Y-axis direction is the same between the upper and lower images. Specifically, for example, the shift amount in the X-axis direction between the image 1 and the image 2, the shift amount between the image 2 and the image 3, and the shift amount between the image 4 and the image 5 are the same. The amount of deviation in the X-axis direction between other images is the same (see FIG. 15B). Further, the amount of deviation between the images 5 and 2 in the Y-axis direction, the amount of deviation between the images 5 and 8, and the amount of deviation between the images 1 and 4 are the same. The amount of deviation in the axial direction is the same (see FIG. 16B). That is, the HDR image generation unit 131 can easily obtain the parallax between each image necessary for creating the HDR image. The image that the HDR image generation unit 131 selects to obtain the parallax is the same as the image pair that the distance calculation unit 133 (to be described later) selects to obtain the parallax. Which image pair is used will be described in the section <Imaging mode>. Note that the image pair is not limited to the adjacent image pair in the Y-axis direction in the X-axis direction, and one or more images may be sandwiched therebetween, or may be an image at a licked position.

Moreover, the following documents can be referred for the algorithm which puts the high dynamic range data obtained from a plurality of images into the same image data.
M.M. Devevec: “Recovering High Dynamic Range Radiance Map from Photographic”. In: Proc. SIGGRAPH (1997)
S. B. King, M.M. Uyttendaele, S.M. Winder, R .; Szeliski: “High Dynamic Range Video”. ACM Transactions on Graphics (TOG), 22, 3 (Jully 2003)
The corresponding point search unit 132 calculates a shift amount (parallax) between a plurality of images captured by the array camera 31. Specifically, the corresponding point search unit 132 performs a corresponding point search using any one of images captured by the array camera 31 as a reference image and any other image as a reference image, and parallax (the number of shifted pixels). Ask for. The corresponding point search unit 132 sets, on the reference image, a window (template T) having a predetermined size around a predetermined pixel position where a corresponding point is to be searched. Similarly, the corresponding point search unit 132 sets a plurality of windows W having the same size on the reference image on the epipole line. Then, the corresponding point search unit 132 obtains a matching degree (correlation degree) of each window W on the reference image with respect to the template T on the standard image by a predetermined method (template matching), and in the window W having the highest matching degree. The pixel at the center position is obtained as the corresponding point. The corresponding point search unit 132 calculates the difference between the pixel at the center position of the template T and the corresponding point as a deviation amount. As the predetermined method for obtaining the degree of coincidence, known conventional means, for example, SAD (Sum of Absolute Difference) method (absolute error method), POC (Phase-Only Correlation), or the like is used.

The distance calculation unit 133 calculates the distance to the subject using the parallax obtained by the corresponding point search unit 132. The calculation of the distance will be described with reference to FIG. Each image of the subject is obtained by a pair of single-lens cameras that are spaced apart by a predetermined interval (baseline length), and corresponding point search is performed for each of these images in units of pixels. The parallax between the pair of cameras is determined in units of pixels. Then, the distance to the subject is obtained based on the so-called triangulation principle based on the obtained parallax. More specifically, in FIG. 14, two first and second units having at least the focal length (f), the number of pixels on the imaging surface (single-eye imaging unit 3121), and the size (μ) of one pixel are equal to each other. The cameras are arranged in parallel with the optical axes AX-1 and AX-2 separated from each other by a predetermined base line length (B). When the object (subject) Ob is captured by the first and second cameras, and the parallax (the number of shifted pixels) on the imaging surfaces IP-1 and IP-2 is d (= d1 + d2), the object The distance (Z) to (subject) Ob is similar to the triangle shown by hatching,
Z: f = B: (d × μ)
There is a relationship
Z = (B × f) / (d × μ)
Can be obtained.

  The distance calculation unit 133 selects two images (image pairs) used to calculate the parallax when calculating the distance to the subject, passes the selected image pair to the corresponding point search unit 132, and obtains the parallax. Let Which image pair is used to calculate the parallax differs depending on the imaging mode set by the user. Which image pair is used will be described in the section <Imaging mode>.

  In the case where a plurality of parallaxes are obtained from a plurality of image pairs, the distance calculation unit 133 determines the distance to the subject based on the calculated distance. For example, the average of each distance is used as the distance to the measurement object, or the more reliable distance is employed.

<Imaging mode>
Hereinafter, imaging modes provided in the imaging apparatus D will be described. The imaging device D has roughly three imaging modes. The first mode is a “normal HDR mode”, the second mode is a “super HDR mode”, and the third mode is a “high quality HDR mode”. The “super HDR mode” and the “high quality HDR mode” each have a “still body shooting mode”. Therefore, the imaging device D has five imaging modes.

<Normal HDR mode>
Hereinafter, the sensitivity adjustment of the sensitivity adjustment unit 12 and the image pair selected by the distance calculation unit 133 in the normal HDR mode will be described with reference to FIGS.

  The normal HDR mode is a mode in which the most accurate distance measurement is performed and the HDR image with the narrowest dynamic range is created among the three imaging modes provided in the imaging apparatus D. Therefore, this mode can be said to be an imaging mode that is set when the accuracy of distance is more important than the width of the dynamic range.

  FIG. 7 is a diagram of the optical filter unit 32 for explaining sensitivity adjustment of a single-eye camera and image pairs. FIG. 8 is a diagram for explaining a range of relative luminance of an image output from the array camera 31.

The sensitivity adjustment unit 12 adjusts the group A and the group B in FIG. 6 to the same sensitivity. That is, the sensitivity adjustment unit 12 sets the light reception times of the imaging elements of all the single-eye imaging units 3121 constituting the array imaging unit 312 to be the same time. In the reliability graph of FIG. 8, a graph G3 shows a range of relative luminance of an image output by a single-eye camera provided with the ND filter 321 having an optical density (3), and a graph G2 shows an optical density (2). The range of the relative luminance of the image output by the single camera provided with the ND filter 321 is shown, and the graph G1 shows the relative luminance of the image output by the single camera provided with the ND filter 321 of optical density (1). Indicates the range. In each of the ND filters of optical densities (1), (2), and (3), the ND portion of each of the graphs G1, G2, and G3 has the next lowest optical density value (low attenuation factor). Each optical density is set so as to overlap the filter graph. In the embodiment, the optical density (3) is an optical density “0.0”, the optical density (2) is an optical density “1.4”, and the optical density (1) is an optical density “2.8”. It is. Therefore, if the optical density of the ND filter is different by “1.4”, the range of relative luminance of the image output by the single-eye camera is different by “10 ^1.4 ”. range of relative luminance becomes approximately ^{10 4.5} (see double arrow 81).

  The distance calculation unit 133 selects an image pair for calculating parallax for distance measurement as a pair of images captured by a single-eye camera provided with the ND filter 321 having the same optical density. That is, the distance calculation unit 133 selects an image pair from the four images. In FIG. 7, for example, three image pairs of an image pair 71, an image pair 72, and an image pair 73 are selected. In this case, a corresponding point search is performed using an image common to the three image pairs, that is, an image captured by the single-eye camera 33 provided with the ND filter 321 of 3 rows and 3 columns as a reference image. In such three image pairs, the base line lengths between the paired single-eye cameras are different, and the directions of the base line lengths are different. Corresponding point search using such an image pair is a high-precision parallax even when the image showing the subject includes a pattern in which a predetermined image repeatedly appears at regular intervals, for example, a striped pattern. It is highly possible to calculate. Even if occlusion occurs, if the number of image pairs is large, there is a high possibility that the parallax can be calculated with high accuracy. Therefore, in the normal HDR mode, since the number of images captured by the single-eye camera provided with the ND filter 321 having the same optical density is large, the selection range of the image pair is wide, and the distance calculation unit 133 can select an appropriate image. Since it is possible to select a pair, ranging accuracy is increased.

  Furthermore, the images of the single camera 11 provided with the ND filter 321 of 1 row and 1 column and the single camera 34 provided with the ND filter 321 of 3 rows and 4 columns may be selected as an image pair. .

  As described above, the HDR image generation unit 131 selects an image pair necessary for calculating the shift amount between images from among the image pairs selected by the distance calculation unit 13.

<Super HDR mode>
Hereinafter, the sensitivity adjustment of the sensitivity adjustment unit 12 and the image pair selected by the distance calculation unit 133 in the super HDR mode will be described with reference to FIGS.

  The super HDR mode is a mode in which an image in a relatively wide relative luminance range can be acquired and a distance measurement with a relatively high accuracy is performed. Therefore, this mode can be said to be an imaging mode that is set when both the width of the dynamic range and the accuracy of the distance are emphasized to some extent.

  FIG. 9 is a diagram of the optical filter unit 32 for explaining sensitivity adjustment of a single-eye camera and image pairs. FIG. 10 is a diagram for explaining the relative luminance range of the array camera 31.

The sensitivity adjustment unit 12 makes the sensitivity of the group A single camera different from the sensitivity of the group B single camera. For example, the light reception time of group A is adjusted to be longer than the light reception time of group B. That is, among the single-eye imaging units 3121 constituting the array imaging unit 312, the light-receiving time of the imaging device of the single-eye imaging unit 3121 belonging to group A is determined from the light-receiving time of the imaging device of the single-eye imaging unit 3121 belonging to group B. Also make it longer. As shown in FIG. 10, the time for increasing the light receiving time is a time such that the ranges of the relative luminance of the images output from the single-eye cameras partially overlap. In the reliability graph of FIG. 10, a graph GA3 indicates a range of relative luminance of an image output by a single-lens camera provided with the ND filter 321 of the optical density (3) of group A, and a graph GA2 indicates the group A The range of the relative luminance of the image output by the single camera provided with the ND filter 321 having the optical density (2) is shown. The graph GA1 is a single eye provided with the ND filter 321 having the optical density (1) of group A. Indicates the range of relative luminance of the image output by the camera. A graph GB3 indicates a range of relative luminance of an image output by a single-eye camera provided with the ND filter 321 of the optical density (3) of the group B, and a graph GB2 indicates the optical density (2) of the group B. The range of the relative luminance of the image output by the single camera provided with the ND filter 321 is shown, and the graph GB1 shows the image output by the single camera provided with the ND filter 321 of the optical density (1) of group B. Indicates the range of relative luminance. As shown in FIG. 10, the relative luminance range of the array camera 31 becomes approximately ¹⁰⁹ (see double arrow 101). As illustrated in FIG. 10, the imaging apparatus D can widen the relative luminance range of the array camera 31 by making the light reception time of the group A different from the light reception time of the group B.

  Note that, as shown in FIG. 10, the light reception time during which a part of the range of the relative luminance of the image output from each single camera overlaps is obtained and determined in advance.

  The distance calculation unit 133 selects the image pair 91 or the image pair 92 as shown in FIG. As shown in FIG. 9, two images are included in each of groups A and B as shown in FIG. 9 by the single-eye camera provided with the ND filter 321 having the same optical density. For example, in group B, the image pair by the ND filter 321 having the optical density (1) is the image pair 91, and the image pair by the ND filter 321 having the optical density (2) is the image pair 92. In other words, since there is one image pair in the same relative luminance range, the accuracy of calculating the parallax by the corresponding point search may not be high.

  Therefore, as shown in FIG. 11, the sensitivity adjustment unit 12 may adjust the light reception times of the groups A and B so that the graph GA1 and the graph GB3 overlap. In this case, two images obtained by two single-eye cameras provided with the ND filter 321 having the optical density (1) of the group A and two single eyes provided with the ND filter 321 having the optical density (3) of the group B. The distance calculation unit 133 can perform a corresponding point search using a total of four images including two images from the camera. That is, the distance calculation unit 133 can acquire the parallax obtained with high accuracy. When the relative luminance range of the array camera 31 shown in FIG. 10 is wider than the desired relative luminance range, as shown in FIG. 11, the imaging device D is configured so that the sensitivity ranges of some single-eye cameras overlap. By causing the sensitivity adjustment unit 12 to adjust the sensitivity of the single-eye camera, highly accurate distance measurement can be performed.

  The HDR image generation unit 131 selects an image pair necessary for calculating a shift amount between images from among the image pairs selected by the distance calculation unit 13.

<Super HDR mode: Still body imaging mode>
When the subject is a still body, the subject does not change in the captured image even if the image is captured a plurality of times. Therefore, the imaging apparatus D captures an image in the super HDR mode, and further captures an image again in the normal HDR mode. That is, as illustrated in FIG. 9, the imaging device D captures the subject by changing the sensitivity of the single-camera of the group A and the sensitivity of the single-camera of the group B. The same subject is imaged with the sensitivity of the single camera and the sensitivity of the single camera of group B being the same.

  By imaging twice in this way, the imaging apparatus D can create an HDR image from an image captured by the array camera 31 having a wide relative luminance range as shown in FIG. 10, and at the same time as shown in FIG. With the image pair, it becomes possible to perform high-precision distance measurement.

<High quality HDR mode>
Hereinafter, the sensitivity adjustment of the sensitivity adjustment unit 12 and the image pair selected by the distance calculation unit 133 in the high image quality HDR mode will be described with reference to FIGS.

  The high image quality HDR mode is a mode for creating an HDR image with the best image quality, that is, a high S / N ratio, and a mode for performing high-precision distance measurement. Therefore, this mode can be said to be an imaging mode that is set when importance is attached to the image quality of the HDR image and the ranging accuracy.

  FIG. 12 is a diagram of the optical filter unit 32 for explaining sensitivity adjustment of a single-eye camera and image pairs. FIG. 13 is a diagram for explaining the relative luminance range of the array camera 31.

The sensitivity adjustment unit 12 makes the sensitivity of the group A single camera different from the sensitivity of the group B single camera. For example, the sensitivity adjustment unit 12 adjusts the light reception time of group A to be longer than the light reception time of group B. As shown in FIG. 13, the time for increasing the light reception time is a time such that the ranges of relative luminance of the images output from the single-eye cameras overlap. As described with reference to FIG. 4, the reliability of the output value of the image sensor becomes lower as it becomes darker in the sensitivity range (relative luminance 10 ⁻² to 10 ¹ ) of the image sensor (in FIG. 4). (See arrow 41). Therefore, in the relative luminance range of the array camera 31, there is no portion where the reliability is low, that is, the portion where the reliability of the range of the relative luminance of the image output by the single camera is low is that of the other single camera. The sensitivity adjustment unit 12 adjusts the light reception times of the group A and the group B so as to overlap with the range of relative luminance with high reliability.

  Note that, as shown in FIG. 13, the light reception time in which the relative luminance range of the image output from the single camera supplements the low reliability portion of the relative luminance range of the image output from the other single camera. Is determined and determined in advance.

In FIG. 13, the low reliability portion of the graph GB1 is supplemented by a high reliability portion of the graph GA1, and the low reliability portion of the graph GB2 is supplemented by a high reliability portion of the graph GA2. The portion with low reliability is supplemented by the portion with high reliability of the graph GA3. The relative luminance range of the array camera 31 becomes approximately 10 ⁴ (see double arrow 131). Although the relative luminance range of the array camera 31 is narrower than that in the super HDR mode, the image quality is improved as compared with the super HDR mode.

  The distance calculation unit 133 selects three pairs of an image pair 121, an image pair 122, and an image pair 123 as illustrated in FIG. That is, as shown in FIG. 13, the relative luminance range of the image output from the group A single camera and the relative luminance range of the image output from the group B single lens have an overlapping portion. Therefore, the distance calculation unit 133 can select an image pair from the images obtained by the group A and group B single-eye cameras. That is, the distance calculation unit 133 can select an image pair from four images. Accordingly, there is a high possibility that the distance calculation unit 133 can calculate the parallax with high accuracy as in the normal HDR mode. In order to calculate the parallax, when performing a corresponding point search using an image from the group A single camera and an image from the group B single camera, sensitivity correction is performed on one of the data. The corresponding point search is performed using the data after the sensitivity correction.

  The HDR image generation unit 131 selects an image pair necessary for calculating a shift amount between images from among the image pairs selected by the distance calculation unit 13.

<High quality HDR mode: Still body imaging mode>
When the subject is a still body, the subject does not change in the captured image even if the image is captured a plurality of times. Therefore, the imaging device D captures an image in the high quality HDR mode, and further captures an image in the normal HDR mode again. That is, as illustrated in FIG. 12, the imaging device D images the subject by changing the sensitivity of the group A single camera and the sensitivity of the group B single camera. The same subject is imaged with the sensitivity of the single camera and the sensitivity of the single camera of group B being the same.

  By imaging twice in this way, the imaging apparatus D can create a high-quality HDR image using an image captured by the array camera 31 having a sensitivity range as shown in FIG. 13, and at the same time, shown in FIG. Such an image pair makes it possible to perform highly accurate distance measurement.

<Operation>
Next, the operation of the imaging apparatus D in this embodiment will be described with reference to FIG. FIG. 17 is a flowchart of the imaging process of the imaging device D. In the flowchart of FIG. 17, it is assumed that processes with the same step number have the same processing contents.

  The user operates the input unit 4 to set the imaging mode. The control unit 11 that has received the imaging mode set by the user via the control processing unit 1, when the set imaging mode is the “normal HDR mode” (step S <b> 11: Yes), the sensitivity adjustment unit 12. Instruct the group A and group B to have the same sensitivity. Upon receiving the instruction, the sensitivity adjustment unit 12 sets the same time as the light reception time of the single-eye imaging unit 3121 of the array camera 31 of the image acquisition unit 3 (step S13).

  When the user operates the input unit 4 to input a command for instructing imaging, the control unit 11 that has received the imaging instruction via the control processing unit 1 receives the sensitivity set by the sensitivity adjustment unit 12 from the image acquisition unit 3. A plurality of images picked up in (1) are acquired and stored in the storage unit 2 (step S13).

  Then, the control unit 11 instructs the HDR image generation unit 131 to create an HDR image. Upon receiving the instruction, the HDR image generation unit 131 reads out the image from the storage unit 2, selects an image pair that can calculate the shift between the images, and passes the selected image pair to the corresponding point search unit 132. , Request the calculation of the parallax (deviation amount). Upon receiving the request, the corresponding point search unit 132 performs a corresponding point search using the pair of images, obtains the parallax, and passes the parallax to the HDR image generation unit 131.

  The HDR image generation unit 131 that has received the shift amount between images from the corresponding point search unit 132 combines the images read from the storage unit 2 by shifting the shift amount, and generates one HDR image (step S14). ).

  Further, the control unit 11 instructs the distance calculation unit 133 to calculate the distance to the subject. Upon receiving the instruction, the distance calculation unit 133 reads out the image pair as described above from the storage unit 2, passes the read image pair to the corresponding point search unit 132, and requests the calculation of parallax. Upon receiving the request, the corresponding point search unit 132 performs a corresponding point search using the pair of images, obtains the parallax, and passes the parallax to the distance calculation unit 133.

  The distance calculation unit 133 that has received the parallax from the corresponding point search unit 132 calculates the distance to the subject as described with reference to FIG. 14 (step S15).

  The control unit 11 outputs the HDR image created by the HDR image generation unit 131 and the distance to the subject calculated by the distance calculation unit 133 to the output unit 5.

  In step S11, the set imaging mode is not “normal HDR mode” (step S11: No), the set imaging mode is “super HDR mode” (step S21: Yes), and “still body imaging mode”. If not (step S22: No), the control unit 11 sets the sensitivity of the group A and the sensitivity of the group B to the sensitivity adjustment unit 12 as described in the <Super HDR mode>, respectively. Instructs the sensitivity to be predetermined for the mode. Upon receiving the instruction, the sensitivity adjustment unit 12 sets the light reception time predetermined for the super HDR mode for each group as the light reception time of the single-eye imaging unit 3121 of the array camera 31 of the image acquisition unit 3 (Step S1). S23).

  When the user operates the input unit 4 to input a command for instructing imaging, the control unit 11 that has received the imaging instruction via the control processing unit 1 receives the sensitivity set by the sensitivity adjustment unit 12 from the image acquisition unit 3. A plurality of images picked up in (1) are acquired and stored in the storage unit 2 (step S24).

  Then, the control unit 11 instructs the HDR image generation unit 131 to create an HDR image. Upon receiving the instruction, the HDR image generation unit 131 reads an image from the storage unit 2, selects an image pair, passes the selected image pair to the corresponding point search unit 132, and calculates parallax (deviation amount). One HDR image is generated using the shift amount (step S25).

  Further, the control unit 11 instructs the distance calculation unit 133 to calculate the distance to the subject. Upon receiving the instruction, the distance calculation unit 133 reads out the image pair as described in the <Super HDR mode> section from the storage unit 2, passes the read image pair to the corresponding point search unit 132, and calculates and calculates the parallax. Using the parallax, the distance to the subject is calculated (step S26).

  The control unit 11 outputs the HDR image created by the HDR image generation unit 131 and the distance to the subject calculated by the distance calculation unit 133 to the output unit 5.

  In step S22, when it is “still body imaging mode” (step S22: Yes), the control unit 11 performs processing from step S23 to step S25, that is, performs imaging in the super HDR mode, and displays an HDR image. create.

  Then, the control unit 11 performs the processing of steps S12, S13, and S15, that is, performs imaging in the normal HDR mode, and calculates the distance to the subject.

  The control unit 11 outputs the created HDR image and the calculated distance to the subject to the output unit 5.

  In step S21, when the set imaging mode is the “high image quality HDR mode” (step S21: No) and not the “still body imaging mode” (step S41: No), the control unit 11 determines that <high As described in the section “Image Quality HDR Mode>, the sensitivity adjustment unit 12 is instructed to set the sensitivity of the group A and the sensitivity of the group B to the predetermined sensitivity for the high image quality HDR mode. Upon receiving the instruction, the sensitivity adjustment unit 12 sets the light reception time predetermined for the high image quality HDR mode as the light reception time of the single-eye imaging unit 3121 of the array camera 31 of the image acquisition unit 3 for each group ( Step S42).

  When the user operates the input unit 4 to input a command for instructing imaging, the control unit 11 that has received the imaging instruction via the control processing unit 1 receives the sensitivity set by the sensitivity adjustment unit 12 from the image acquisition unit 3. A plurality of images picked up in (1) are acquired and stored in the storage unit 2 (step S43).

  Then, the control unit 11 instructs the HDR image generation unit 131 to create an HDR image. Upon receiving the instruction, the HDR image generation unit 131 reads an image from the storage unit 2, passes the image to the corresponding point search unit 132, calculates parallax (deviation amount), and uses the calculated deviation amount to produce one HDR image. Is generated (step S44).

  Further, the control unit 11 instructs the distance calculation unit 133 to calculate the distance to the subject. Upon receiving the instruction, the distance calculation unit 133 reads out the image pair as described in the section of <High Quality HDR Mode> from the storage unit 2, passes the read image pair to the corresponding point search unit 132, and calculates the parallax. A distance to the subject is calculated using the calculated parallax (step S45).

  The control unit 11 outputs the HDR image created by the HDR image generation unit 131 and the distance to the subject calculated by the distance calculation unit 133 to the output unit 5.

  In step S41, when it is “still body imaging mode” (step S41: Yes), the control unit 11 performs the processing from step S42 to step S44, that is, performs imaging in the high image quality HDR mode, and performs the HDR image. Create

  Then, the control unit 11 performs the processing of steps S12, S13, and S15, that is, performs imaging in the normal HDR mode, and calculates the distance to the subject.

  The control unit 11 outputs the created HDR image and the calculated distance to the subject to the output unit 5.

  As described above, in the imaging apparatus D, the user can easily select the dynamic range of the image and the ranging accuracy.

  In the embodiment, the user inputs the imaging mode, but a desired dynamic range may be input. In this case, the imaging device D sets an appropriate imaging mode according to the input dynamic range.

  In the embodiment, the width of the dynamic range is determined in advance for each imaging mode. However, the user may input an arbitrary width of the dynamic range. For example, the array camera 31 can be changed by changing the overlap width between the relative luminance range of the image output from the single-camera group A in FIG. 11 and the relative luminance range of the image output from the single-camera group B in FIG. This is because the relative luminance range can be changed to an arbitrary width. This adjustment can be easily realized by adjusting the light reception time of group A and the light reception time of group B.

  Further, the imaging device D may calculate a necessary dynamic range (relative luminance range) while imaging the subject as a moving image, and automatically set the sensitivity of each of the group A and the group B. For example, the super HDR mode is set when the dark area in the captured image is a certain ratio or more of the entire image, and the standard HDR mode is set when the ratio of the dark area is lower than the certain ratio. It is. Further, when it is determined that there is no change in the time-series images and the subject is a still body, the still body imaging mode of the super HDR mode is set.

  Furthermore, the user may be able to adjust the dynamic range while viewing the automatically set captured image displayed on the output unit 5. In addition, the imaging apparatus D first captures a subject in the super HDR mode and displays a captured image, and then changes the dynamic range in accordance with the user's operation of a slide switch or the like, and sets the appropriate mode. It is good.

  In addition, the imaging device D first captures a subject in a plurality of imaging modes, displays a plurality of captured images captured in different imaging modes on the output unit 5, and allows the user to select and select an image as desired. It is also possible to set the imaging mode for the captured image. In addition, the imaging apparatus D may cause a user to select a desired image by causing a plurality of captured images captured in the imaging mode to be connected to a personal computer or the like that is communicably connected.

  In the embodiment, the imaging apparatus D creates a high dynamic range image. However, the imaging apparatus D may create a super-resolution image using a plurality of images obtained from the array camera 31. Ranging using a super-resolution image may be performed.

  In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.

D Distance measuring device 12 Sensitivity adjustment unit 13 Calculation unit 131 HDR image generation unit 132 Corresponding point search unit 133 Distance calculation unit 3 Image acquisition unit 31 Array camera 311 Array lens unit 312 Array imaging unit 32 Optical filter unit 321 ND filter

Claims (7)

A compound eye having a plurality of imaging optical systems arranged in an integrated manner and a plurality of imaging units corresponding to the plurality of imaging optical systems and imaging optical images of objects formed by the plurality of imaging optical systems, respectively. A camera,
A plurality of filters respectively corresponding to the plurality of imaging optical systems, each having a plurality of filters that attenuate and transmit incident light at different preset predetermined attenuation rates; , At least two filters, the optical filter unit having the same predetermined attenuation rate set in advance,
A sensitivity changing unit that changes the sensitivity of each of the plurality of imaging units;
A corresponding point search unit that calculates parallax between the plurality of images by corresponding point search using images captured by the plurality of imaging units;
The imaging unit corresponding to the plurality of filters having the same preset predetermined attenuation rate, and using a plurality of images captured by the imaging unit changed to the same sensitivity by the sensitivity changing unit, Image creation that causes a corresponding point search unit to calculate parallax, aligns the object using the calculated parallax, and synthesizes images captured by the plurality of imaging units to create one image And
The image creation using a plurality of modes having different luminance ranges, causing the sensitivity changing unit to change the sensitivity of the plurality of imaging units according to the mode, and using images captured by the plurality of imaging units An image pickup apparatus comprising: a control unit that causes the unit to create an image of a luminance range corresponding to each mode. The preset predetermined attenuation rate is n types (n ≧ 2),
An image captured by the first imaging unit corresponding to the filter of the first attenuation rate in a graph with the luminance range of the image captured by the imaging unit on the horizontal axis and the reliability of the luminance in the luminance range on the vertical axis In the first graph, a part of the skirt portion where the luminance is low and the reliability is gradually decreased is a second attenuation factor having the second lowest attenuation factor after the first attenuation factor. The first attenuation factor and the second attenuation factor are set so as to overlap with the second graph of the image captured by the second imaging unit set to the same sensitivity as the first imaging unit. ,
The plurality of filters each include a plurality of filters of n types of attenuation factors, and each of the plurality of filters includes a first group and a second group each including a filter of n types of attenuation factors.
The plurality of modes are
A first mode in which the imaging units corresponding to the filters of the first group and the second group are changed to the same sensitivity by the sensitivity changing unit;
The imaging unit corresponding to the filter of the first group is changed to the first sensitivity by the sensitivity changing unit, and the imaging unit corresponding to the filter of the second group is changed to the second sensitivity having a higher sensitivity than the first sensitivity. The first sensitivity and the second sensitivity are all or a part of the graph of the image captured by the imaging unit corresponding to the filter having the lowest attenuation rate among the imaging units changed to the first sensitivity. A second mode that is a sensitivity that overlaps the graph of the image captured by the imaging unit corresponding to the filter with the highest attenuation rate among the imaging units changed to the second sensitivity;
The imaging unit corresponding to the filter of the first group is changed to the third sensitivity by the sensitivity changing unit, and the imaging unit corresponding to the filter of the second group is changed to the fourth sensitivity having a higher sensitivity than the third sensitivity. The third sensitivity and the fourth sensitivity are all the bottom portions of the graph of the image captured by the imaging unit corresponding to the filter having the highest attenuation rate among the imaging units changed to the third sensitivity. Is a third mode in which the sensitivity is such that it overlaps the graph of the image captured by the imaging unit corresponding to the filter having the highest attenuation rate among the imaging units changed to the fourth sensitivity. The imaging device according to claim 1. A distance calculation unit for obtaining a distance to the object based on an image captured by an imaging unit corresponding to at least two of the plurality of imaging optical systems;
The distance calculation unit is an imaging unit corresponding to the plurality of filters having the same predetermined attenuation rate set in advance, and a plurality of images captured by the imaging unit changed to the same sensitivity by the sensitivity changing unit The imaging apparatus according to claim 1, wherein the corresponding point search unit calculates parallax using an image, and calculates a distance to the object based on the calculated parallax. It further comprises setting input means for setting the mode,
The said control part makes the said sensitivity change part change the sensitivity of these image pick-up parts according to the mode set by the said setting input means. Imaging device. The control unit determines a mode based on images captured by the plurality of imaging units, and causes the sensitivity changing unit to change the sensitivity of the plurality of imaging units according to the determined mode. The imaging device according to any one of claims 1 to 4. The imaging unit has a solid-state imaging device,
The sensitivity changing unit changes the sensitivity of the plurality of imaging units by changing a storage time of a solid-state imaging element included in each of the plurality of imaging units. The imaging device according to item. A compound eye having a plurality of imaging optical systems arranged in an integrated manner and a plurality of imaging units corresponding to the plurality of imaging optical systems and imaging optical images of objects formed by the plurality of imaging optical systems, respectively. A plurality of filters corresponding respectively to the camera and the plurality of imaging optical systems, each having a plurality of filters that attenuate and transmit incident light at different preset predetermined attenuation rates; Among the filters, at least two filters are imaging methods used in an imaging device including the optical filter unit having the same predetermined attenuation rate set in advance,
A sensitivity change step of changing the sensitivity of each of the plurality of imaging units;
A corresponding point search step for calculating parallax between the plurality of images by searching for corresponding points using images captured by the plurality of imaging units;
The imaging unit corresponding to the plurality of filters having the same preset predetermined attenuation rate, and using a plurality of images captured by the imaging unit changed to the same sensitivity by the sensitivity changing step, Image creation in which a parallax is calculated in the corresponding point search step, the object is aligned using the calculated parallax, and images captured by the plurality of imaging units are combined to create a single image Steps,
The plurality of modes having different luminance ranges, the sensitivity of the plurality of imaging units is changed in the sensitivity changing step according to the mode, and the image creation is performed using images captured by the plurality of imaging units And a control step of creating an image of a luminance range corresponding to each mode in the step.