JP6776776B2 - Imaging equipment, display equipment, electronic devices, imaging methods, programs, imaging systems, display systems and image processing equipment - Google Patents

Description

The present invention relates to an imaging device, a display device, an electronic device, an imaging method, a program, an imaging system, a display system, and an image processing device.

Conventionally, in an electronic device including a plurality of camera modules, there is known one that generates composite image data by synthesizing image data generated by each camera module (for example, Patent Document 1). In some cases, it was not possible to generate an image suitably.

Japanese Unexamined Patent Publication No. 2016-99598

According to the first aspect, the image pickup apparatus includes a first image sensor that outputs a first image signal in which the subject is imaged in the first exposure time, and a second image signal in which the subject is imaged in the second exposure time. A second image sensor that outputs the image, an amplification unit that amplifies the first image signal and the second image signal with different gains, and first image data based on the first image signal amplified by the amplification unit. A control unit that synthesizes the second image data based on the second image signal amplified by the amplification unit and generates the composite image data is provided, and the exposure start of imaging at the first exposure time and the first At least one of the exposure ends of the imaging in the one exposure time is performed during the imaging of the second exposure time, and the second image sensor is the next second from the exposure end of the imaging in the second exposure time. Before the start of exposure for imaging with an exposure time, an image is taken with a third exposure time shorter than the second exposure time and a third image signal is output, and the image based on the third image signal is the composite image data. reproduction stopping still appears when it is set during reproduction of the moving image based on.
According to a second aspect, the display device includes an image pickup device of the first aspect, and the display unit.
According to the third aspect, the electronic device includes the imaging device of the first aspect.
According to the fourth aspect, the imaging method outputs a first image signal in which the subject is imaged in the first exposure time, outputs a second image signal in which the subject is imaged in the second exposure time, and is described above. The first image signal and the second image signal are amplified with different gains, and the first image data based on the amplified first image signal and the second image data based on the amplified second image signal are obtained. Combining, generating composite image data, at least one of the start of exposure for imaging at the first exposure time and the end of exposure for imaging at the first exposure time is performed during the imaging at the second exposure time. The second image pickup element takes an image in a third exposure time shorter than the second exposure time between the end of the exposure of the image pickup in the second exposure time and the start of the exposure of the next image pickup in the second exposure time. It was carried out to output the third image signal, an image based on the third image signal, the stop of the reproduced during reproduction of the moving image based on the synthesized image data that is displayed when it is set.
According to the fifth aspect, it is a program to be executed by a computer, that is, a process of outputting a first image signal in which the subject is imaged in the first exposure time, and a process in which the subject is imaged in the second exposure time. The process of outputting the second image signal, the process of amplifying the first image signal and the second image signal with different gains, and the first image data based on the amplified first image signal were amplified. At least the process of synthesizing the second image data based on the second image signal to generate the composite image data, the start of exposure for imaging at the first exposure time, and the end of exposure for imaging at the first exposure time. One is the second exposure time between the process performed during the imaging of the second exposure time and the period from the end of the exposure of the imaging in the second exposure time to the start of the exposure of the imaging in the next second exposure time. A process of taking an image with a shorter third exposure time and outputting a third image signal, and stopping playback of an image based on the third image signal during playback of a moving image based on the composite image data are set. Let the computer perform the process to be displayed in case .
According to the sixth aspect, the image pickup apparatus includes a first image sensor that outputs a first image signal in which the subject is imaged in the first exposure time, and a second image signal in which the subject is imaged in the second exposure time. A second image sensor that outputs the image, an amplification unit that amplifies the first image signal and the second image signal with different gains, a first communication unit that communicates with an external electronic device, and the first gain amplification. The external electronic device obtains the first image data based on the first image signal whose gain is amplified by the unit and the second image data based on the second image signal whose gain is amplified by the second gain amplification unit. Is provided with a first control unit that transmits via the first communication unit, and at least one of the start of exposure for imaging during the first exposure time and the end of exposure for imaging during the first exposure time is the second. It is performed during the imaging of the exposure time, and the second image sensor has the second exposure time between the end of the exposure of the image pickup in the second exposure time and the start of the exposure of the image pickup in the next second exposure time. An image is taken with a shorter third exposure time and a third image signal is output, and the image based on the third image signal is a moving image based on the composite image data obtained by synthesizing the first image data and the second image data. stop playback during playback of the image still appears when it is set.
According to the seventh aspect, the imaging system includes the imaging device according to the sixth aspect, the second communication unit that receives the first image data and the second image data, the first image data, and the first image data. It is provided with an image processing device including a second control unit that synthesizes the two image data and generates the composite image data.
According to an eighth aspect, the display system includes an imaging system of the seventh aspect, a display unit for displaying the synthesized image based on the synthesized image data.
According to the ninth aspect, the image processing device includes a first image sensor that outputs a first image signal in which the subject is imaged in the first exposure time, and a second image in which the subject is imaged in the second exposure time. It has a second image sensor that outputs a signal, and an amplification unit that amplifies the first image signal and the second image signal with different gains, and starts exposure for imaging at the first exposure time and the first image sensor. At least one of the exposure ends of the imaging at the exposure time is performed during the imaging of the second exposure time, and the second image sensor is subjected to the next second exposure from the exposure end of the imaging at the second exposure time. until exposure start of the imaging time, from the shorter than the second exposure time third exposure time imaging device you output the third image signal for imaging, the gain is amplified by the first gain amplifier unit A communication unit that receives the first image data based on the first image signal and the second image data based on the second image signal whose gain is amplified by the second gain amplification unit, the first image data, and the above. A control unit that synthesizes the second image data and generates the composite image data is provided , and the image based on the third image signal is set to stop playing during the reproduction of the moving image based on the composite image data. that it is displayed in the case.
According to a tenth aspect, the display device includes an image processing apparatus according to a ninth aspect, a display unit for displaying an image based on the synthesized image data.

It is a figure which shows an example of the appearance of the image pickup apparatus which concerns on 1st Embodiment. It is a block diagram which shows an example of the main part composition of the image pickup apparatus which concerns on 1st Embodiment. It is a timing chart which shows typically the relationship between the exposure time by the 1st image sensor and the 2nd image sensor, and a frame rate. It is a flowchart explaining operation at the time of taking a moving image of the image pickup apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the structure of the image pickup apparatus which concerns on the modification 1 of the 1st Embodiment. It is a block diagram which shows an example of the main part structure of the image pickup apparatus which concerns on the modification 2 of the 1st Embodiment. It is a figure which shows typically the appearance of the image pickup apparatus which concerns on another example of the modification 2 of the 1st Embodiment. It is a figure which shows typically the main part structure of the image pickup apparatus which concerns on another example of the modification 2 of the 1st Embodiment. 6 is a timing chart schematically showing the relationship between the exposure time of the first image sensor and the exposure time of the second image sensor in the second embodiment. It is a flowchart for demonstrating operation at the time of moving image reproduction by the image pickup apparatus 1 in the 2nd Embodiment. 6 is a timing chart schematically showing the relationship between the exposure time of the first image sensor and the second image sensor and the frame rate in the second modification 1. In the first modification of the second embodiment, it is a timing chart schematically showing the relationship between the image reproduced and displayed on the display, the exposure time of the base image data, and the exposure time of the highlight image data. It is a figure which shows typically the structure of the image pickup apparatus 1 which concerns on a modification.

<First Embodiment>
The image pickup apparatus according to the first embodiment will be described with reference to the drawings.
1A and 1B are views illustrating the appearance of the image pickup apparatus 1, FIG. 1A is a front perspective view, and FIG. 1B is a rear perspective view. FIG. 2 is a block diagram illustrating a configuration of a main part of the image pickup apparatus 1 shown in FIG.
As shown in FIG. 1, a first imaging unit 3 and a second imaging unit 4 are arranged in front of the main body 2 of the imaging device 1. A display 5 is arranged on the back surface of the main body 2 of the image pickup apparatus 1.
For convenience of explanation, the coordinate system including the X-axis, the Y-axis, and the Z-axis is set for the image pickup apparatus 1 as shown in the figure. The coordinate system will be described as a Cartesian coordinate system including an X-axis, a Y-axis, and a Z-axis, but the description is not limited to this, and a polar coordinate system or a cylindrical coordinate system may be adopted. That is, the X-axis is set in the long side direction of the rectangular surface of the main body 2 of the image pickup device 1, the Y-axis is set in the short side direction of the rectangular surface of the main body 2 of the image pickup device 1, and the Z-axis is the rectangular surface. It is set in the direction perpendicular to (that is, the optical axis of the first imaging unit 3 and the second imaging unit 4 described later).

The first imaging unit 3 and the second imaging unit 4 are arranged along the X direction, for example, as shown in FIG. The first imaging unit 3 and the second imaging unit 4 may be arranged along the Y direction, or may be arranged at a predetermined angle with respect to the X direction or the Y direction.
As shown in FIG. 2, the first imaging unit 3 passes through an imaging lens 31 including various optical lens groups such as an imaging lens and a focus adjustment lens, a first imaging element 32, an aperture 33, and an optical member. A density filter (ND filter) 34 for reducing the amount of light to be emitted is provided. In the description here, the optical member will be described as a density filter (ND filter) 34, but the diaphragm 33 and the density filter (ND filter) 34 may be included as the optical member. Alternatively, the aperture 33 may be used as an optical member without arranging the density filter (ND filter) 34.
In FIG. 1, the image pickup lens 31 is represented by one lens. The first image sensor 32 is composed of a large number of photoelectric conversion elements (pixels) such as CMOS and CCD arranged in a matrix on the image pickup surface and various circuits for controlling the drive of the pixels. The first image sensor 32 is driven according to the control of the control unit 6 (see FIG. 2) described later, captures a subject image input through the image pickup lens 31, and outputs an image signal obtained by capturing the image. The first image sensor 32 includes an amplification unit 321 and is controlled by a control unit 6 described later to amplify an image signal with a predetermined gain (amplification rate). The aperture 33 regulates the light from the subject that passes through the image pickup lens 31 and heads toward the first image sensor 32. The density filter 34 functions as a light amount reducing unit that reduces the amount of light from the subject passing through the image pickup lens 31 by a predetermined amount in a predetermined wavelength band. The density filter 34 is arranged between the image pickup lens 31 and the first image pickup element 32. When the image pickup lens 31 is a lens module composed of a plurality of lenses, it may be arranged between the lenses in the lens module. The details of the transmittance of the density filter 34 will be described later.

The second image pickup unit 4 includes an image pickup lens 41, a second image pickup element 42, and an aperture 43, similarly to the first image pickup unit 3. The second image sensor 42 captures a subject image that has passed through the image pickup lens 41, and outputs an image signal obtained by capturing the image. Similar to the first image sensor 32, the second image sensor 42 is composed of a large number of photoelectric conversion elements (pixels) such as CMOS and CCD arranged in a matrix and various circuits for controlling the drive of the pixels. The second image sensor 42 is driven according to the control of the control unit 6 to image the subject image input through the image pickup lens 41, and outputs the image signal obtained by the image. The second image sensor 42 includes an amplification unit 421 and is controlled by a control unit 6 described later to amplify an image signal with a predetermined gain (amplification rate). Like the diaphragm 33, the diaphragm 43 regulates the light from the subject passing through the image pickup lens 41 and heading toward the second image sensor 42.

The first image sensor 32 and the second image sensor 42 are provided with a memory (not shown) for each pixel, so that image pickup can be performed by a global shutter operation. That is, each of the first image sensor 32 and the second image sensor 42 collectively transfers or generates charges generated by photoelectric conversion in the entire region of the image pickup surface to a memory (not shown) at the same timing. After converting the electric charge into an electric signal, it is collectively transferred to a memory (not shown) at the same timing. By the global shutter operation, it is possible to align the exposure start and end times for all the pixels of the first image sensor 32 and to align the exposure start and end times for all the pixels of the second image sensor 42.
The first image sensor 32 and the second image sensor 42 are not limited to those equipped with a global shutter. For example, even if the first image sensor 32 and the second image sensor 42 perform a so-called rolling shutter operation in which charge accumulation and image signal output are sequentially performed for each part of the image pickup surface (for example, a predetermined line). Good.

R (red), G (green), and B (blue) first color filters 322 are provided on the imaging surface of the first image sensor 32 so as to correspond to pixel positions. The second image sensor 42 is provided with R (red), G (green), and B (blue) second color filters 422 corresponding to the pixel positions, respectively. Since the first image sensor 32 and the second image sensor 42 image the subject image through the first color filter 322 and the second color filter 422, respectively, the image signals output from the first image sensor 32 and the second image sensor 42 are It has color information of RGB color system.
The focus detection pixels may be arranged in a part or the entire area of the image pickup surface of the first image sensor 32 and the second image sensor 42. In this case, a signal obtained by adding signals corresponding to the pair of optical images output from the focus detection pixels is used as the image signal.

The display 5 is composed of, for example, a liquid crystal display, an organic EL display, or the like, and has a plurality of display pixel arrays arranged in a two-dimensional manner. The display 5 is driven based on the control of the control unit 6 described later, and displays an image corresponding to the image data generated by the control unit 6. On the display 5, a menu screen for setting various operations of the image pickup apparatus 1 is displayed.

Next, the configuration of the control system of the image pickup apparatus 1 will be described with reference to FIG. The image pickup apparatus 1 includes a control unit 6 and an operation detection unit 7. The control unit 6 has a CPU, ROM, RAM, and the like (not shown), and is an arithmetic circuit that controls each component of the image pickup apparatus 1 and executes various data processes based on a control program. The control program is stored in a storage unit 65 described later in the control unit 6.

The control unit 6 includes an image pickup condition setting unit 61, an image sensor control unit 62, an image processing unit 63, a display control unit 64, and a storage unit 65. The imaging condition setting unit 61 sets imaging conditions for performing imaging, such as exposure time (shutter speed), aperture, and ISO sensitivity, for each of the first imaging unit 3 and the second imaging unit 4. The imaging condition setting unit 61 sets the ISO sensitivity of the second imaging unit 4, that is, the second gain (amplification factor) to a larger value than the ISO sensitivity, that is, the first gain (amplification factor) of the first imaging unit 3. To do. The image sensor control unit 62 controls the operations of the first image sensor 32 and the second image sensor 42 so that image pickup is performed under the image pickup conditions set by the image pickup condition setting unit. In the present embodiment, as shown in FIG. 2, the image sensor control unit 62 operates the first image sensor control unit 621 that controls the operation of the first image sensor 32 and the operation of the second image sensor 42. Although the description will be made assuming that the second image sensor control unit 622 to be controlled is provided as a function, the present invention is not limited to this example. That is, the control unit 6 may not include the image sensor control unit 62, but the control unit 6 may include the first image sensor control unit 621 and the second image sensor control unit 622.

The image processing unit 63 generates the first image data by performing various image processing after the first image signal output from the first imaging element 32 is amplified by the amplification unit 321 with the first gain. The image processing unit 63 generates the second image data by performing various image processing after the second image signal output from the second image sensor 42 is amplified by the amplification unit 421 with the second gain. The image processing unit 63 synthesizes the generated first image data and the second image data, and performs high dynamic range (HDR) processing to enhance the reproducibility of the brightness of the subject (hereinafter, the composite image data). HDR image data) is generated. As an example of generating the composite image data, for example, the image processing unit 63 generates the composite image data by adding the brightness of the first image data and the second image data and then averaging them. Alternatively, the image processing unit 63 corrects the magnitude of the brightness based on the first image data with respect to the low brightness region of the second image data amplified by the second gain, which is lower than the first gain. , The correction may be performed so as to brighten the low-luminance region of the second image data. The display control unit 64 causes the display 5 to display various image data generated by the image processing unit 63, that is, images of the first image data, the second image data, and the HDR image data. The storage unit 65 is, for example, a non-volatile memory and stores HDR image data.
The details of the image pickup condition setting unit 61, the image sensor control unit 62, the image processing unit 63, and the display control unit 64 described above will be described later.

The operation detection unit 7 is composed of, for example, a touch panel or the like that is laminated on the display unit 5, and when a tap operation in which a user's finger or the like touches the screen of the display unit 5 is performed, the coordinates are output to the control unit 6. To do. When the user's operation is detected by the operation detection unit 7, for example, a shooting instruction to the image pickup device 1, a selection instruction or a determination instruction such as various settings, and a reproduction mode for reproducing and displaying an image on the display 5. And an instruction to switch between the still image shooting mode and the moving image shooting mode for shooting by the imaging device 1. In this case, the control unit 6 causes the display 5 to display an icon or the like corresponding to each of the above instructions. When the user taps the icon displayed on the display 5 according to the desired instruction, the control unit 6 gives which instruction based on the coordinates detected by the operation detection unit 7. Is determined, and an instruction signal is output to each part according to the content of the instruction.

The operation detection unit 7 is not limited to a touch panel or the like, and various configurations that can be used as a user interface can be used. For example, the operation by the user is performed by pressing a button or the like, and the operation detection unit 7 may detect the user's button operation. Alternatively, the operation detection unit 7 may detect a user's action, such as waving a hand or a predetermined gesture. In this case, the operation detection unit 7 can be configured by an imaging device. Further, the operation detection unit 7 may be configured by a sound collecting device such as a microphone that detects the user's voice.

The image pickup device 1 of the present embodiment having the above-described configuration generates HDR-processed moving image data based on the image signals output from the first image pickup device 32 and the second image pickup device 42.
The image pickup apparatus 1 obtains dark image data in which overexposure is suppressed by amplifying the first image signal output by the first image sensor 32 of the first image pickup unit 3 with the first gain. Generate as. As described above, since the first gain is set to a value smaller than the second gain by the image pickup condition setting unit 61, the first image signal from the first image sensor 32 is amplified so as to be underexposed by the first gain. Then, the first image data with improved reproducibility of the bright part of the subject, that is, the high-luminance region is generated. The image pickup apparatus 1 obtains bright image data in which blackout is suppressed by amplifying the second image signal output by the second image sensor 42 of the second image pickup unit 4 with the second gain. Generate as. Since the second gain is set to a value larger than the first gain by the image pickup condition setting unit 61, the second image signal from the second image pickup element 42 is amplified so as to be overexposed at the second gain. As a result, the second image data with improved reproducibility in the dark part of the subject, that is, the low-luminance region is generated. The image pickup apparatus 1 is based on composite image data based on the first image data which is image data with improved reproducibility in a high brightness region and the second image data which is image data with improved reproducibility in a low brightness region. Is generated, and a moving image is generated based on this composite image data.

As described above, the image pickup condition setting unit 61 sets the first gain of the first image sensor 32, but it may not be possible to suppress the occurrence of overexposure in the subject depending on the image pickup environment or the like. Therefore, in the image pickup apparatus 1 of the present embodiment, the first image pickup unit 3 includes a density filter 34. Hereinafter, how much the density filter 34 needs to reduce the transmittance of light from the subject will be described with reference to specific examples.

An example is taken when an image is taken in a bright cloudy environment where the exposure value (EV value) is about 13. In this case, when the ISO sensitivity is set to 100 and the aperture value (F value) is set to 2.8 and the exposure time (shutter speed) is set to 1/1000 s, the imaging conditions are approximately appropriate. However, as will be described later, when a moving image is captured at 60 fps, the exposure time of 1/1000 s is shorter than the time of 1 frame of 1/60 s. Therefore, in order to reduce the shutter speed to about 1 / 120s, that is, to shorten the exposure time, the density filter 34 needs to reduce the amount of light from the subject to 1/8, that is, the transmittance is 12.5%. is there.

An example is taken when an exposure value of about 16 is taken, for example, a snow scene in fine weather. In this case, when the ISO sensitivity is set to 100 and the aperture value (F value) is set to 2.8 and the exposure time (shutter speed) is set to 1/8000 s, the imaging conditions are approximately appropriate. Also in this case, when a moving image is captured at 60 fps, the exposure time of 1/8000 s is shorter than the time of 1 frame of 1/60 s. Therefore, in order to reduce the shutter speed to about 1 / 120s, that is, to shorten the exposure time, the density filter 34 needs to set the amount of light from the subject to 1/64, that is, the transmittance of 1.56%. is there.

As described in the above example, the imaging device 1 can be provided with a density filter 34 having a transmittance of 1.5% so that imaging can be performed at a shutter speed of about 1/120 s even if the EV value is about 16. .. However, in an environment where the EV value is smaller than 16 (for example, EV value 13 or the like), the image data imaged and generated by the first image sensor 32 at ISO sensitivity 100 becomes a dark image. In such a case, the imaging condition setting unit 61 may set the ISO sensitivity to a value larger than 100. For example, when the EV value is 13 and the shutter speed is 1 / 125s, the imaging condition setting unit 61 may set the ISO sensitivity to 800.
In the above example, the ISO sensitivity is 100, but the value may be even smaller. For example, in the case of ISO sensitivity 50, if the aperture value (F value) is set to 2.8 and the exposure time (shutter speed) is set to 1 / 8000s, the EV value becomes 15. Compared with the EV value 16 with an ISO sensitivity of 100, an aperture value (F value) of 2.8, and an exposure time (shutter speed) of 1 / 8000s, the EV value is one smaller. That is, the required transmittance of the density filter 34 differs depending on the lowest ISO sensitivity. In this case, in order to reduce the shutter speed to about 1 / 120s, the transmittance of the density filter 34 is about 1/32, that is, the transmittance is changed to about 3.0%. That is, as the ISO sensitivity can be set lower, the transmittance of the density filter 34 can be increased (transmissive). Of course, as the settable ISO sensitivity can be set higher, the transmittance of the density filter 34 needs to be set lower.
Similarly, the EV value changes depending on the aperture value (F value), and as the aperture value (F value) increases, the EV value decreases, so that the transmittance of the density filter 34 can be increased (to be transparent). Become). Of course, as the aperture value (F value) decreases, the EV value increases, so it is necessary to lower the transmittance of the density filter 34.

Hereinafter, the image pickup condition of the first image sensor 32 is set to, for example, an exposure time of 1/120 s and an ISO sensitivity of 100 so that the image pickup condition of the first image sensor 32 is suppressed by the image pickup condition setting unit 61. The image pickup process of the operation image when the exposure time is set to 1 / 120s and the ISO sensitivity is set to 12800 will be described so as to suppress the occurrence of black crushing.
In the following description, it is assumed that the frame rate of the moving image is 60 fps. That is, the image pickup condition setting unit 61 sets the exposure time of the first image pickup element 32 and the second image pickup element 42 to be 1/2 of the time 1 / 60s of one frame at a frame rate of 60 fps. The image pickup condition setting unit 61 determines that the movement of the subject during video reproduction becomes unnaturally discontinuous with respect to the exposure time of the first image pickup element 32 and the second image pickup element 42 based on the frame rate of the moving image. The exposure time may be set to a degree that can be suppressed. In this case, the image pickup condition setting unit 61 preferably sets the time of 1/4 to 1/1 of the frame rate as the exposure time of the first image pickup element 32. When the exposure time is set to be longer than 1/2 of the frame rate by the image pickup condition setting unit 61, the first image data captured by the first image sensor 32 and the second image sensor 42 and the first image data In the second image data, the blurring of the subject moving relative to the image sensor 1 becomes large. In this way, the moving image generated by setting a long exposure time in each frame can be reproduced as a moving image having a sense of speed and a sense of dynamism.
Further, in the following description, an image in which the occurrence of overexposure is suppressed and the reproducibility of the high-brightness region of the subject is improved, that is, the first image data is referred to as highlight image data, and the occurrence of blackout is suppressed and the low brightness of the subject is suppressed. Image data with improved area reproducibility, that is, second image data is called base image data.

The image sensor control unit 62 includes the first image sensor 32 and the second image sensor 42 under the image pickup conditions for the first image sensor 32 and the second image sensor 42 set as described above by the image sensor setting unit 61. To expose. That is, the first image sensor control unit 621 starts and ends charge accumulation in the first image sensor 32, that is, so that the exposure is performed at the exposure time (1 / 120 s) set by the image pickup condition setting unit 61. Outputs a signal instructing the start and end of exposure. The first image sensor control unit 621 sets the amplification factor by the amplifier circuit (not shown) of the first image sensor 32 so that the ISO sensitivity (100) set by the image pickup condition setting unit 61 is obtained. The second image sensor control unit 622 starts and ends charge accumulation in the second image sensor 42, that is, starts exposure so that the exposure is performed at the exposure time (1 / 120 s) set by the image sensor setting unit 61. And outputs a signal instructing the end of exposure. The second image sensor control unit 622 sets the amplification factor by the amplifier circuit (not shown) of the second image sensor 42 so that the ISO sensitivity (12800) set by the image pickup condition setting unit 61 is obtained.

FIG. 3 is a timing chart schematically showing the relationship between the exposure time by the first image sensor 32 and the second image sensor 42 and the frame rate, and the horizontal axis is time t.
FIG. 3A shows the relationship between the exposure time by the first imaging element 32 and the frame rate. The first image sensor control unit 621 instructs the first image sensor 32 to start exposure at time t1-1, and the first image pickup is performed at time t1-2 when 1/120 s of the exposure time has elapsed from time t1-1. The element 32 is instructed to end the exposure. The image signal imaged and output by the first image sensor 32 during the time t1-1 to t1-2 is amplified by the amplification unit 321 and the highlight image data of the first frame is generated by the image processing unit 63. Will be done. At time t2-1, which is 1 / 60s after time t1-1, the first image sensor control unit 621 instructs the first image sensor 32 to start exposure of the highlight image data of the second frame, and thereafter. , The same process is repeated until the imaging of the moving image is completed.
The highlight image data generated as described above is stored in the storage unit 65.

FIG. 3B shows the relationship between the exposure time by the second image sensor 42 and the frame rate. The second image sensor control unit 622 instructs the second image sensor 42 to start exposure at time t1-1, and the second image pickup is performed at time t1-2 when 1/120 s of the exposure time has elapsed from time t1-1. The element 42 is instructed to end the exposure. The image signal imaged and output by the second image sensor 42 during the time t1-1 to t1-2 is amplified by the amplification unit 421, and the base image data of the first frame is generated by the image processing unit 63. To. At time t2-1, which is 1 / 60s after time t1-1, the second image sensor control unit 622 instructs the second image sensor 42 to start exposure of the base image data of the second frame. The same process is repeated until the imaging of the moving image is completed.
The base image data generated as described above is stored in the storage unit 65.

In the above description, the case where the first image sensor 32 and the second image sensor 42 perform imaging during the time t1-1 to t1-2 is given as an example, but the present invention is not limited to this. For example, the exposure start of the first image sensor 32 may be earlier than the exposure start of the second image sensor 42 by a predetermined time, or may be later than the exposure start of the second image sensor 42 by a predetermined time. That is, the image sensor control unit 62 starts exposure of the first image sensor 32 and the second image sensor 42 so that the exposure time of the first image sensor 32 and the exposure time of the second image sensor 42 partially overlap. You may control the time of. That is, even if the image sensor control unit 62 controls that at least one of the start of imaging in the first exposure time and the end of imaging in the first exposure time is performed during the imaging in the second exposure time. Good. However, the above-mentioned predetermined time is preferably shorter than 1 / 120s, which is the exposure time of the first image sensor 32 and the second image sensor 42.
In the above description, the configuration includes the density filter 34, which is an optical member, but it is not always necessary to provide the configuration. That is, at least one of the start of imaging in the first exposure time by the first image sensor 32 and the end of imaging in the imaging in the first exposure time is during the imaging of the second exposure time by the second image sensor 42. The image sensor control unit 62 controls, and the first image signal and the second image signal imaged by the first image sensor 32 and the second image sensor 42 are amplified by the amplification unit 421 with different gains. It may be only the control that does.

The image processing unit 63 synthesizes the generated highlight image data and the base image data by HDR processing to generate HDR image data. In this case, for example, the image processing unit 63 increases the weighting of the highlight image data for the high-luminance region on the image and increases the weighting of the base image data for the low-luminance region to perform the composition. By this composition, in the HDR image data, a low-brightness or medium-brightness subject is clearly reproduced by the base image data captured with a slight overexposure. In the HDR image data, a high-brightness subject that may cause overexposure is clearly reproduced by the highlight image data captured by underexposure. As a result, the HDR image data becomes an image having a wide dynamic range as a whole.

The image processing unit 63 also generates HDR image data by synthesizing the highlight image data and the base image data generated in the same manner for the second and subsequent frames. As a result, HDR image data having a frame rate of 60 fsp and exposure times t1-1 to t1-2 is generated as moving image data. The generated HDR image data, that is, moving image data is stored in the storage unit 65.

The imaging operation of the imaging device 1 according to the first embodiment will be described with reference to the flowchart shown in FIG. Each process shown in the flowchart of FIG. 4 is performed by executing a program in the control unit 6. This program is stored in the storage unit 65, and when it is detected that the operation detecting unit 7 has performed an operation instructing the imaging of a moving image, the control unit 6 activates and executes the program.
In step S1, the image pickup condition setting unit 61 sets the image pickup conditions of the first image pickup element 32 and the second image pickup element 42, and proceeds to step S2. In step S2, the image pickup element control unit 62 instructs the first image pickup element 32 and the second image pickup element 42 to image one frame based on the set imaging conditions, and proceeds to step S3.

In step S3, the image processing unit 63 generates highlight image data after the image signal output from the first image sensor 32 is amplified, and after the image signal is amplified from the second image sensor 42, the image processing unit 63 generates the highlight image data. The base image data is generated and the process proceeds to step S4. In step S4, the image processing unit 63 synthesizes the highlight image data and the base image data to generate HDR image data, and proceeds to step S5.

In step S5, the control unit 6 stores the generated base image data, highlight image data, and HDR image data in the storage unit 65, and proceeds to step S6. In step S6, it is determined whether or not the imaging of the moving image is completed. When the operation detection unit 7 detects that the imaging of the moving image is finished, that is, the operation of ending the imaging of the moving image is detected by the user, the operation detection unit 7 determines affirmatively and ends the process. If the imaging of the moving image is not completed, that is, if the operation of ending the imaging of the moving image by the user is not performed, the negative determination in step S6 is made and the process returns to step S2.

When the operation detection unit 7 detects that the user has performed an operation instructing the playback of the moving image, the display control unit 64 stores the HDR image data generated as described above and stored in the storage unit 65. It is read from 65 and is reproduced and displayed as a moving image on the display 5.

A case where the frame rate at the time of playing back a moving image is increased, for example, a case where the frame rate is changed from 60 fps to 240 fps at 4 times speed will be described. In this case, the display control unit 64 reduces the interval between the HDR image data of a certain frame and the HDR image data of the next frame, that is, the time (1 / 120s) from time t1-2 to t2-1 to 1/4. Double the time. As a result, the interval between one frame and the next frame is shortened, so that the movement of the subject can be continuously reproduced.

A slow playback in which the frame rate when playing a moving image is slowed down, for example, a case where the speed is changed to 1/4 times speed will be described. In this case, the display control unit 64 quadruples the time (1 / 120s) from time t1-2 to t2-1, which is the interval between the HDR image data of a certain frame and the HDR image data of the next frame. Make time. In this case, since the interval between one frame and the next frame becomes long, the movement of the subject becomes discontinuous, and an unnatural moving image is reproduced. Therefore, as described above, the imaging condition setting unit 61 may set a time of 1/4 to 1/1 with respect to the frame rate as the exposure time.

In the first embodiment, the density filter 34 reduces the amount of light from the subject and causes the light to be incident on the first image sensor 32. The image processing unit 63 generates highlight image data after the image signal from the first image pickup element 32 is amplified, and generates base image data after the image signal from the second image pickup element 42 is amplified. , Highlight image data and base image data are combined to generate HDR image data. As a result, it is possible to generate moving image data having a large dynamic range in which the gradation in the high-luminance region and the gradation in the low-luminance region of the subject are reproduced.

In the first embodiment, the exposure time of the first image sensor 32 is longer than half the time of one frame at the frame rate. As a result, the interval between image data generated by imaging a moving subject (that is, time t1-2 to t2-1) can be shortened, so that the movement of the subject becomes discontinuous when reproduced as a moving image. It is possible to prevent it from becoming unnatural.

In the first embodiment, the exposure time of the second image sensor 42 is longer than half the time of one frame at the frame rate. As a result, the interval between image data generated by imaging a moving subject (that is, time t1-2 to t2-1) can be shortened, so that the movement of the subject becomes discontinuous when reproduced as a moving image. It is possible to prevent it from becoming unnatural.

In the first embodiment, the exposure time of the first image sensor 32 and the exposure time of the second image sensor 42 are the same time. As a result, during the HDR processing, the positions of the subjects in the highlight image data and the base image data are the same, so that it is not necessary to perform processing such as alignment at the time of compositing, and the processing load can be reduced.

<Modification 1 of the first embodiment>
In the above-described first embodiment, the case where the first imaging unit 3 includes the density filter 34 has been described as an example, but the present invention is not limited to this example. Examples 1 to 4 will be described below.

<Example 1>
The first imaging unit 3 does not include a density filter 34, or includes an imaging lens 31 that is coated with the density filter 34 to reduce the transmittance of light from the subject. As a result, the amount of light incident on the first image sensor 32 is reduced, so that the highlight image data generated by the image processing unit 63 becomes a dark image, and it is possible to generate an image in which the occurrence of overexposure is suppressed.
When coating is applied to reflect light to reduce the transmittance, it is preferable to coat the lens closest to the object side (subject side) among the various lenses constituting the image pickup lens 31. As a result, it is possible to suppress the generation of ghosts on the image due to the internal reflection caused by coating the internal lens.
The image pickup apparatus 1 of Example 1 of the modification 1 having the above configuration performs the same processing as the image pickup apparatus 1 of the first embodiment shown in FIG. 4 to generate moving image data.
In Example 1 of Modification 1 of the first embodiment, the amount of light incident on the first image sensor 32 from the subject is reduced by coating the image pickup lens 31. As a result, the density filter 34 does not have to be arranged in the first imaging unit 3, which contributes to miniaturization of the apparatus.

<Example 2>
The first image pickup unit 3 has a first image pickup element 32 as an optical member, which includes a color filter having a low light transmittance instead of the density filter 34. In this case, the first color filter 322 of the first image sensor 32 has a lower light transmittance than the second color filter 422 of the second image sensor 42. As a result, the pixel values of the R, G, and B color components of the image signal output from the first image sensor 32 are the pixels of the R, G, and B color components of the image signal output from the second image sensor 42. It becomes smaller than the value, and the highlight image data generated by the image processing unit 63 becomes a dark image.
The image pickup device 1 of the second embodiment of the modification 1 having the above configuration performs the same processing as the image pickup device 1 of the first embodiment shown in FIG. 4 to generate moving image data.

In Example 2 of Modification 1 of the first embodiment, the light transmittance of the first color filter 322 of the first image sensor 32 is the light transmittance of the second color filter 422 of the second image sensor 42. Lower than the rate. As a result, even when the density filter 34 is not provided, it is possible to generate an image in which the occurrence of overexposure is suppressed based on the image signal from the first image sensor 32, which contributes to miniaturization of the apparatus. ..

<Example 3>
With reference to FIG. 5, the first image pickup unit 3 includes a density filter having a configuration capable of changing the transmittance of light from the subject in the optical path between the image pickup lens 31 and the first image pickup element 32. An example will be described. The density filter has a plurality of regions having different transmittances on a plane intersecting the optical axis of the image pickup lens 31, and is controlled by a control unit to switch the region for transmitting light from the subject to transmit the light transmittance. To change.
FIG. 5 schematically shows an example of the configuration of the density filter 35 in Example 3 and an example of the configuration of a main part of the image pickup apparatus 1. FIG. 5A schematically shows the structure of the density filter 35 in the XY plane, that is, in the plane orthogonal to the optical axis of the image pickup lens 31. The density filter 35 has, for example, a configuration in which a first region 352 and a second region 353 are provided on a disk-shaped base member 351. Filters having different light transmittances are arranged in the first region 352 and the second region 353. For example, the transmittance of the first region 352 is 1.5% and the transmittance of the second region 353 is 12.5%. can do. The density filter 35 may be provided with three or more regions having different transmittances.

The density filter 35 can rotate on the XY plane around the center of the base member 351 so as to block the optical path between the image pickup lens 31 and the first image pickup element 32 while the first region 352 or the second region 353 is blocked. It is composed of. In this case, the base member 351 can be rotated by a rotation mechanism such as a gear (not shown).

FIG. 5B is a block diagram schematically showing a configuration of a main part of the image pickup apparatus 1 in Example 3 of Modification 1. The control unit 6 of the image pickup apparatus 1 in Example 3 of the modified example 1 includes a transmittance control unit 66 as a function. As for other configurations, the configuration included in the image pickup apparatus 1 according to the first embodiment of FIG. 2 can be applied. The transmittance control unit 66 selects one of the first region 352 and the second region 353 provided in the density filter 35 shown in FIG. 5A, and reduces the amount of light from the subject in the selected region. The base member 351 is rotated by the above-mentioned rotation mechanism. The transmittance control unit 66 selects the first region 352, for example, when imaging is performed in the above-mentioned environment where the EV value is about 16. The transmittance control unit 66 selects the second region 353, for example, when imaging is performed in the above-mentioned environment where the EV value is about 13.
As the process for generating the moving image data of the image pickup device 1 of Example 3 of the modification 1 having the above configuration, the same process as that of the image pickup device 1 of the first embodiment shown in FIG. 4 can be applied. it can. However, when the imaging conditions of the first image sensor 32 are set in step S1 of FIG. 4, the transmittance control unit 66 controls the density filter 35, and based on the selected transmittance, the first region 352. Alternatively, the second region 353 is positioned from the image pickup lens 31 to the optical path of the first image pickup element 32.

In Example 3 of Modified Example 1 of the first embodiment, the amount of reduction of light from the subject incident on the first image sensor 32 can be changed by the density filter 35, so that transmission suitable for different image pickup environments can be changed. The amount can be set, and an image in which the occurrence of overexposure is suppressed can be generated based on the image signal from the first image sensor 32.

<Example 4>
In the above description, the amount of light from the subject incident on the first image sensor 32 is reduced by locating one of the first region 352 and the second region 353 of the density filter 35 in the optical path of the image pickup lens 31. Met. However, it is possible to switch between a state in which the amount of light from the subject incident on the first image sensor 32 is reduced and a state in which the amount of light is not reduced. For example, as described above, a filter having a transmittance of 1.5% is arranged in the first region 352 of the density filter 35. A filter having a transmittance of 100% is provided in the second region 353 of the density filter 35. Alternatively, no filter is provided in the second region 353 of the density filter 35, and an opening is provided so that the transmittance is 100%.

The transmittance control unit 66 is described above so that, for example, when an imaging mode for generating HDR image data, that is, a mode for capturing a moving image is set, the first region 352 is located in the optical path of the imaging lens 31. The base member 351 is rotated by the rotation mechanism of. When a mode for capturing an image other than HDR image data is selected, the transmittance control unit 66 selects the second region 353 so that the second region 353 is located in the optical path of the imaging lens 31. The base member 351 is rotated by the above rotation mechanism.

As an example of capturing an image other than the HDR image, for example, the focal length is different between the first imaging unit 3 and the second imaging unit 4, and the image data based on the image signal from the first imaging unit 3 is used. , There is a case where the image data based on the image signal from the second imaging unit 4 is combined to generate an image in which the depth of field is changed. When generating an image in which the subject depth is changed, the image data based on the image signal from the first image sensor 32 and the image data based on the image signal from the second image sensor 42 have the same brightness. It is preferable that the image is. However, when the first image sensor 3 always includes the density filter 34 as in the case of the first embodiment (see FIG. 2), the amount of light incident on the first image sensor 32 is always reduced. It will be in the state of being. Therefore, in order to generate an image having the same brightness, it is necessary to make the imaging condition at the time of imaging by the first imaging unit 3 different from the imaging condition at the time of imaging by the second imaging unit 4. When the aperture value (F value) and the exposure time are set to be equal between the first imaging unit 3 and the second imaging unit 4, images having the same brightness are generated by making the ISO sensitivity different. .. In this case, for example, when the ISO sensitivity of the second image sensor 42 of the second image sensor 4 is set to 200, the ISO sensitivity of the first image sensor 32 of the first image sensor 3 is set to 12800. Generally, when the ISO sensitivity is high, the image quality of the generated image data tends to deteriorate due to the influence of noise and the like. Therefore, the image quality is different between the image data captured with the ISO sensitivity of 12800 and the image data captured with the ISO sensitivity of 200. Therefore, when these image data are combined, the generated composite image data is displayed. Then, a feeling of strangeness occurs.
In Example 4 of Modification 1, when an image with a changed depth of field is generated, the second region 353 having a transmittance of 100% of the density filter 35 is located in the optical path. The amount of light incident on the image sensor 32 is not reduced. As a result, it is not necessary to set the ISO sensitivity of the first image sensor 32 to a value higher than the ISO sensitivity of the second image sensor 42 and perform imaging as described above. Therefore, deterioration of the image quality of the image data based on the image signal from the first image sensor 32 can be suppressed, and even if the compositing process is performed, it is possible to suppress the occurrence of a sense of discomfort on the composite image data.

As the process for generating the moving image data of the image pickup device 1 of the example 4 of the modification 1 having the above configuration, the same process as that of the image pickup device 1 of the first embodiment shown in FIG. 4 can be applied. it can. However, when the imaging conditions of the first image sensor 32 are set in step S1 of FIG. 4, the transmittance control unit 66 controls the density filter 35, and based on the selected transmittance, the first region 352. Alternatively, the second region 353 is positioned from the image pickup lens 31 to the optical path of the first image pickup element 32.

The density filter 35 in Examples 3 and 4 described above is not limited to the one having the above configuration, and may be configured by, for example, a liquid crystal panel. In this case, the transmittance control unit 66 controls the voltage applied to the liquid crystal panel based on the transmittance determined based on the environment in which the imaging is performed. A first electrode and a second electrode are provided on the image pickup lens 31 side and the first image pickup element 32 side of the density filter 35 composed of the liquid crystal panel, respectively. By applying a voltage to the first electrode and the second electrode, the arrangement direction of the liquid crystal elements in the density filter 35 is controlled, so that the transmittance, that is, the amount of light reduction is controlled.

In Example 4 of the first embodiment, it is possible to switch between a state in which the amount of light incident on the first image sensor 32 is reduced and a state in which the amount of light is not reduced. As a result, when an image other than the HDR image data is generated, the first image sensor 32 can take an image while suppressing the reduction in the amount of light by the density filter 35. Therefore, the image generated by the plurality of image sensors The data can be used for other functional purposes.

<Modification 2 of the first embodiment>
The image pickup apparatus of the second modification of the first embodiment will be described with reference to FIG. The imaging device of the second modification includes an imaging unit different from the first imaging unit 3 and the second imaging unit 4 included in the imaging device 1 of the first embodiment shown in FIG. The details will be described below.

FIG. 6 is a block diagram schematically showing a configuration of a main part of the image pickup apparatus 1 of the second modification. The imaging device 1 of the second modification includes a third imaging unit 9, a display 5, a control unit 6, and an operation detection unit 7. The third imaging unit 9 includes an imaging lens 91 including various optical lens groups such as an imaging lens and a focus adjustment lens, a first imaging element 92, a second imaging element 93, and a half mirror 94 as an optical member. A diaphragm 95 and a density filter 96 are provided. Similar to the first image sensor 32 and the second image sensor 42 of the first embodiment, the first image sensor 92 and the second image sensor 93 are CMOS, CCD, etc. arranged in a large number in a matrix on the image pickup surface. It is composed of a photoelectric conversion element (pixel) and various circuits that control the drive of the pixel. The first image sensor 92 has the same configuration as the first image sensor 32 of the first embodiment, and includes an amplification unit 921 and a first color filter 922. The second image sensor 93 has the same configuration as the second image sensor 42 of the first embodiment, and includes an amplification unit 931 and a second color filter 932. The half mirror 94 is a spectroscopic member that transmits a part of the light flux of the subject light that has passed through the image pickup lens 91 and guides it to the first image pickup element 92, and reflects the remaining light beam and guides it to the second image pickup element 93. The density filter 96 has the same configuration as the density filter 34 and the density filter 35 of the first embodiment and its modifications, and is provided on the optical path between the half mirror 94 and the first image sensor 92.

The display 5, the control unit 6, and the operation detection unit 7 have the same configurations as those in the first embodiment. However, the first image sensor control unit 621 controls the image pickup operation of the first image sensor 92, and the second image sensor control unit 622 controls the image pickup operation of the second image sensor 93. That is, the highlight image data is generated by the image processing unit 63 based on the image signal from the first image sensor 92, and the base image data is generated by the image processing unit 63 based on the image signal from the second image sensor 93. Will be generated.

The transmittance and the reflectance of the subject light of the half mirror 94 may be set to the same value or may be set to different values. When the transmittance and the reflectance are different, it is preferable that the amount of light guided to the first image sensor 92 is set to be smaller than the amount of light guided to the second image sensor 93. In this case, in order to reduce the amount of incident light on the first image sensor 92 for generating the highlight image data of underexposure, for example, the transmittance of the half mirror 94 may be set to 40% or 30%. it can. When the transmittance and the reflectance of the half mirror 94 are different, the above-mentioned density filter 96 may not be provided.
The process performed by the image pickup device 1 of the first embodiment shown in FIG. 4 can be applied to the process for generating moving image data in the image pickup device 1 of the modification 2 having the above configuration.

In the second modification of the first embodiment, the subject light that has passed through one image pickup lens 91 is guided to the first image pickup element 92 and the second image pickup element 93. There is no displacement of the subject between the highlight image data based on the image signal from the first image sensor 92 and the base image data based on the image signal from the second image sensor 93. As a result, when the HDR image data is generated, it is not necessary to align the base image data and the highlight image data, and the load of the compositing process can be reduced.

The processes described in the first embodiment described above and the modifications 1 and 2 thereof can be applied to, for example, an imaging device capable of imaging at a wide angle (for example, 360 °) in the horizontal direction. In particular, it may be applied to a small wide-angle image pickup device (so-called wearable camera) that can be attached to a part of the body (arms, head, etc.), a bicycle, a helmet, or the like. In this case, since the range that can be imaged is wide, there is a high possibility that a high-brightness subject such as the sun and a low-brightness subject in the shadow portion are captured in the same image, so the above-mentioned HDR processing was performed. It is good to generate a video.

The configuration of the image pickup apparatus 1 capable of the above wide-angle imaging will be described with reference to FIGS. 7 and 8. 7 (a) and 7 (b) are external views of the image pickup apparatus 1, and FIG. 12 is a block diagram showing an example of a configuration of a main part of the image pickup apparatus 1.
A first image pickup in which the same configurations as those of the first image pickup unit 3 and the second image pickup unit 4 (see FIG. 2) of the first embodiment are applied to the front surface of the main body 2 on the Z-axis + side of the image pickup apparatus 1. A first imaging unit group 114 including the unit 3A and the second imaging unit 4A is provided (see FIG. 7A). A first imaging system to which the same configurations as those of the first imaging unit 3 and the second imaging unit 4 (see FIG. 2) of the first embodiment are applied to the back surface of the main body 2 on the Z-axis side of the imaging device 1. A second imaging unit group 115 including the unit 3B and the second imaging unit 4B is provided (see FIG. 7B). The first imaging unit group 114 is attached to the imaging lenses 31A and 41A of the first imaging unit 3A and the second imaging unit 4A so that wide-angle (for example, 180 °) imaging can be performed on the Z axis + side in the ZX plane. A wide-angle lens is used. The second imaging unit group 115 is attached to the imaging lenses 31B and 41B of the first imaging unit 3B and the second imaging unit 4B so that wide-angle (for example, 180 °) imaging can be performed on the Z-axis-side in the ZX plane. A wide-angle lens is used. As a result, the image pickup apparatus 1 of FIG. 7 is capable of imaging 360 ° in the ZX plane.
Note that FIG. 7 shows an example in which wide-angle imaging is performed using the first imaging unit group 114 and the second imaging unit group 115, but the number of imaging unit groups is not limited to two, but three. Wide-angle imaging may be performed using the above imaging unit group.

As shown in FIG. 8, the control unit 6 of the image pickup apparatus 1 includes an image pickup condition setting unit 61, a storage unit 65, a first image pickup unit group control unit 67, a second image pickup unit group control unit 68, and an entire image. It is provided with a processing unit 69. The imaging condition setting unit 61 sets imaging conditions for imaging the first imaging units 3A and 3B and the second imaging units 4A and 4B of the first imaging unit group 114 and the second imaging unit group 115, respectively. Set. The imaging conditions include, for example, exposure time (shutter speed), aperture, and ISO sensitivity, as in the case of the first embodiment. The imaging condition setting unit 61 sets the same imaging conditions for the first image sensor 32A of the first image sensor group 114 and the first image sensor 32B of the second image sensor group 115. The imaging condition setting unit 61 sets the same imaging conditions for the second image sensor 42A of the first image sensor group 114 and the second image sensor 42B of the second image sensor group 115.

The first imaging unit group control unit 67 is a control unit that controls the operation of the first imaging unit group 114. The first image sensor group control unit 67 is the first image sensor control unit 621A and the first image sensor control unit 621A to which the same configurations as those of the image sensor control unit 62 and the image processing unit 63 (see FIG. 2) of the first embodiment are applied. 2. The image sensor control unit 62A having the image sensor control unit 622A and the image processing unit 63A are provided. The image processing unit 63A uses the image signals output from the first image sensor 32A and the second image sensor 42A of the first image sensor group 114 in the same manner as in the case of the first embodiment and its modification. HDR image data is generated. The HDR image data generated by the image processing unit 63A is a wide-angle image of 180 ° in the ZX plane in front of the image pickup apparatus 1 (Z direction + side). The generated base image data, highlight image data, and HDR image data are stored in the storage unit 65 to which the same configuration as that of the first embodiment (see FIG. 2) is applied.

The second imaging unit group control unit 68 is a control unit that controls the operation of the second imaging unit group 115. The second image sensor group control unit 68 has the same configuration as the image sensor control unit 62 and the image processing unit 63 (see FIG. 2) of the first embodiment, and the first image sensor control unit 621B and the second image sensor control unit 68 are applied. 2. The image sensor control unit 62B having the image sensor control unit 622B and the image processing unit 63B are provided. The image processing unit 63B uses the image signals output from the first image sensor 32B and the second image sensor 42B of the second image sensor group 115 in the same manner as in the case of the first embodiment and its modification. HDR image data is generated. The HDR image data generated by the image processing unit 63B is a wide-angle image of 180 ° in the ZX plane on the back surface (Z direction − side) of the image pickup apparatus 1. The generated base image data, highlight image data, and HDR image data are stored in the storage unit 65 to which the same configuration as that of the first embodiment (see FIG. 2) is applied.

The overall image processing unit 69 includes wide-angle HDR image data on the front side of the image pickup device 1 generated by the image processing unit 63A and wide-angle HDR image on the back side of the image pickup device 1 generated by the image processing unit 63B. Connect with the data. As a result, the overall image processing unit 69 generates overall HDR image data in the entire circumferential direction on the ZX plane of the image pickup apparatus 1. The image pickup condition setting unit 61 described above sets the same image pickup conditions for the first image pickup elements 32A and 32B of the first image pickup unit group 114 and the second image pickup unit group 115, and sets the same image pickup conditions for the first image pickup unit group 114 and the first image pickup unit group 114 and the second. The same imaging conditions are set for the second image pickup elements 42 and 42B of the two image pickup unit group 115. Therefore, between the HDR image data generated by the image processing unit 63A of the first imaging unit group control unit 67 and the HDR image data generated by the image processing unit 63B of the second imaging unit group control unit 68, Differences in brightness are suppressed. Therefore, even when the above two HDR image data are joined by the whole image processing unit 69, an unnatural line appears on the whole HDR image data at the joined portion due to the difference in brightness. It is possible to suppress the occurrence of image deterioration such as the occurrence.
The generated overall HDR image data is stored in the storage unit 65 to which the same configuration as that of the first embodiment (see FIG. 2) is applied.

The first imaging unit group 114 and the second imaging unit group 115 described with reference to FIGS. 7 and 8 may be replaced with the third imaging unit 9 shown in FIG. 6, respectively. Also in this case, the configuration shown in FIG. 8 can be applied to the configuration of the control unit 6.
Further, the configuration of the density filter 35 shown in FIG. 5 may be applied as the density filters 34A and 34B included in the first imaging units 3A and 3B of the first imaging unit group 114 and the second imaging unit group 115.

Further, the first imaging units 3, 3A and 3B in the above-described first embodiment and its modification may not have the imaging lenses 31, 31A and 31B. In this case, the first image pickup units 3, 3A and 3B replace the image pickup lenses 31, 31A and 31B with, for example, a mask member having a two-dimensional aperture (for example, a known coded aperture) as the first image sensor 32. The first image sensor 32, 32A, 32B may be arranged in front of the 32A, 32B to capture an input subject image passing through the aperture. Similarly, the second imaging units 4, 4A, 4B and the third imaging unit 9 may be provided with the above-mentioned mask member without the imaging lenses 41, 41A, 41B and the imaging lens 91.

In addition, the image pickup apparatus described in the first embodiment and the modified examples 1 and 2 thereof can be used in mobile phones such as smartphones, televisions, tablet terminals, digital audio players, personal computers, endoscopes, capsule endoscopes, and home appliances. It can be applied to surveillance cameras including security cameras for personal computers, and imaging devices incorporated in electronic devices such as home appliances such as air conditioners, refrigerators and microwave ovens. Further, it can be applied to an imaging device incorporated in an electronic device such as a digital signage (electronic signboard). The digital signage may be, for example, a small display built in a vending machine or the like, or may be a display larger than the size of a general adult, for example, provided on the wall surface of a building or the like. .. Further, it may be applied to an imaging device mounted on various robots (for example, a self-propelled robot or an electronic device such as a self-propelled vacuum cleaner).

The first embodiment described above and the modified examples 1 and 2 include a first imaging unit 3, a second imaging unit 4, a display 5, a control unit 6, and an operation detection unit 7. Although the image pickup device 1 has been described as an example, the image pickup device may be composed of the first image pickup unit 3, the second image pickup unit 4, and the control unit 6. Further, the control unit 6 may include at least an image sensor control unit 62 and an image processing unit 63. In order to obtain the effects described in the first embodiment described above or the modified examples 1 and 2, a configuration may be appropriately added from the above configuration as necessary. Further, the above-mentioned imaging device may be incorporated into various electronic devices applied to the first embodiment and its modifications.

<Second Embodiment>
The image pickup apparatus of the second embodiment will be described with reference to FIG. In the image pickup apparatus of the second embodiment, as described in the first embodiment and its modification, the first image pickup element 32 and the second image pickup element 42 perform an image pickup, and then a short exposure is performed. Have the second image taken in time. When an image based on the image signal obtained by the second imaging is input regarding the stop of reproduction such as pause (pause) during reproduction of the HDR-processed moving image. To display. The details will be described below.

In the second embodiment, the same configuration as that of the image pickup apparatus 1 having the configurations shown in FIGS. 1 and 2 of the first embodiment can be applied. The image pickup device 1 of the second embodiment may be an image pickup device of another aspect shown in FIGS. 7 and 8. Further, the image pickup apparatus 1 of the second embodiment can be incorporated into various electronic devices described as a modification of the first embodiment.

FIG. 9 is a timing chart schematically showing the relationship between the exposure time of the first image sensor 32 and the exposure time of the second image sensor 42 during one frame, and the horizontal axis is time t. Further, the imaging condition setting unit 61 sets the imaging conditions for the first imaging unit 3 and the second imaging unit 4 for generating moving image data in the same manner as in the case of the first embodiment. ..
The image sensor control unit 62 instructs the first image sensor 32 and the second image sensor 42 to start exposure at time t1-1 and to instruct the end of exposure at time t1-2. That is, as in the case of the first embodiment, the image pickup by the first image pickup element 32 and the second image pickup element 42 is controlled, the output image signal is amplified, and then the image processing unit 63 raises the image. Light image data and base image data are generated. The image processing unit 63 synthesizes the highlight image data and the base image data by HDR processing, and generates HDR image data as moving image data.

The first image sensor control unit 621 refers to the first image sensor 32 after the exposure by the first image sensor 32 is completed at time t1-2 and before the time t2-1 when the exposure of the next frame is started. And have the second image taken. In the example shown in FIG. 9, the first image sensor control unit 621 sets the exposure time of the second image pickup to the first image sensor 32 to be shorter than the exposure time for generating moving image data, for example. Let it be done in 1 / 500s. The exposure time of 1 / 500s is an example, and the exposure time may be shorter than this, or the exposure time may be longer. As shown in FIG. 9, the first image sensor control unit 621 instructs the first image sensor 32 to start exposure at time t1-3 when a predetermined time has elapsed from time t1-2, and 1 / 500s has elapsed. The end of exposure is instructed at the time t1-4.

The image signal output after the second image pickup is performed by the first image sensor 32 is amplified by the amplification unit 321 and then subjected to various image processing by the image processing unit 63 and displayed at the time of pause (pause). It is generated as image data to be displayed on the device 5 (hereinafter, pose image data). The generated pose image data is stored in the storage unit 65. The image sensor control unit 62 similarly controls the operations of the first image sensor 32 and the second image sensor 42 for the image pickup after the second frame.

When the display control unit 64 reproduces and displays the HDR image data generated as described above on the display 5 as a moving image, an operation for instructing the user to pause (pause) is performed. The case will be described.
When the operation detection unit 7 detects that the operation for pausing has been performed by the user, the display control unit 64 reads out the pause image data stored in the storage unit 65 and pauses in the display 5. Display as an image. At this time, the display control unit 64 is generated during the same frame as the highlight image data and the base image data used for generating the HDR image data being reproduced and displayed when the pause operation is performed. The pose image data is read out from a plurality of pose image data. The display control unit 64 may read out the pause image data generated in the frame immediately before the frame when the pause operation is performed.

The same processing as that performed by the imaging device 1 of the first embodiment shown in FIG. 4 is applied to the process for generating the moving image data of the imaging device 1 of the second embodiment. However, in step S2, the first image sensor control unit 621 causes the first image sensor 32 to take an image for the pause image data after the image for the highlight image data is completed. In step S3, the image processing unit 63 further generates pause image data based on the image signal from the first image sensor 32. The generated pose image data is stored in the storage unit 65 in step S6.

The operation of reproducing and displaying the moving image of the imaging device 1 according to the second embodiment will be described with reference to the flowchart shown in FIG. Each process shown in the flowchart of FIG. 10 is performed by executing a program in the control unit 6. This program is stored in the storage unit 65, and when it is detected that the operation detecting unit 7 has performed an operation instructing the imaging of a moving image, the control unit 6 activates and executes the program.
In step S11, the display control unit 64 reads the HDR image data from the storage unit 65, reproduces and displays it on the display 5 as a moving image at a frame rate of 60 fps, and proceeds to step S12. In step S12, it is determined whether or not the pause operation has been performed. When the operation detection unit 7 detects that the pause operation has been performed, step S12 is determined affirmatively, and the process proceeds to step S13. If the operation detection unit 7 has not detected that the pause operation has been performed, the negative determination in step S12 is made, and the process proceeds to step S16, which will be described later.

In step S13, the display control unit 64 reads the pause image data from the storage unit 65 and proceeds to step S14. In step S14, the display control unit 64 causes the display 5 to display the read pause image data as a pause image, and proceeds to step S15. In step S15, it is determined whether or not the operation for ending the pause has been performed. When the operation detection unit 7 performs an operation to end the pause, step S15 is determined to be affirmative and the process proceeds to step S16. If the operation detection unit 7 has not performed an operation to end the pause, step S15 is determined to be negative and the process returns to step S14. In step S16, it is determined whether or not to end the reproduction display of the moving image. When the reproduction display of the moving image is finished, the affirmative determination is made in step S16, and the process is finished. When the reproduction display of the moving image is continued, the negative determination in step S16 is made and the process returns to step S11.

In the second embodiment, when the display control unit 64 receives an input regarding stopping the reproduction of the motion image during the reproduction of the moving image, the exposure for capturing the moving image at a time different from that of the moving image. The display 5 is displayed with a pose image based on the pose image data captured with an exposure time shorter than the time. As a result, the pose image data is generated based on the image signal captured and output by the first image sensor 32 in a short exposure time, so that the display 5 has a clear pose that is less affected by blurring. Images can be displayed.

The read pause image data is exposed with a short exposure time. For this reason, the occurrence of overexposure is suppressed, but the image as a whole is a dark image that is underexposed rather than the proper exposure. The image processing unit 63 performs, for example, gradation correction as image processing on the read out pose image data. That is, the image processing unit 63 corrects the gradation of the dark portion of the pause image data to be brightened. The image processing unit 63 can determine the correction amount of the gradation correction based on the base image data. In this case, the image processing unit 63 refers to the pixel value of the low-brightness region of the base image data, and the pixel value of the dark portion of the pause image data matches the pixel value of the referenced base image data, or the image value. The correction amount may be determined so that the difference between the above values is within the range of the predetermined value.
As described above, the image processing unit 63 performs image processing for correcting the gradation of the dark portion with respect to the pause image data. As a result, it is possible to display a pose image in which the dark part of the subject is easily visible.

Further, in the above description, the image sensor control unit 62 instructs the first image sensor 32 to take an image for the pause image data, but causes the second image sensor 42 to take an image for the pause image data. You may. Further, the image sensor control unit 62 may cause the first image sensor 32 and the second image sensor 42 to perform a second image pickup in a short exposure time (for example, 1/500 s). In this case, the image processing unit 63 generates image data for underexposure pause based on the image signal from the first image sensor 32, and pauses for overexposure based on the image signal from the second image sensor 42. Generate image data for. Then, the image processing unit 63 synthesizes the underexposure pose image data and the overexposure pose image data by HDR processing to generate the pose HDR image data, and stores it in the storage unit 65. When the pause operation is performed, the display control unit 64 reads the HDR image data for pause from the storage unit 65 and causes the display 5 to display it as a pause image.
As a result, the gradation in the low-luminance region and the high-luminance region of the subject is reproduced, and a clear pose image in which the occurrence of blurring or the like is suppressed can be displayed on the display 5.

It should be noted that the present invention is not limited to the case where the pause image data is used as the pause image. The display control unit 64 may use the pause image data for the representative image of the moving image or the thumbnail image showing the representative image of the moving image. Further, the image processing unit 63 may perform processing such as subject detection using the generated pose image data. Since the pose image data is an image captured with a short exposure time, the influence of blurring on the subject on the image is small. Therefore, it is possible to detect the subject with high accuracy.

<Modification 1 of the second embodiment>
The image pickup apparatus of the first modification of the second embodiment will be described with reference to FIGS. 11 to 12. The image pickup apparatus of the present modification 2 performs imaging for generating base image data and imaging for generating highlight image data a plurality of times in one frame, and the generated base image data and highlight image. It differs from the second embodiment in that moving image data and pose images are generated using the data. The details will be described below.

The imaging device of the first modification can be applied with the same configuration as the imaging device 1 of the second embodiment, that is, the same configuration as the imaging device 1 of the first embodiment shown in FIGS. 1 and 2. The image pickup device 1 of the modified example 2 of the second embodiment may be an image pickup device of another aspect shown in FIGS. 7 and 8. Further, the image pickup apparatus 1 of the modification 2 of the second embodiment can be incorporated into various electronic devices described as the modification of the first embodiment.

FIG. 11 is a timing chart schematically showing the relationship between the exposure start of the first imaging element 32 and the exposure time of the second imaging element 42 and the frame rate, and the horizontal axis is time t.
In the following description, the case where the image pickup by the first image sensor 32 and the second image sensor 42 is performed three times in one frame is taken as an example, but the number of times of imaging is not limited to three times. It may be twice or four times or more.

In the example shown in FIG. 11, the image pickup condition setting unit 61 sets the exposure time to, for example, 1/500 s, and the ISO sensitivity to 100 for the first image sensor 32, as in the case of the first embodiment. It shall be. The first image sensor control unit 621 instructs the first image sensor 32 to take the first image at an exposure time of 1/500 s at time t1-1, and thereafter, for example, every 1 / 240s for the second and third times. Instruct to take an image of. The imaging interval is not limited to the above 1 / 240s, for example, a plurality of times during a time of 1/2 to 1/1 of the time 1 / 60s between one frame (in the example of FIG. 11). The interval can be set so that imaging (3 times) is possible. Each time an image is taken by the first image sensor 32 and an image signal is output, the image processing unit 63 generates highlight image data from the image signal in the same manner as in the case of the first embodiment. This highlight image data is stored in the storage unit 65.

In the example shown in FIG. 11, the image pickup condition setting unit 61 sets the exposure time of the second image pickup element 42 to 1/500 s, which is equal to that of the first image pickup element 32, and sets the ISO sensitivity to the ISO sensitivity of the first embodiment. The value is set higher than the case (100,000 in the example of FIG. 11). That is, since the image pickup condition setting unit 61 sets the exposure time shorter than that of the second image pickup element 42 for capturing the overexposure base image data, the ISO sensitivity is set to the value exemplified in the first embodiment. Is also set to a high value. The second image sensor control unit 622 instructs the second image sensor 42 to take the first image at time t1-1, and thereafter instructs the second image sensor 42 to take the second and third images every 1 / 240s. That is, both the first image sensor 32 and the second image sensor 42 perform imaging at time t1-1 to t1-3, time t1-4 to t1-5, and time t1-6 to t1-2. .. Each time an image is taken by the second image sensor 42 and an image signal is output, the image processing unit 63 generates base image data from the image signal. This base image data is captured with the same exposure time as the exposure time of 1 / 500s when the highlight image data is captured. This base image data is stored in the storage unit 65.

The image processing unit 63 generates the base addition average image data by adding or averaging the above three base image data. Further, the image processing unit 63 generates highlight addition average image data by adding or averaging the above three highlight image data. The image processing unit 63 performs HDR processing on the generated base addition average image data and highlight image data in the same manner as in the case of the first embodiment to generate HDR image data. The generated HDR image data is stored in the storage unit 65. The same processing is performed for the subsequent frames, and HDR-processed moving image data is generated.

FIG. 12 shows an image reproduced and displayed on the display 5, an exposure time of the base image data used for generating the HDR image data, and an exposure time of the highlight image data used for generating the HDR image data. It is a timing chart schematically showing the relationship, and the time at the time of imaging is t and the time at the time of reproduction is T. In FIG. 12, it is assumed that the time at the time of imaging corresponding to the reproduction times T1 and T2 of the moving image is t1-1 and t2-1. In FIG. 12, when the display control unit 64 is playing back and displaying the HDR image data as a moving image on the display 5 according to the frame order of the HDR image data, a pause (pause) instruction is given at time T10. Suppose you were.

When the operation detection unit 7 detects that the operation for pausing has been performed by the user, the display control unit 64 is used to generate the HDR image data during playback display (3 in the illustrated example). One of the highlight image data is read from the storage unit 65. Further, the display control unit 64 reads out one base image data from the storage unit 65 from the plurality of (three in the illustrated example) base image data used for generating the HDR image data during playback display. In this case, the display control unit 64 sets the highlight image data and the base image data (in FIG. 12 in FIG. 12), the highlight image data and the base image data (in FIG. 12) The highlight image data and the highlight image data exposed at the times t1-1 to t1-3 before the broken line L in the figure are read out.

The image processing unit 63 synthesizes the read highlight image data and the base image data by HDR processing to generate HDR image data for a pause image. The display control unit 64 causes the display to display the HDR image data for the pause image generated by the image processing unit 63 at the time T2 when the next frame of the HDR image data which is the moving image data is displayed. The HDR image data for the pose image is generated by the base image data and the highlight image data having an exposure time of 1/500 s as described above. Since the base image data and the highlight image data are captured in a short exposure time, blurring that occurs in the subject even when a subject that is moving relative to the imaging device 1 is imaged. It becomes a clear image in which the influence of is suppressed. Therefore, even in the HDR image data for the pose image generated from the base image data and the highlight image data, the influence of the blur generated on the subject is suppressed and the image becomes clear.

The same processing as the processing performed by the imaging device 1 of the first embodiment shown in FIG. 4 is applied to the processing for generating the moving image data of the imaging device 1 of the modification 1 of the second embodiment. Will be done. However, in step S2, the image sensor control unit 62 causes the first image sensor 32 and the second image sensor 42 to perform image pickup a plurality of times. In step S3, the image processing unit 63 generates a plurality of highlight image data based on the image signal output a plurality of times from the first image sensor 32, and converts the image signal output from the second image sensor 42 a plurality of times. Generate multiple base image data based on.

The process for reproducing and displaying the moving image of the image pickup device 1 of the modification 1 of the second embodiment includes the same process as the process performed by the image pickup device 1 of the second embodiment shown in FIG. Applies. However, in step S13, the image processing unit 63 reads out the highlight image data for the pose and the base image data for the pose, synthesizes them by HDR processing, and generates HDR image data for the pose. In step S14, the display control unit 64 displays the generated HDR image data for the pose on the display 5 as a pause image.

In the first modification of the second embodiment, the image sensor control unit 62 causes the first image sensor 32 and the second image sensor 42 to take an image in a short exposure time, and the generated base image data and the high image sensor 62 are used. The HDR image data for the pose image is generated by synthesizing the light image data. As a result, even if the pause is performed during the reproduction of the moving image, the display 5 can display a clear pose image in which the occurrence of blurring of the subject is suppressed.

In the second embodiment described above and the first modification thereof, an image pickup device including a first image pickup unit 3, a second image pickup unit 4, a display 5, a control unit 6, and an operation detection unit 7. Although 1 has been described as an example, it may be an imaging device composed of a first imaging unit 3, a second imaging unit 4, a display 5, and a control unit 6. Further, the control unit 6 may include at least an image sensor control unit 62, an image processing unit 63, and a display control unit 64. In order to obtain the effects described in the second embodiment described above or the modified examples 1 and 2, a configuration may be appropriately added from the above configuration as necessary. Further, the above-mentioned imaging device may be incorporated into various electronic devices applied to the second embodiment and its modifications.

In the first embodiment and the second embodiment, the image pickup apparatus 1 having the configurations shown in FIGS. 1 and 2 has been described as an example, but the image pickup apparatus has three or more imaging units. May have. Hereinafter, a case of having three imaging units will be described in detail by taking as an example.
FIG. 13A is an external view of the image pickup apparatus 1. A fourth imaging unit 8 is provided on the front surface of the main body 2 of the imaging device 1 in addition to the first imaging unit 3 and the second imaging unit 4 similar to those in the first embodiment (see FIGS. 1 and 2). .. In the example shown in FIG. 13A, the center of the optical axis of the first imaging unit 3, the center of the optical axis of the second imaging unit 4, and the center of the optical axis of the fourth imaging unit 8 are equidistant from each other. Indicates the case where it is arranged in. The arrangement of the first imaging unit 3, the second imaging unit 4, and the fourth imaging unit 8 is not limited to this, and may be arranged along the X direction or the Y direction, or in the X direction or the Y direction. On the other hand, they may be arranged at a predetermined angle.

FIG. 13B is a block diagram schematically showing a configuration of a main part of the image pickup apparatus 1 shown in FIG. 13A. The fourth image pickup unit 8 included in the image pickup apparatus 1 includes an image pickup lens 81, a third image pickup element 82, and an aperture 83. The image pickup lens 81 has the same configuration as the image pickup lenses 31 and 41 of the first image pickup unit 3 and the second image pickup unit 4, and the third image pickup element 82 has the same configuration as the first image pickup element 32 and the second image pickup element 42. The configuration of is applied, and has an amplification unit 821 for amplifying an image signal. The image sensor control unit 62 of the control unit 6 includes a third image sensor control unit 623 for controlling the drive of the third image sensor 82 of the fourth image sensor 8. The third image sensor control unit 623 has the same configuration as the first image sensor control unit 621 and the second image sensor control unit 622. For other configurations, the configuration of the imaging device 1 of the first embodiment (see FIG. 2) is applied. That is, the image pickup device 1 of the modification 1 has, in addition to the above configuration, a first image pickup unit 3, a second image pickup unit 4, a display 5, a control unit 6, and an operation detection unit 7. ing. The control unit 6 includes an image pickup condition setting unit 61, an image sensor control unit 62, an image processing unit 63, a display control unit 64, and a storage unit 65.
The image pickup device 1 may be an image pickup device of another aspect shown in FIGS. 7 and 8, and can be incorporated into various electronic devices described as a modification of the first embodiment. is there.

In the imaging device 1, the fourth imaging unit 8 causes imaging under imaging conditions set so that gradation can be obtained in an intermediate luminance region between a high luminance region and a low luminance region. The image pickup condition setting unit 61 sets the image pickup conditions for the first image pickup element 32 and the second image pickup element 42 in the same manner as in the first embodiment. That is, the image pickup condition setting unit 61 sets the exposure time 1/120 s and the ISO sensitivity 100 for the first image sensor 32 as the image pickup conditions, and exposes the second image sensor 42 as the image pickup conditions. Set the time 1/120 s and ISO sensitivity 12800. For the third image sensor 82, the image pickup condition setting unit 61 includes highlight image data based on the image signal from the first image sensor 32 and base image data based on the image signal from the second image sensor 42. The image pickup conditions are set so that image data with intermediate brightness is generated. The imaging condition setting unit 61 sets the exposure time to 1 / 120s and the ISO sensitivity to, for example, 800.

The image processing unit 63 generates the highlight image data and the base image data in the same manner as in the cases of the first and second embodiments. The image processing unit 63 generates image data (hereinafter referred to as intermediate brightness image data) after the image signal output from the third imaging element 82 is amplified based on the set ISO sensitivity (800). The image processing unit 63 synthesizes the generated base image data, highlight image data, and intermediate brightness image data by HDR processing to generate HDR image data. The HDR image data generated in this way is clearly reproduced by the base image data captured by overexposure for a low-luminance subject as in the case of the first embodiment. In the HDR image data, a high-brightness subject that may cause overexposure is clearly reproduced by the highlight image data captured under the first imaging condition of underexposure. In the HDR image data, for a medium-brightness subject, an intermediate-brightness image captured under imaging conditions that is an intermediate exposure amount between the imaging conditions at the time of imaging the base image data and the imaging conditions at the time of imaging the highlight image data It is clearly reproduced by the data.

In the image pickup apparatus 1 shown in FIG. 13, the image processing unit 63 generates intermediate brightness image data based on the image signal from the fourth image pickup element 82, combines it with the highlight image data and the base image data, and makes an HDR image. Generate data. As a result, for example, even when a subject having a large difference in brightness is imaged, the gradation of the medium-luminance subject is reproduced based on the intermediate-luminance image data on the HDR image data, so that the HDR image data can be obtained. The image has a wide dynamic range as a whole.

Although the image pickup apparatus 1 described in the above-described embodiment has been described with the configuration having the display device 5, it is not always necessary to have the display device 5.
Further, although the image processing unit 63 of the image pickup apparatus 1 synthesizes the highlight image data and the base image data to generate the HDR image data, it is not always necessary for the image pickup apparatus 1 to generate the HDR image data. The image processing unit 63 is arranged outside the image pickup apparatus 1 (for example, the image processing unit 63 is a server or an image pickup function installed outside the image pickup apparatus, and the image processing function and the display 5 are separate bodies. The highlight image data and the base image data may be transmitted to the image processing unit 63 by the communication unit included in the image processing device 1 and the image processing unit 63 may generate HDR image data. When the image pickup device 1 includes the display device 5, the image processing unit 63 may transmit the generated HDR image data to the image pickup device 1 so that the display device 5 of the image pickup device 1 can display the HDR image data. .. Further, when the image processing unit 63 side includes the display 5, the generated HDR image data may be displayed on the display 5.

A program for various processes executed by the image pickup apparatus 1 to perform an image pickup operation may be recorded on a computer-readable recording medium, and this program may be read into a computer system to execute calibration. The "computer system" referred to here may include hardware such as an OS (Operating System) and peripheral devices.

The above-mentioned "computer system" also includes a homepage providing environment (or display environment) using a WWW system. The "computer-readable recording medium" includes a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, and the like. It refers to the storage device of. Further, the above-mentioned "computer-readable recording medium" is a volatile memory inside a computer system (for example, DRAM (Dynamic Random)) that serves as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Includes those that hold the program for a certain period of time, such as AccessMemory)).

The above-mentioned "program" may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned functions in combination with a program already recorded in the computer system.

The present invention is not limited to the above-described embodiment as long as the features of the present invention are not impaired, and other embodiments considered within the scope of the technical idea of the present invention are also included within the scope of the present invention. ..

1 ... Imaging device, 3 ... First imaging unit, 4 ... Second imaging unit,
5 ... Display, 6 ... Control unit, 7 ... Operation detection unit,
8 ... 4th imaging unit, 9 ... 3rd imaging unit,
34, 35 ... Density filter, 32, 92 ... First image sensor,
42, 93 ... 2nd image sensor, 61 ... Image condition setting unit,
62 ... Image sensor control unit, 63 ... Image processing unit, 64 ... Display control unit,
65 ... Storage unit, 66 ... Transmittance control unit, 67 ... First imaging unit group control unit,
68 ... 2nd imaging unit group control unit, 69 ... overall image processing unit, 82 ... 3rd imaging element,
114 ... 1st imaging unit group, 115 ... 2nd imaging unit group,
321, 421, 821, 921 ... Amplification unit,
322 ... 1st color filter, 422 ... 2nd color filter 621 ... 1st image sensor control unit, 622 ... 2nd image sensor control unit,
623 ... Third image sensor control unit

Claims (27)

A first image sensor that outputs a first image signal in which the subject is imaged in the first exposure time,
A second image sensor that outputs a second image signal captured by the subject in the second exposure time, and
An amplification unit that amplifies the first image signal and the second image signal with different gains,
Control to combine the first image data based on the first image signal amplified by the amplification unit and the second image data based on the second image signal amplified by the amplification unit to generate composite image data. With a department
At least one of the start of exposure for imaging at the first exposure time and the end of exposure for imaging at the first exposure time is performed during the imaging at the second exposure time .
The second image sensor captures an image in a third exposure time shorter than the second exposure time between the end of the exposure of the image in the second exposure time and the start of the exposure of the image in the next second exposure time. Output the third image signal
It said third image based on the image signal, scale in which these imaging device when the stop of reproduction during playback of the moving image based on the composite image data is set. The imaging device according to claim 1.
An optical member installed in the optical path of at least the first incident light so that the amount of the first incident light incident on the first image sensor is smaller than the amount of light of the second incident light incident on the second image sensor. With more
The amplification unit is an imaging device in which the gain for amplifying the first image signal is smaller than the gain for amplifying the second image signal. The imaging device according to claim 2.
A lens that collects the light incident on the first image sensor is further provided.
The optical member is an imaging device coated on the lens. The imaging device according to claim 2.
It further has a first filter that transmits a specific wavelength of light incident on the second image sensor.
The optical member is an image pickup apparatus that is a first filter that transmits a specific wavelength of light incident on the first image pickup device and has a lower transmittance than the second filter. The imaging device according to claim 2.
The optical member is an imaging device that divides light incident on the optical member into the first incident light and the second incident light. The imaging device according to claim 2.
The optical member is an imaging device capable of changing between a state in which the light of the first incident light is reduced and a state in which the amount of light of the first incident light is not reduced. The imaging device according to claim 2.
The optical member is an imaging device that changes the transmittance of the first incident light. The imaging device according to claim 2.
The optical member is an imaging device that reduces the amount of light of the first incident light without changing the luminous flux diameter of the first incident light. The imaging device according to claim 1.
The composite image data is a moving image generated at a predetermined frame rate, and is
The second exposure time is an imaging device having a time of 1/4 or more of the predetermined frame rate. The imaging device according to claim 2.
An optical member installed in the optical path of at least the first incident light so that the amount of the first incident light incident on the first image sensor is smaller than the amount of light of the second incident light incident on the second image sensor. With more
The optical member is set so that the EV value based on the gain for amplifying the second image signal by the amplification unit during the second exposure time is 14 EV or more.
The amplification unit is an imaging device in which the gain for amplifying the second image signal is smaller than the gain for amplifying the first image signal. The imaging device according to claim 3.
The optical member is an imaging device having a transmittance of 1/16 × 100% or less. The imaging device according to claim 1.
The composite image data is an imaging device generated by correcting a low-luminance region of the second image data to a higher brightness than the second image data based on the first image data. The imaging device according to claim 1.
The composite image data is a moving image generated at a predetermined frame rate, and is
The first exposure time and the second exposure time are longer than half of the predetermined frame rate. The imaging device according to claim 1.
The image pickup apparatus in which the first exposure time is the same as the second exposure time. The imaging device according to claim 1.
The first image sensor outputs the first image signal at the first frame rate.
The second image sensor outputs the second image signal at a second frame rate.
The control unit obtains the first image data based on the first image signal output at the first frame rate and the second image data based on the second image signal output at the second frame rate. An imaging device that synthesizes and generates a moving image based on the composite image data at a third frame rate. The imaging device according to claim 15.
An imaging device in which the first frame rate and the third frame rate are the same frame rate. The imaging device according to any one of claims 1 to 16 .
The display unit is further provided with an input unit for setting the reproduced moving image.
The control unit is an imaging device that displays an image based on the third image signal on the display unit when an input regarding stopping reproduction of the moving image is input by the input unit during reproduction of the moving image. The imaging device according to claim 17 .
The first image sensor has a fourth exposure time shorter than the first exposure time between the end of the exposure of the image pickup in the first exposure time and the start of the next exposure of the image pickup in the first exposure time. Take an image, output the 4th image signal,
When the input unit receives an input regarding stopping the reproduction of the moving image during the reproduction of the moving image, the control unit converts the third image data based on the third image signal and the fourth image signal into the third image data. An imaging device that displays an image based on the image data obtained by combining the fourth image data based on the image data on the display unit. The imaging device according to claim 17 or 18 .
Display device having said display portion. An electronic device including the imaging device according to any one of claims 1 to 18. Outputs the first image signal captured by the subject during the first exposure time,
The second image signal captured by the subject in the second exposure time is output.
The first image signal and the second image signal are amplified with different gains.
The first image data based on the amplified first image signal and the second image data based on the amplified second image signal are combined to generate composite image data.
At least one of the start of exposure for imaging at the first exposure time and the end of exposure for imaging at the first exposure time is performed during the imaging at the second exposure time .
Between the end of the exposure of the imaging in the second exposure time and the start of the exposure of the imaging in the next second exposure time, the image is taken in the third exposure time shorter than the second exposure time and the third image signal is output. And
The image is based on the third image signal, scale in which these imaging methods when stopping playback during playback of the moving image based on the synthesized image data is set. A program that lets a computer run
The process of outputting the first image signal captured by the subject during the first exposure time,
A process of outputting a second image signal captured by the subject in the second exposure time, and
A process of amplifying the first image signal and the second image signal with different gains,
A process of synthesizing the first image data based on the amplified first image signal and the second image data based on the amplified second image signal to generate composite image data.
At least one of the start of exposure for imaging at the first exposure time and the end of exposure for imaging at the first exposure time is a process performed during the imaging at the second exposure time and
Between the end of the exposure of the imaging in the second exposure time and the start of the exposure of the imaging in the next second exposure time, the image is taken in the third exposure time shorter than the second exposure time and the third image signal is output. Processing to do and
A program for causing a computer to execute a process of displaying an image based on the third image signal when playback stop is set during playback of a moving image based on the composite image data . A first imaging element that outputs a first image signal imaged by the subject in the first exposure time, and
A second image sensor that outputs a second image signal captured by the subject in the second exposure time, and
An amplification unit that amplifies the first image signal and the second image signal with different gains,
The first communication unit that communicates with external electronic devices,
The external first image data based on the first image signal whose gain is amplified by the first gain amplification unit and the second image data based on the second image signal whose gain is amplified by the second gain amplification unit are external. The electronic device is provided with a first control unit that transmits data via the first communication unit.
At least one of the start of exposure for imaging at the first exposure time and the end of exposure for imaging at the first exposure time is performed during the imaging at the second exposure time .
The second image sensor captures an image in a third exposure time shorter than the second exposure time between the end of the exposure of the image in the second exposure time and the start of the exposure of the image in the next second exposure time. Output the third image signal
Images, the first image data and the second image data and the synthesized composite image scale in which these imaging device when the data playback is stopped during playback of a moving image based on is set based on the third image signal .. The imaging device according to claim 23 ,
A second communication unit that receives the first image data and the second image data, and
An image pickup system including an image processing device including a second control unit that synthesizes the first image data and the second image data and generates the composite image data. The imaging system according to claim 24 ,
Display system comprising a display unit for displaying the synthesized image based on the previous SL composite image data. A first image sensor that outputs a first image signal in which the subject is imaged in the first exposure time, a second image sensor that outputs a second image signal in which the subject is imaged in the second exposure time, and the first image sensor. It has an amplification unit that amplifies the image signal and the second image signal with different gains, and at least one of the exposure start of the image pickup at the first exposure time and the exposure end of the image pickup at the first exposure time is It is performed during the imaging of the second exposure time, and the second image sensor performs the second exposure between the end of the exposure of the imaging in the second exposure time and the start of the exposure of the imaging in the next second exposure time. from the imaging device you output the third image signal for imaging at shorter than 2 exposure time third exposure time, and the first image data based on the gain is amplified by the first gain amplifying unit first image signal, A communication unit that receives the second image data based on the second image signal whose gain is amplified by the second gain amplification unit, and
A control unit that synthesizes the first image data and the second image data and generates composite image data is provided .
Said third image based on the image signal, an image processing apparatus that will be displayed when the composite image data playback is stopped during playback of a moving image based on is set. The image processing apparatus according to claim 26 and
Display device comprising a display unit for displaying an image based on the synthesized image data.

Images