JP2015050498A - Imaging device, imaging method, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To photograph a wide luminance area at a proper brightness and to prevent generation of difference at a connection part of photographed images, in a case where a subject having different luminances is photographed by a plurality of imaging optical systems.SOLUTION: An imaging device comprises: a first photometry value calculation part 51 calculating a photometry value in a plurality of solid-state imaging elements; a second photometry value calculation part 52 calculating a photometry value in each solid-state imaging element; a first exposure value calculation part 53 calculating an exposure value from both photometry values; a second exposure value calculation part 54 calculating an exposure value of each solid-state imaging element from the photometry value at the second photometry value calculation part; an exposure control value setting part 55 setting a solid-state imaging element having a photometry value larger than the photometry value at the first photometry value calculation part among the photometry values at the second photometry value calculation part as a first solid-state imaging element, setting the remaining solid-state imaging element as a second solid-state imaging element, setting an exposure control value from the exposure value at the first exposure value calculation part to the first solid-state imaging element, and setting an exposure control value from the exposure value at the second exposure value calculation part to the second solid-state imaging element; and an exposure difference correction part 56 correcting an exposure difference from the first solid-state imaging element to the second solid-state imaging element.

Description

  The present invention relates to an imaging apparatus capable of providing appropriate imaging conditions for a plurality of imaging optical systems, an imaging method executed by the imaging apparatus, and a recording medium on which a program for realizing the imaging apparatus is recorded.

  2. Description of the Related Art An omnidirectional imaging system that uses a plurality of wide-angle lenses such as fish-eye lenses and super-wide-angle lenses to capture images in all directions (hereinafter referred to as omnidirectional spheres) at once is known. In the omnidirectional imaging system, an image from each lens is projected onto the sensor surface, and the obtained images are combined by image processing to generate an omnidirectional image. For example, an omnidirectional image can be generated using two wide-angle lenses having an angle of view exceeding 180 degrees. In the image processing, distortion correction and projective transformation are performed on the partial images photographed by each lens optical system, and the partial images are connected using the overlapping regions included in the partial images, thereby producing one omnidirectional image. Is performed.

  Conventionally, an exposure correction technique in a digital camera that obtains proper exposure from a captured image is known (see, for example, Patent Document 1). In Patent Document 1, an overlapping area is extracted from each of captured images captured by a plurality of imaging devices arranged so that a part of the imaging area overlaps each other. Then, based on each image of the extracted overlapping area, at least one of exposure and white balance of the plurality of imaging devices is adjusted, and differences in brightness and color of the captured images are reduced to perform post-processing such as synthesis. I try to reduce the burden.

  However, the above conventional technique has a problem that it is difficult to obtain an appropriate exposure because only the overlapping region is considered. That is, in the technique of Patent Document 1, since exposure correction is performed using only the overlapping area, if the overlapping area becomes small with respect to the entire image, appropriate exposure correction is performed when an imbalance occurs in the exposure condition. It becomes difficult to get a value. In addition, when exposure correction is performed individually, there is also a problem that the brightness changes at the boundary between images.

  The problem of the present invention is that even when a subject with different brightness is photographed with a plurality of imaging optical systems, it is possible to photograph with appropriate brightness from a low brightness area to a high brightness area, and there is an unnatural difference between the joints of the photographed images. An object of the present invention is to provide an imaging apparatus capable of avoiding the occurrence.

  In order to solve the above-described problems, the present invention is an image pickup apparatus that joins picked-up images acquired from a plurality of solid-state image sensors to form and output a single image, which is acquired from the plurality of solid-state image sensors. First photometric value calculating means for calculating a photometric value based on an imaging signal, and calculating a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor among the plurality of solid-state image sensors. Second photometric value calculating means; first exposure value calculating means for calculating an exposure value from at least one of the photometric value calculated by the first photometric value calculating means and the photometric value calculated by the second photometric value calculating means; A second exposure value calculating means for calculating an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculating means, and calculating by the second photometric value calculating means. Among the measured photometric values, a solid-state imaging device larger than the photometric value calculated by the first photometric value calculating means is a first solid-state imaging device, the other solid-state imaging device is a second solid-state imaging device, and the first solid state For the image sensor, an exposure control value is set based on the exposure value calculated by the first exposure value calculator, and for the second solid-state image sensor, the exposure calculated by the second exposure value calculator. Exposure control value setting means for setting an exposure control value based on the value; and exposure difference correction means for correcting an exposure difference between the second solid-state image sensor and the first solid-state image sensor. And

  According to the present invention, an exposure control value is set for the second solid-state imaging device based on the exposure value calculated by the second exposure value calculating means, and further, an exposure difference from the first solid-state imaging device is corrected. . As a result, when subjects with different luminance are photographed with a plurality of imaging optical systems, it is possible to photograph with appropriate brightness from low luminance to high luminance. In addition, it is possible to avoid an unnatural difference between the joints of the captured images.

It is the conceptual diagram which showed a mode that a subject (person) was image | photographed when the sun has come out using the imaging system by a present Example. It is sectional drawing which shows an imaging system. It is the figure which showed the hardware constitutions of the imaging system. It is a figure explaining the flow of the whole image processing in an imaging system. It is a figure explaining the flow of a continuation of FIG. 4A. It is a figure which shows the flow of a series of AE processes at the time of monitoring of an imaging device. It is the figure which showed a mode that the area integration value was equally divided | segmented by the block unit of horizontal 16xvertical 16 from the raw-RGB data of an image. It is a figure which shows the AE table showing the difference (delta Ev) with the appropriate exposure corresponding to AE evaluation value. It is a figure which shows the flow of a series of AE processes at the time of still of an imaging device. It is the block diagram which showed schematically the part relevant to a present Example among the internal structures of CPU in the processor in FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  Examples of the present invention will be described below, but the examples of the present invention are not limited to the examples described below. In the following embodiments, as an example of an imaging apparatus, the imaging system includes an imaging body including two fisheye lenses in an optical system, and has a function of determining imaging conditions based on a captured image captured by two fisheye lenses. A description will be given using an omnidirectional imaging system. That is, in another embodiment, the omnidirectional imaging system includes an imaging body including three or more fisheye lenses in an optical system, and determines imaging conditions based on captured images captured by three or more fisheye lenses. It may have a function. In the present embodiment, the fisheye lens includes a so-called wide-angle lens and a super-wide-angle lens.

  FIG. 1 shows a state in which a subject (person) 11 is photographed using the omnidirectional imaging system 10. At this time, the sun 13 is also photographed together with the subject 11. That is, the subject 11 is photographed by a camera A (camera on the left side in the figure) which is an imaging optical system of the omnidirectional imaging system 10, and the sun 13 is photographed by a camera B (camera on the right side in the figure) which is an imaging optical system. Take a picture.

  Then, the captured image captured by the camera A and the captured image captured by the camera B are connected, and the combined image is synthesized after performing tilt correction and distortion correction on the combined captured image. create.

  Hereinafter, the overall configuration of the omnidirectional imaging system 10 according to the present embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a cross-sectional view showing an omnidirectional imaging system (hereinafter simply referred to as an imaging system) 10 according to the present embodiment. The imaging system 10 illustrated in FIG. 2 includes an imaging body 12, a housing 14 that holds the imaging body 12, and components such as a controller and a battery (not shown), and a shutter button 18 provided on the housing 14. .

  2 includes two lens optical systems 20A and 20B (also referred to as lens optical system 20), and two solid-state image sensors such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. 22A, 22B (also referred to as solid-state image sensor 22). In this embodiment, a combination of the lens optical system 20 and the solid-state imaging element 22 one by one is referred to as an imaging optical system. Each of the lens optical systems 20 can be configured as a fish-eye lens with, for example, 7 elements in 6 groups. In the example shown in FIG. 2, the fisheye lens has a total angle of view larger than 180 degrees (= 360 degrees / n; n = 2), preferably has an angle of view of 185 degrees or more, and more preferably Has an angle of view of 190 degrees or more.

  The optical elements (lenses, prisms, filters, and aperture stops) of the two lens optical systems 20A and 20B are positioned so that their optical axes are orthogonal to the center of the light receiving region of the corresponding solid-state imaging element 22, and The positional relationship is determined with respect to the solid-state imaging devices 22A and 22B so that the light receiving area is the image plane of the corresponding fisheye lens. Each of the solid-state imaging devices 22 is a two-dimensional imaging device in which a light receiving region forms an area area, and converts light collected by the combined lens optical system 20 into an image signal.

  In the embodiment shown in FIG. 2, the lens optical systems 20A and 20B have the same specifications, and are combined in opposite directions so that their optical axes match. The solid-state imaging devices 22A and 22B convert the received light distribution into image signals and output them to image processing means on the controller. In the image processing means, the captured images respectively input from the solid-state imaging devices 22A and 22B are connected and combined to generate an image with a solid angle of 4π radians (hereinafter referred to as “global celestial image”). The omnidirectional image is an image of all directions that can be seen from the shooting point. Here, in the embodiment shown in FIG. 2, the omnidirectional image is generated, but in another embodiment, a so-called panoramic image obtained by photographing 360 degrees only on the horizontal plane may be used.

  As described above, since the fisheye lens has a full angle of view exceeding 180 degrees, when composing an omnidirectional image, overlapping image portions represent the same image in the captured images captured by the respective imaging optical systems. It is used as reference data for image stitching as reference data. The generated omnidirectional image is, for example, a display device, a printing device, an SD (registered trademark) card, a compact flash (registered trademark), or the like that is provided in the imaging body 12 or connected to the imaging body 12. Output to an external storage medium.

  FIG. 3 shows a hardware configuration of the imaging system 10 according to the present embodiment. The imaging system 10 includes a digital still camera processor (hereinafter simply referred to as a processor) 100, a lens barrel unit 102, and various components connected to the processor 100. The lens barrel unit 102 includes the above-described two sets of lens optical systems 20A and 20B and the solid-state imaging elements 22A and 22B. The solid-state image sensor 22 is controlled by a control command from the CPU 130 in the processor 100. Details of the CPU 130 will be described later.

  The processor 100 includes an image signal processor (ISP) 108, a direct memory access controller (DMAC) 110, and an arbiter (ARBMEMC) 112 for arbitrating memory access. The processor 100 further includes a MEMC (Memory Controller) 114 that controls memory access, a distortion correction / image synthesis block 118, and a face detection block 201. The ISPs 108A and 108B perform automatic exposure (AE) control, white balance setting, and gamma setting on images input through signal processing of the solid-state imaging devices 22A and 22B, respectively.

  An SDRAM 116 is connected to the MEMC 114. The SDRAM 116 temporarily stores data when the ISPs 108A and 180B and the distortion correction / image synthesis block 118 perform processing. The distortion correction / image synthesis block 118 performs correction on the two captured images obtained from the two imaging optical systems using the information from the three-axis acceleration sensor 120 and the top / bottom correction (tilt correction) to correct the distortion. Composite later images. Note that the face detection block 201 performs face detection using the tilt-corrected image and specifies the face position.

  The processor 100 further includes a DMAC 122, an image processing block 124, a CPU 130, an image data transfer unit 126, an SDRAM C 128, a memory card control block 140, a USB block 146, a peripheral block 150, and an audio unit 152. Serial block 158, LCD driver 162, and bridge 168.

  The CPU 130 controls the operation of each unit of the imaging system 10. The image processing block 124 performs various image processing on the image data. The resize block 132 is a block for enlarging or reducing the size of the image data by interpolation processing. The JPEG block 134 is a codec block that performs JPEG compression and expansion. H. H.264 block 136 is an H.264 block. It is a codec block that performs video compression and decompression such as H.264. The processor 100 is provided with a power controller 202.

  The image data transfer unit 126 transfers the image processed by the image processing block 124. The SDRAM C 128 controls the SDRAM 138 connected to the processor 100. The SDRAM 138 temporarily stores image data when various processes are performed on the image data in the processor 100.

  The memory card control block 140 controls reading and writing with respect to the memory card inserted into the memory card slot 142 and the flash ROM 144. The memory card slot 142 is a slot for detachably attaching a memory card to the imaging system 10. The USB block 146 controls USB communication with an external device such as a personal computer connected via the USB connector 148. A power switch 166 is connected to the peripheral block 150.

  The audio unit 152 is connected to a microphone 156 from which a user inputs an audio signal and a speaker 154 from which a recorded audio signal is output, and controls audio input / output. The serial block 158 controls serial communication with an external device such as a personal computer, and is connected with a wireless NIC (Network Interface Card) 160. An LCD (Liquid Crystal Display) driver 162 is a drive circuit that drives the LCD monitor 164, and converts it into signals for displaying various states on the LCD monitor 164.

  The flash ROM 144 stores a control program and various parameters described by codes that can be read by the CPU 130. When the power is turned on by operating the power switch 166, the control program is loaded into the main memory, and the CPU 130 controls the operation of each part of the apparatus according to the program read into the main memory. At the same time, data necessary for control is temporarily stored in the SDRAM 138 and a local SRAM (not shown).

  By using the rewritable flash ROM 144, the control program and parameters for control can be changed, and the function can be easily upgraded.

  4A and 4B are diagrams for explaining the overall flow of image processing in the imaging system 10 according to the present embodiment, and show main functional blocks for controlling imaging conditions. First, as shown in FIG. 4A, an image is captured under a predetermined exposure condition parameter by each of the sensor A (solid-state imaging device 22A) and the sensor B (solid-state imaging device 22B). Subsequently, ISP1-A and ISP1-B are processed by the ISP 108 (108A, 108B) shown in FIG. 3 for the images output from the sensors A and B, respectively. As processing of ISP1-A and ISP1-B, an optical black (OB) correction process, a defective pixel correction process, a linear correction process, a shading correction process, and an area division average process are executed. Saved in memory.

  The optical black (OB) correction process is a process in which the output signal in the effective pixel area is clamp-corrected with the output signal in the optical black area in the sensors A and B as the black reference level. A solid-state imaging device such as a CMOS is manufactured by forming a large number of photosensitive elements on a semiconductor substrate. However, pixel values are locally captured due to impurities mixed into the semiconductor substrate during the manufacturing. An impossible defective pixel may occur. The defective pixel correction process is a process of correcting the pixel value of the defective pixel based on the combined signal from a plurality of pixels adjacent to the defective pixel as described above.

  The linear correction processing is processing for performing linear correction for each RGB. The shading correction process is a process for correcting shading (shading) distortion of the effective pixel region by multiplying the output signal of the effective pixel region by a predetermined correction coefficient. In the area division average process, an image area constituting the captured image is divided into a plurality of areas, and a luminance integrated value (or integrated average value) is calculated for each divided area. This processing result is used for AE processing.

  When the processing of ISP1-A and ISP1-B is completed, the processing of ISP2-A and ISP2-B is subsequently performed by the ISP 108 (108A, 108B) as shown in FIG. 4B. As processing of ISP2-A and ISP2-B, white balance (WB Gain) processing, gamma (γ) correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement (YCFLT) processing, and color correction processing are executed. Is stored in memory.

  A color filter of any one of R (red), G (green), and B (blue) is attached to each of the photodiodes on the sensors A and B for accumulating the amount of light from the subject. Since the amount of transmitted light varies depending on the color of the filter, the amount of charge accumulated in the photodiode differs. The most sensitive is G, and R and B are less sensitive than G, about half. In the white balance (WB) process, a process of multiplying R and B is performed in order to compensate for these sensitivity differences and make white in the captured image appear white. Moreover, since the color of an object changes with light source colors (for example, sunlight, a fluorescent lamp, etc.), even if a light source changes, it has the function to change and control the gain of R and B so that white may appear white. ing. The white balance processing parameters are calculated based on the RGB integrated value (or integrated average value) data for each divided region calculated by the region dividing average processing.

  The output signal has a non-linear relationship with respect to the input signal. In the case of such a non-linear output, there is no gradation in brightness, and the image becomes dark, so that a person cannot see the image correctly. In view of the characteristics of the output device, the gamma (γ) correction processing is performed on the input signal in advance so that the output maintains linearity.

  In CMOS, an array called a Bayer array has a color filter of one of R (red), G (green), and B (blue) attached to one pixel, and RAW data is one color information per pixel. There is only. However, in order to view an image from RAW data, information of three colors R, G, and B is required for one pixel, and Bayer interpolation processing is a process of interpolating from surrounding pixels to compensate for two missing colors. is there. In a JPEG image of a file format generally used in a digital camera or the like, since an image is created from YUV data, RGB data is converted into YUV data.

  The edge enhancement (YCFLT) process is a process for extracting an edge portion from a luminance signal of an image, multiplying the edge by a gain, and removing image noise in parallel with the edge extraction. Specifically, an edge extraction filter that extracts an edge portion from the luminance (Y) signal of the image, a gain multiplication unit that multiplies the gain extracted by the edge extraction filter, and an image in parallel with the edge extraction. An LPF (low-pass filter) that removes noise, and a data addition unit that adds edge-extracted data after gain multiplication and image data after LPF processing are included.

  In the color correction process, saturation setting, hue setting, partial hue change setting, color suppression setting, and the like are performed. The saturation setting is a parameter setting that determines the color depth and indicates the UV color space. For example, in the second quadrant, the length of the vector from the origin to the R dot for the R color is The longer the color, the darker the color.

  The color-corrected data is stored in a memory (DRAM), and a crop process is executed based on the stored data. This cropping process is a process for generating a thumbnail image by cutting out the central area of the image.

  Based on FIGS. 4A and 4B, the operation of the imaging apparatus of the present embodiment will be described.

  For the Bayer RAW image output from the sensor A, in the ISP1-A, an optical black (OB) correction process, a defective pixel correction process, a linear correction process, a shading correction process, and an area division average Process. The image is stored in DRAM. Similarly, for the Bayer RAW image output from the sensor B, in the ISP1-B, an optical black (OB) correction process, a defective pixel correction process, a linear correction process, a shading correction process, Perform area division averaging. The image is stored in DRAM.

  The sensors A and B have an independent simple AE processing function, and each of the sensors A and B can be set to an appropriate exposure independently. When the change in exposure condition of each of the sensors A and B becomes small and stable, each sensor is used by using the area integrated value obtained by the area division averaging process so that the brightness of the image boundary portion of the binocular image matches. Set A and B to proper exposure.

  For the data on the sensor A side where the processing of ISP1-A is completed, white balance (WB Gain) processing, gamma (γ) correction processing, Bayer interpolation processing, YUV conversion processing, edge enhancement (YCFLT) in ISP2-A ) Processing and color correction processing are executed. The data after execution is stored in the DRAM. Similarly, for data on the sensor B side after the processing of ISP1-B, white balance (WB Gain) processing, gamma (γ) correction processing, Bayer interpolation processing, YUV conversion processing, edge in ISP2-B Emphasis (YCFLT) processing and color correction processing are executed. The data after execution is stored in the DRAM.

  The data for which the processing of ISP2-A or ISP2-B has been completed is subjected to a processing (crop processing) in which each of the sensor A side or the sensor B side is cut into a regular image, and thereafter, distortion correction / combination processing is performed. The In the course of distortion correction / combination processing, information from the triaxial acceleration sensor is obtained and tilt correction (top-to-bottom correction) is performed. The image is further compressed by JPEG compression with a compression coefficient of about 0.16.

  This data is stored in the DRAM, and file storage (tagging) is performed. Further, the data is stored in a medium such as an SD card via SDIO. When transferring to a smartphone (such as a portable terminal), wireless LAN (Wi-Fi, Bluetooth (registered trademark), etc.) is used to transfer to the smartphone (such as a portable terminal).

  In the data handled in the processing in FIGS. 4A and 4B, the spherical image is rectangular image data and is a part of the spherical image. It is intended to be displayed as a thumbnail image or the like. The present embodiment can be realized by an image capable of capturing two or more omnidirectional images. In FIGS. 4A and 4B, as a specific example, image synthesis is performed using two image sensors. The flow about is shown.

  Next, the characteristic part of a present Example is demonstrated using FIG.

  FIG. 5 shows a flow of a series of AE processes during monitoring of the imaging apparatus according to the present embodiment. First, initial values of Tv (shutter speed) and Sv (sensitivity) are set for the sensors A and B (step S11). The value to be set is common to the sensor A and the sensor B.

  Then, Bv (subject luminance data) and Ev (exposure value) are calculated by the following equation (1). Av represents the aperture value. In this embodiment, however, the Av value is fixed because there is no aperture mechanism.

      Bv = Ev = Tv + Av− (Sv−0x50) (1)

  Tv, Av, Sv, Bv, and Ev are values in Apex format (1/48 step), and are used for the calculation.

  Next, it is determined whether or not Tv (shutter speed) and Sv (sensitivity) are reflected in the detected value (step S12), and if they are reflected, the respective area integrated values of the sensors A and B are acquired. (Step S13). In step S12, if Tv (shutter speed) and Sv (sensitivity) are not reflected in the detected value, the process proceeds to step S18.

  As shown in FIG. 6, the area integration value is equally divided into 16 horizontal 16 × vertical 16 blocks from the RAW-RGB data of the image, and the RGB values are integrated for each of the divided blocks. Then, the Y value (luminance value) is obtained from the RGB values in units of blocks using the following equation (2). In this embodiment, the area integrated value is a circular inner portion that is not shielded from light, and this is the Y value.

      Luminance value = R × 0.299 + G × 0.587 + B × 0.114 (2)

  In this embodiment, the number of block divisions is 16 × 16 = 256, but the present invention is not limited to this. However, when n divided blocks, n ≧ 4 is satisfied. Further, in the present embodiment, although it is equally divided, it is not necessarily limited to this. However, it is preferable that all the divided blocks are equally divided into the same area and the same shape.

  Here, the area integrated value will be described in detail. The calculation of the area integrated value is performed for each of the above-described divided blocks. In this embodiment, the divided blocks are obtained by equally dividing the captured image as described above. For example, if the captured image has about 10 million pixels, each divided block has about 39,000. It has a pixel. Each pixel included in each of the divided blocks is information on R, G, or B components of the corresponding subject portion. In this embodiment, each component is recorded and used as 12-bit information (0 to 255). That is, in each of the 256 divided blocks, there is 12-bit R, G, and B component information corresponding to the number of pixels (approximately 10 million pixels / 256 = approximately 39,000 pixels) of each block.

  The area integrated value is calculated by averaging the R component, the G component, and the B component of all the pixels included in each block for each of the divided blocks. In this embodiment, each of the divided blocks divided into 256 pieces is output as 12-bit information for each of the R, G, and B components.

  In this embodiment, the ratio of R, G, and B pixels is R: G: B = 1: 2: 1, and each of the divided blocks has R pixels = about 0.975 million pixels, G pixel = about 19.95 million pixels and B pixel = 0.975 million pixels.

  Next, an AE evaluation value that is a value obtained by dividing the area integrated value by the integrated number is calculated (step S14). This AE evaluation value is used for the subsequent exposure calculation.

  The AE evaluation values of sensor A and sensor B that are below a certain value are averaged, and the difference (delta Ev1) from the appropriate exposure is calculated based on the AE table (step S15). As shown in FIG. 7, the AE table represents the difference (delta Ev) from the appropriate exposure corresponding to the AE evaluation value. When the AE evaluation value is 920, it is +1 Ev brighter than the appropriate exposure. Moreover, in the case of the AE evaluation value 230, it will be -1Ev dark from appropriate exposure. For the AE evaluation values between the points (that is, AE evaluation values not shown in FIG. 7), delta Ev is calculated by linear interpolation.

  When the AE evaluation value is +3 or more or −3 or less, the delta Ev is clipped at +3 or −3. Then, the AE evaluation values of the sensor A that are below a certain value are averaged, and a difference (delta Ev2) from the appropriate exposure is calculated based on the AE table. Further, the AE evaluation values of sensor B that are below a certain value are averaged, and a difference (delta Ev3) from the appropriate exposure is calculated based on the AE table.

  Next, delta Ev1 is added to Bv calculated previously to update subject luminance data 1 (Bv1) (step S16). Further, an exposure value 1 (Ev1) is calculated based on Bv1, and when delta Ev becomes 0, it is determined that the exposure is appropriate (brightness), and Ev1 is calculated so that delta Ev becomes 0. .

  According to the calculated Ev1 to EV diagram, the exposure conditions, that is, the CMOS shutter speed (Tv) and the CMOS sensitivity (Sv (gain value)) are calculated, and the calculated shutter speed and CMOS sensitivity are represented by the sensors A and B. (Step S17).

  The EV diagram is a table of combinations of shutter speeds and sensitivities corresponding to Ev values, and may be different between monitoring and still.

  It is determined whether or not the AE process at the time of monitoring is finished (step S18). If not finished, the process returns to step S12, where Tv (shutter speed) and Sv (sensitivity) set in step S17 are reflected. The integrated value is acquired in step S13. The processes in steps S12 to S18 are repeatedly executed during monitoring.

  FIG. 8 shows a flow of a series of AE processes when the imaging apparatus according to the present embodiment is still. First, Ev2 is calculated based on the Ev value and Ev1 of the sensor having the larger values of delta Ev2 and delta Ev3 calculated at the time of monitoring immediately before still photography (step S21). For example, when delta Ev2> delta Ev3, delta Ev2 is set by the following equation (3).

      Ev2 = Ev1 + Delta Ev2 (3)

  Thereby, the sensor on the high luminance side can be brought close to the appropriate exposure.

  Next, shutter speed 1 (Tv1) and sensitivity 1 (Sv1) are calculated according to the EV diagram from Ev2 (step S22).

  The difference between Ev1 and Ev2 is added to Sv1 to calculate sensitivity 2 (Sv2), that is, gain correction is performed (step S23).

  You may subtract some delta Ev of the sensor with the smaller value of delta Ev2 and delta Ev3 to Sv2. For example, when delta Ev2> delta Ev3, delta Sv2 is set by the following equation (4).

      Sv2 = Sv2-Delta Ev3 / 2 (4)

  By executing the processing of step S21 and step S23, even when subjects with different luminance are photographed by the sensor A and the sensor B, it is possible to approach the appropriate exposure for each sensor.

  Tv1 and Sv1 are set to the sensor having the larger values of delta Ev2 and delta Ev3, and Tv1 and Sv2 are set to the sensor having the smaller values of delta Ev2 and delta Ev3 (step S24).

  By making the shutter speeds of the sensor A and the sensor B the same, it is possible to satisfactorily connect to a moving subject extending over the sensor A and the sensor B. Further, at this time, by making the upper and lower readout directions of the sensors on both sides the same, the exposure timings can be matched and a better moving subject can be joined. In addition, by making the scanning directions of the solid-state imaging device coincide with each other, each captured image can be easily joined. In other words, by matching the scanning direction and order of the respective solid-state imaging devices at the portions where they are connected to each other, an effect can be obtained in connecting objects at the boundaries of the cameras, particularly moving objects. For example, when the upper left part of the captured image captured by the solid-state image sensor 22A and the lower left part of the captured image captured by the solid-state image sensor 22B coincide with each other as a part where the images are joined, the solid-state image sensor 22A The scanning is performed from right to left from the top to the bottom of the solid-state imaging device. On the other hand, the solid-state image sensor 22B scans from the right to the left from the bottom to the top of the solid-state image sensor. In this way, by controlling the scanning direction of each solid-state imaging device based on the portion where the images are to be joined, an effect can be obtained in the joining.

  Next, the sensitivity difference between sensor A and sensor B is corrected. For the image output from a sensor with low sensitivity, the center is set to 1 time, the outermost periphery is set to k times, and the distance between the center and the outermost periphery is obtained by linear interpolation according to the distance from the image center, that is, the sensitivity Difference correction is performed (step S25). k is 1 or more, and the gain magnification is increased from the center to the outermost periphery. Here, the calculation method of k is as follows.

  For example, when delta Ev2> delta Ev3, k is obtained by the following equation (5).

      k = Sv2 / Sv1 (5)

  Thereby, when the images of the sensor A and the sensor B having different sensitivity differences are joined together, the brightness difference of the joined portions does not appear, and a favorable exposure can be achieved.

  FIG. 9 is a block diagram schematically showing a portion related to the present embodiment in the internal configuration of the CPU 130 in the processor 100 in FIG. In the CPU 130, a first photometric value calculation unit 51, a second photometric value calculation unit 52, a first exposure value calculation unit 53, in order to realize the processing shown in FIG. 5 and the processing shown in FIG. A second exposure value calculation unit 54, an exposure control value setting unit 55, and an exposure difference correction unit 56 are provided.

  The first photometric value calculation unit 51 calculates a photometric value based on imaging signals acquired from a plurality of solid-state imaging devices (for example, the solid-state imaging devices 22A and 22B). The second photometric value calculation unit 52 calculates a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor alone among the plurality of solid-state image sensors. Note that the CPU 130 controls the solid-state imaging devices 22A and 22B by a control command. Conversely, the first photometric value calculation unit 51 and the second photometric value calculation unit 52 receive the imaging signals from the solid-state imaging devices 22A and 22B. The photometric value can be calculated based on this.

  The first exposure value calculation unit 53 calculates an exposure value from at least one of the photometry value calculated by the first photometry value calculation unit 51 and the photometry value calculated by the second photometry value calculation unit 52. In this case, the first exposure value calculator 53 may calculate the exposure value based on the maximum photometric value obtained by the second photometric value calculator 52.

  The second exposure value calculator 54 calculates an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculator 52.

  The exposure control value setting unit 55 sets a solid-state image sensor that is larger than the photometric value calculated by the first photometric value calculator 51 among the photometric values calculated by the second photometric value calculator 52 as a first solid-state image sensor. When a solid-state imaging device other than the second solid-state imaging device is used, an exposure control value is set for the first solid-state imaging device based on the exposure value calculated by the first exposure value calculation unit 53, and the second solid-state imaging device is used. For the image sensor, an exposure control value is set based on the exposure value calculated by the second exposure value calculation unit 54.

  The exposure difference correction unit 56 corrects the exposure difference from the first solid-state image sensor for the second solid-state image sensor.

  The flow shown in FIGS. 5 and 8 can be realized by a computer-executable program written in a legacy programming language such as an assembler, C, C ++, C #, Java (registered trademark) or an object-oriented programming language. . And stored in a recording medium such as ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, etc. it can.

DESCRIPTION OF SYMBOLS 10 Omnisphere imaging system 12 Image pick-up body 20A, 20B Lens optical system 22A, 22B Solid-state image sensor (sensor A, B)
51 1st photometric value calculation part (1st photometric value calculation means)
52 Second photometric value calculation unit (second photometric value calculation means)
53 1st exposure value calculation part (1st exposure value calculation means)
54 2nd exposure value calculation part (2nd exposure value calculation means)
55 Exposure control value setting section (exposure control value setting means)
56 Exposure difference correction unit (exposure difference correction means)
100 Digital still camera processor (processor)
108A, 108B ISP
118 Distortion Correction / Image Synthesis Block 120 3-Axis Acceleration Sensor 130 CPU
138 SDRAM
144 Flash ROM

Japanese Patent No. 48699795

Claims (10)

An image pickup apparatus that joins captured images acquired from a plurality of solid-state image sensors to form and output a single image,
First photometric value calculating means for calculating a photometric value based on imaging signals acquired from the plurality of solid-state imaging devices;
Second photometric value calculating means for calculating a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor among the plurality of solid-state image sensors;
First exposure value calculating means for calculating an exposure value from at least one of the photometric value calculated by the first photometric value calculating means and the photometric value calculated by the second photometric value calculating means;
Second exposure value calculation means for calculating an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculation means,
Among the photometric values calculated by the second photometric value calculating means, a solid-state image sensor that is larger than the photometric value calculated by the first photometric value calculating means is defined as a first solid-state image sensor, and the other solid-state image sensors are designated as second solid-state image sensors. A solid-state image sensor,
For the first solid-state image sensor, an exposure control value is set based on the exposure value calculated by the first exposure value calculator, and for the second solid-state image sensor, the second exposure value calculator Exposure control value setting means for setting an exposure control value based on the exposure value calculated by
An image pickup apparatus comprising: an exposure difference correction unit that corrects an exposure difference with respect to the second solid-state image pickup element.   The imaging apparatus according to claim 1, wherein the first exposure value calculating unit calculates an exposure value based on a maximum value of the photometric value obtained by the second photometric value calculating unit.   The imaging apparatus according to claim 1, wherein the exposure control value setting unit sets different gains by matching shutter speeds set for the respective solid-state imaging devices.   The imaging apparatus according to claim 1, wherein the exposure difference correction unit increases the gain magnification from the center to the periphery. An image pickup apparatus that joins captured images acquired from a plurality of solid-state image sensors to form and output a single image,
First photometric value calculating means for calculating a photometric value based on imaging signals acquired from the plurality of solid-state imaging devices;
Second photometric value calculating means for calculating a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor among the plurality of solid-state image sensors;
First exposure value calculating means for calculating an exposure value from at least one of the photometric value calculated by the first photometric value calculating means and the photometric value calculated by the second photometric value calculating means;
Second exposure value calculation means for calculating an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculation means,
Among the photometric values calculated by the second photometric value calculating means, a solid-state image sensor that is larger than the photometric value calculated by the first photometric value calculating means is defined as a first solid-state image sensor, and the other solid-state image sensors are designated as second solid-state image sensors. A solid-state image sensor,
For the first solid-state image sensor, a shutter speed and a gain are set based on the exposure value calculated by the first exposure value calculator, and for the second solid-state image sensor, the second exposure value is calculated. Exposure control value calculating means for setting the gain by matching the shutter speed with the first solid-state imaging device based on the exposure value calculated by the means;
Read direction setting means for setting the read direction for each solid-state imaging device;
An image pickup apparatus comprising: an exposure difference correction unit that corrects an exposure difference with respect to the second solid-state image pickup element. An imaging method for joining together captured images acquired from a plurality of solid-state image sensors to form and output a single image,
A first photometric value calculating step of calculating a photometric value based on an imaging signal acquired from the plurality of solid-state imaging devices;
A second photometric value calculation step of calculating a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor among the plurality of solid-state image sensors;
A first exposure value calculating step of calculating an exposure value from at least one of the photometric value calculated by the first photometric value calculating step and the photometric value calculated by the second photometric value calculating step;
A second exposure value calculation step of calculating an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculation step,
Among the photometric values calculated in the second photometric value calculating step, a solid-state image sensor larger than the photometric value calculated in the first photometric value calculating step is used as the first solid-state image sensor, and the other solid-state image sensors are used as the second solid-state image sensor. When a solid-state image sensor is used,
For the first solid-state image sensor, an exposure control value is set based on the exposure value calculated by the first exposure value calculation step, and for the second solid-state image sensor, the second exposure value calculation step. An exposure control value calculation step for setting an exposure control value based on the exposure value calculated by
An imaging method comprising: an exposure difference correction step for correcting an exposure difference with respect to the second solid-state imaging element.   The imaging method according to claim 6, wherein the first exposure value calculating step calculates an exposure value based on a maximum value of the photometric values obtained by the second photometric value calculating step.   8. The imaging method according to claim 6, wherein the exposure control value calculation step sets different gains by matching shutter speeds set for the respective solid-state imaging devices.   The imaging method according to any one of claims 6 to 8, wherein the exposure difference correction step increases the gain magnification from the center to the periphery. A computer-readable recording medium recording a program for connecting a captured image acquired from a plurality of solid-state image sensors to a computer included in an apparatus having an image processing function to form and output a single image. ,
A first photometric value calculation process for calculating a photometric value based on imaging signals acquired from the plurality of solid-state imaging devices;
A second photometric value calculation process for calculating a photometric value corresponding to each solid-state image sensor based on an image signal acquired by each solid-state image sensor among the plurality of solid-state image sensors;
A first exposure value calculating process for calculating an exposure value from at least one of the photometric value calculated by the first photometric value calculating process and the photometric value calculated by the second photometric value calculating process;
A second exposure value calculation process for calculating an exposure value corresponding to each solid-state image sensor from the photometric value of each solid-state image sensor calculated by the second photometric value calculation process,
Among the photometric values calculated by the second photometric value calculation process, the solid-state image sensor that is larger than the photometric value calculated by the first photometric value calculation process is used as the first solid-state image sensor, and the other solid-state image sensors are used as the second solid-state image sensor. When a solid-state image sensor is used,
For the first solid-state image sensor, an exposure control value is set based on the exposure value calculated by the first exposure value calculation process, and for the second solid-state image sensor, the second exposure value calculation process is set. Exposure control value calculation processing for setting an exposure control value based on the exposure value calculated by
A computer-readable recording medium storing a program for executing image processing, wherein the second solid-state imaging device includes an exposure difference correction process for correcting an exposure difference from the first solid-state imaging device. recoding media.