JP6501619B2 - Imaging apparatus, imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method for performing image composition processing on image data of a plurality of frames to calculate composite image data of one frame.

  2. Description of the Related Art Conventionally, in a single-lens reflex imaging apparatus, observation of an object image is performed by an optical finder. On the other hand, in recent years, an imaging device for observing an object image by live view display without using an optical finder has become widespread. Here, live view is a method of displaying an image read from an image sensor on a liquid crystal monitor or the like in real time. Moreover, in addition to the liquid crystal monitor, an imaging device further equipped with an electronic viewfinder is also sold, and in such an imaging device, switching between observation via the liquid crystal monitor and observation via the electronic viewfinder is possible. Then, when observing the live view image through the electronic viewfinder, it is possible to observe in a sense closer to the optical finder than observing the live view image through the liquid crystal monitor.

  However, conventionally, either when using an optical finder or using an electronic viewfinder, it is possible to read out an image signal from an image sensor while performing long-time exposure such as bulb photography. Because it was not, it was not possible to confirm the exposure state of the subject in the middle of the photographing, and the confirmation of the image was after the photographing ended. For this reason, the photographer estimates the exposure setting such as the exposure time by the photographer himself based on the brightness of the subject and the like, and starts the exposure and ends the exposure. For this reason, it is not easy to obtain a desired captured image without failing to capture due to underexposure or overexposure.

  Therefore, for example, in Japanese Patent Application Laid-Open No. 2005-117395, an image obtained by reading pixel signals from an image sensor at predetermined time intervals and simply accumulating the image signals every time they are read from the image sensor is used as a liquid crystal monitor. An imaging device for display is described. According to such an imaging device, since the progress is displayed during long time imaging such as bulb imaging, imaging failure can be reduced.

  Further, Japanese Patent No. 4148586 describes an imaging device that generates a valve photographed image by continuously reading out an image signal from an image sensor and subjecting the read out image signal to comparative bright combination processing. Here, the comparison bright combining process is an image combining process in which the pixel values of the pixels forming the image data are compared for each pixel, and the larger pixel value is selected to be the combined pixel value.

  By the way, various noises occur as characteristics of the image sensor. For example, in a photodiode provided in each pixel of an image sensor, dark current noise caused by dark current, dark current shot noise caused by dark current noise, light shot noise generated when photoelectrically converting incident light Etc. Furthermore, circuit noise such as reset noise and amplifier noise also occurs in the read circuit. These noises cause the pixel values to differ for each pixel, line, or row as random noise or fixed pattern noise, and appear in the image as pixel value unevenness (shading) in the image plane.

  The amount of noise of these noises changes in accordance with the imaging conditions, and the amount of noise changes in accordance with what kind of image combining processing is used when image combining is performed. For example, for dark current noise, the amount of noise tends to increase as the temperature of the image sensor increases and as the exposure time (shutter speed) increases. Furthermore, each noise mentioned above becomes so large that the ISO sensitivity setting of an imaging device is high.

  In addition, in the case of using additive composition for image composition processing, the amount of noise increases as noise is integrated as the number of composite images increases. In addition, in the case of using the addition average combining, the fixed pattern noise included in each image to be combined does not basically change the noise amount even after the image combining. On the other hand, since the random noise is averaged as the number of combined images in the addition average combination increases, the amount of noise decreases. Furthermore, when using comparative bright synthesis or comparative dark synthesis, the solid pattern noise does not change basically as in the case of the addition average synthesis. On the other hand, random noise is the largest value among pixel values that change randomly in the case of comparative light combination, and the smallest value among pixel values that change randomly in the case of relative dark composition. Since the pixel value of each pixel converges to the value, the variation of the pixel value for each pixel tends to be small.

  Furthermore, in the bulb imaging, in order to correct fixed pattern noise due to dark current, FPN (Fixed Pattern Noise: fixed pattern noise) cancel processing may be performed. In this FPN cancellation process, after the optical shutter is opened and the bright image is taken, the optical shutter is closed and the dark image is automatically taken with the same exposure time as the bright image, and the image provided in the subsequent stage of the image sensor This processing is to correct fixed pattern noise by subtracting dark image data from bright image data by the processing circuit. When this FPN cancellation processing is performed, fixed pattern noise is corrected, but random noise increases because noise in the bright image and noise in the dark image are accumulated.

  Then, when combining images with different shooting conditions (that is, different amounts of noise), not only random noise but also fixed pattern noise amount fluctuates for each image, so the noise amount changes more complicatedly. It will be.

  As described above, the amount of noise of the combined image differs according to the imaging conditions and the image combining process.

  On the other hand, when photographing, the photographer sensually judges the noise feeling related to the image quality of the composite image. For example, in the technique described in the above-mentioned JP-A-2005-117395, it is possible in principle to confirm the image quality of the image being synthesized to display the progress of the image synthesis on a liquid crystal monitor or the like. In a liquid crystal monitor attached to an imaging device or a smartphone wirelessly connected to the imaging device, for example, it is difficult to confirm noise for each pixel in detail because the screen is small. Also, since the main subject of bulb shooting is, for example, a starry sky, fireworks, fireflies, etc., there are many dark places at night where bulb shooting is performed. Since the visibility of noise is different from that when shooting in the daytime or in a bright room after shooting, it is possible to accurately determine whether the amount of noise the photographer is satisfied with is the dark place at the time of shooting. difficult.

  It is considered that the photographer who places importance on the amount of noise of the composite image may decide the timing to finish the photographing depending on how much the amount of noise after combination is, as described above, Since the amount of noise can not be grasped while photographing an image, it is difficult to accurately determine the timing of the end of photographing.

  JP-A-2005-20562 describes a technique for changing the noise correction process according to the amount of noise calculated based on the pixel value output from the OB (optical black) pixel of the image sensor. . However, with the technique described in this publication, it is not possible for the photographer to grasp the amount of noise during shooting, and in this publication it is necessary to accurately calculate the amount of noise when performing various image combining processes as described above. Is not considered.

JP 2005-117395 A Patent No. 4148586 JP 2005-20562 A

  Thus, in the prior art, it has been difficult to accurately grasp the amount of noise of one frame of composite image data calculated by performing image composition processing on a plurality of frames of image data during imaging.

  The present invention has been made in view of the above circumstances, and it is possible to accurately grasp the amount of noise of one frame of composite image data calculated by performing image composition processing of a plurality of frames of image data during shooting. An object of the present invention is to provide an apparatus and an imaging method.

  An imaging device according to an aspect of the present invention includes an effective pixel group for generating image data related to an optical image of a subject, and a light shielding pixel for shielding and arranging light shielding pixel data around the effective pixel group. The image combining process is performed on the image data of a plurality of frames sequentially read out from the effective pixel group, and an image sensor having a group to calculate composite image data of one frame, and sequentially from the light shielding pixel group Amount of noise based on the light-shielded pixel data subjected to the image synthesis processing, and an image synthesis unit performing the same image synthesis processing as the image data on the light-shielded pixel data of a plurality of frames read And the image combining unit and the noise amount calculating unit read out the image data and the light-shielded pixel data of one frame from the image pickup device. Performing the above-mentioned processing to.

  An imaging method according to an aspect of the present invention includes the steps of: reading image data relating to an optical image of a subject from an effective pixel group, and reading out light shielding pixel data from a light shielding pixel group arranged by shielding light around the effective pixel group; Each time the image data of one frame and the light-shielded pixel data are read, the image combining process is performed on the image data of a plurality of frames sequentially read from the effective pixel group to calculate one image of composite image And performing the same image composition processing as the image data on the plurality of light-shielded pixel data sequentially read from the light-shielded pixel group, the image data of one frame, and the light shield Calculating a noise amount based on the light-shielded pixel data subjected to the image synthesis process each time the pixel data is read.

  According to the imaging apparatus and imaging method of the present invention, it is possible to accurately grasp the amount of noise of one frame of composite image data calculated by performing image composition processing on a plurality of frames of image data during shooting.

FIG. 1 is a block diagram showing a configuration of an imaging device in Embodiment 1 of the present invention. FIG. 2 is a diagram showing a pixel configuration of the image sensor in the first embodiment. 6 is a flowchart showing a flow of processing of capturing and combining images of a plurality of frames in the imaging device of the first embodiment. FIG. 8 is a diagram showing an example of displaying the ratio of the current noise amount to the noise amount at the start of shooting as the image quality index in the first embodiment. FIG. 8 is a view showing an example of displaying as an image quality index how much the current ISO sensitivity has changed to a value corresponding to the ISO sensitivity at the start of shooting in the first embodiment. FIG. 8 is a view showing an example of displaying as an image quality index how many noise levels have been improved as an EV value based on a change from ISO sensitivity at the start of shooting to current ISO sensitivity in the first embodiment. FIG. 8 is a view showing an example of displaying as an image quality index how much the current dynamic range has changed from the dynamic range at the start of shooting in the first embodiment. FIG. 5 is a block diagram showing the configuration of an imaging device in Embodiment 2 of the present invention. 12 is a flowchart showing a flow of processing of capturing and combining images of a plurality of frames in the imaging device of the second embodiment. FIG. 13 is a view showing a display example when the target image quality is set by the ISO sensitivity in the second embodiment. FIG. 13 is a view showing a display example when the target image quality is set by the noise improvement amount in the second embodiment. FIG. 13 is a view showing a display example when the target image quality is set by the number of stages of the improvement amount of the dynamic range in the second embodiment. 12 is a flowchart showing a flow of processing for capturing and combining images of a plurality of frames in the imaging device of Embodiment 3 of the present invention. The timing chart which shows operation of each part in the imaging device of Embodiment 4 of the present invention. 14 is a flowchart showing a flow of processing of capturing and combining images of a plurality of frames in the imaging device of the fourth embodiment. 14 is a flowchart showing noise amount calculation processing after FPN cancellation in the imaging device of the fourth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Embodiment 1

  FIGS. 1 to 7 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing a configuration of an imaging device 1. Note that FIG. 1 mainly shows the electrical configuration of the imaging device 1.

  The imaging device 1 of the present embodiment includes an imaging unit 2, a bus 3, an internal memory 4, an image processing unit 5, a noise amount calculation unit 6, a display unit 7, and an input IF (Interface: Interface) 8. , And system control unit 10. Although the external memory 9 is described in the imaging device 1 shown in FIG. 1, the external memory 9 may be a memory card or the like that can be attached to and detached from the imaging device 1 as described later. The configuration does not have to be unique to the imaging device 1.

  In the present embodiment, it is assumed that the imaging device 1 is, for example, a digital camera (a compact digital camera, a digital single-lens camera, etc.), but the imaging device 1 may be a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone, a smartphone, a personal digital assistant (PDA), a game device or the like. Furthermore, the present invention is not limited to these, and the imaging apparatus 1 can be widely applied as long as it is a device for photographing that can perform long-time exposure (here, “exposure” is also referred to as “exposure”).

  The imaging unit 2 includes a lens 11, a mechanical shutter 12, and an image sensor 13.

  The lens 11 forms an optical image of a subject on the image sensor 13. The lens 11 generally includes a focus lens for performing focus adjustment and a stop for adjusting the amount of light reaching the image sensor 13. Here, when the aperture is adjusted, the aperture value related to the exposure amount changes. In the present embodiment, it is assumed that the lens 11 is integrally formed with the imaging device main body. However, the lens 11 may of course be configured as an interchangeable lens detachable from the imaging device main body.

  The mechanical shutter 12 is an optical shutter that controls the arrival and non-arrival of the optical image of the subject from the lens 11 to the image sensor 13 by opening and closing, and also controls the exposure time (shutter speed).

  The image sensor 13 is an imaging device in which a plurality of pixels are arranged in a two-dimensional manner and converts an optical image of an object formed by the lens 11 into an electrical signal for each pixel to generate image data. , CMOS image sensor, CCD image sensor, etc. Image data generated by the image sensor 13 is output to the image processing unit 5 and the bus 3.

  Here, FIG. 2 is a view showing a pixel configuration of the image sensor 13.

  The image sensor 13 includes an effective pixel group having an effective pixel 13a for generating image data related to an optical image of a subject as a component, and light shielding pixel data generated around the effective pixel group. And a light shielding pixel group having the OB pixel 13b, which is a light shielding pixel, as a component.

  As shown in FIG. 2, in the light-shielded pixel group, (horizontal) OB (Optical Black: optical black) pixels 13b are arranged on the same horizontal line as each horizontal line of the effective pixel group along, for example, the left side of the effective pixel 13a. And a vertical light shielding pixel group in which (vertical) OB pixels 13b are arranged along the same vertical line as the vertical lines of the effective pixel group along, for example, the upper side of the effective pixel 13a. And.

  Here, although each of the effective pixel 13a and the OB pixel 13b includes a PD (photodiode) for converting light into an electric signal, when the mechanical shutter 12 is in the open state, the effective pixel 13a is The light of the subject image formed by the lens 11 arrives at the lens, while the light shielding film is provided on the OB pixel 13b which is the light shielding pixel. The light does not reach. Therefore, the OB pixel 13 b always outputs the light-shielded pixel data regardless of the opening and closing of the mechanical shutter 12.

  The black reference signal level in the image data output from the effective pixel 13a fluctuates due to the temperature of the image sensor 13, exposure time (charge accumulation time), voltage fluctuation of peripheral circuits, and the like. For this purpose, the light-shielded pixel data from the OB pixel 13b output simultaneously with the image data from the effective pixel 13a is used as the black reference signal level of the image data from the effective pixel 13a.

  The bus 3 is a signal line through which each unit in the imaging device 1 transmits and receives a signal. For example, the imaging unit 2, the internal memory 4, the image processing unit 5, the noise amount calculating unit 6, the display unit 7, the input IF 8, the external memory 9, and the system control unit 10 are connected to the bus 3.

  The internal memory 4 stores information related to the imaging device 1 and includes nonvolatile memory such as flash memory or volatile memory such as SDRAM. Here, in the non-volatile memory, for example, a processing program for the system control unit 10 to perform a process of controlling the imaging device 1 or various setting information necessary for the operation of the imaging device 1 is stored. The volatile memory temporarily stores image data and the like during image processing.

  The image processing unit 5 performs various types of image processing on image data output from the image sensor 13 and stored in the internal memory 4 as needed. The image combining unit 15 and FPN (Fixed Pattern Noise: fixed) A pattern noise) cancellation processing unit 16, a pixel defect correction unit 17, and a development processing unit 18 are provided.

  The image composition unit 15 performs image composition processing on image data of a plurality of frames read out sequentially (also expressed as “time-sequentially” or “continuously”) from the effective pixel group and combines one frame The same image combining processing as performed on the image data is performed on the light-shielded pixel data of a plurality of frames sequentially read from the light-shielded pixel group while calculating the image data. It is performed each time data and light-shielded pixel data are read out. Here, the image combining process performed by the image combining unit 15 is a process of combining pixel values for each corresponding pixel position.

  Here, as a specific example, a configuration in which the image combining unit 15 includes the comparison bright combining unit 21, the comparison dark combining unit 22, the addition combining unit 23, and the average combining unit 24 will be described. Alternatively, instead of some of these, a unit that performs other image combining processing may be provided. In the following, processing to be performed by the comparative bright combining unit 21, the comparative dark combining unit 22, the addition combining unit 23, and the average combining unit 24 mainly on the image data read from the effective pixel 13a will be described. However, as described above, the same process is performed on the light-shielded pixel data read from the OB pixel 13b.

  The comparative bright combining unit 21 performs the following cumulative comparative bright combining processing. The image data generated on the basis of the image data read out from the image sensor 13 first (this “image data generated on the basis of the image data read out from the image sensor 13” is an image in this embodiment. Although so-called RAW image data read from the sensor 13 is assumed, the present invention is not limited to this, and image data subjected to processing by the FPN cancellation processing unit 16, the pixel defect correction unit 17, the development processing unit 18, etc. Or the like) is first stored in the internal memory 4 as comparative bright composite image data.

  Next, when the image data is read out from the image sensor 13, the comparative bright combining unit 21 stores the pixel data forming the image data generated based on the read out image data and the internal memory 4. Comparison The pixel data constituting the composite image data is compared at each corresponding pixel position. Then, the pixel data of the corresponding pixel position is used as the pixel data of the corresponding pixel position to reconstruct the comparison bright composite image data.

  By repeating such processing each time image data is read out from the image sensor 13, the latest comparative bright composite image data is stored in the internal memory 4. If such comparative brightening processing is performed, for example, when photographing an astronomical photograph, it is possible to obtain an image of a light trace of a star in the night sky. At this time, the brightness of the background is constant regardless of the exposure time, and the length of the light trace can be adjusted according to the exposure time.

  The comparison dark synthesis unit 22 performs the following cumulative comparison dark synthesis processing. First, image data generated based on the image data read out from the image sensor 13 is stored in the internal memory 4 as comparative dark composite image data.

  Next, when the image data is read out from the image sensor 13, the comparison / dark synthesis unit 22 stores the pixel data forming the image data generated based on the read image data and the internal memory 4. The pixel data constituting the comparison dark composite image data is compared at each corresponding pixel position. Then, the pixel data of the corresponding pixel position is used as the pixel data of the corresponding pixel position to reconstruct comparative dark composite image data, whichever smaller (that is, the darker) of the comparison result is used.

  The latest comparative dark composite image data is stored in the internal memory 4 by repeatedly performing such processing each time image data is read out from the image sensor 13. If such comparative dark synthesis processing is performed, for example, when photographing an astronomical photograph, it is possible to obtain a background-only image in which the light trace of a star moving in the night sky is erased.

  The adding and combining unit 23 performs the following cumulative addition and combining process. Image data generated on the basis of the image data initially read from the image sensor 13 is first stored in the internal memory 4 as addition composite image data.

  Next, when the image data is read out from the image sensor 13, the addition synthesis unit 23 adds pixel data constituting the image data generated based on the read image data and the addition stored in the internal memory 4. The pixel data forming the composite image data is added at each corresponding pixel position. Then, the addition combined image data is reconstructed by using the added pixel data as the pixel data of the corresponding position.

  The latest addition combined image data is stored in the internal memory 4 by repeatedly performing such processing each time image data is read from the image sensor 13. If such addition / combination processing is performed, for example, when photographing an astronomical photograph, the latest addition / combination image data is displayed on the display unit 7 as needed in order to increase the brightness of the image each time it is added. This makes it possible to visually check the progress of the image change during bulb imaging.

  The average combination unit 24 performs the following cumulative addition average combination processing. Image data generated on the basis of the image data read out from the image sensor 13 first is first stored in the internal memory 4 as average composite image data.

  Next, when the image data is read out from the image sensor 13, the average combining unit 24 calculates the pixel data constituting the image data generated based on the read image data and the average stored in the internal memory 4. The pixel data making up the composite image data is added and averaged for each corresponding pixel position. Then, the average combined image data is reconstructed by using the pixel data obtained by the averaging as the pixel data at the corresponding position.

  The averaging at this time is stored in the internal memory 4 because the weight increases as the image data read at a later time from the image sensor 13 if the averaging is simply added and averaged. Assuming that the average composite image data is the result of combining n (n is an integer of 1 or more) images, the average composite image data stored in the internal memory 4 is weighted by n / (n + 1) It is preferable to perform weighted average processing of weighting the image data newly read out from the sensor 13 by 1 / (n + 1).

  The latest average composite image data is stored in the internal memory 4 by repeatedly performing such processing each time image data is read from the image sensor 13. Then, the average combined image data is reconstructed by using the pixel data obtained by the averaging as the pixel data at the corresponding position. If such averaging processing is performed, for example, when photographing an astronomical photograph, random noises are averaged and reduced, so that an image with low noise and high image quality can be obtained.

  The FPN cancellation processing unit 16 subtracts the dark time image data (image data obtained by closing the mechanical shutter 12 and obtained by closing the mechanical shutter 12) from the light time image data (image data obtained by opening the mechanical shutter 12 and obtaining the image data). To correct fixed pattern noise.

  The pixel defect correction unit 17 replaces the pixel value of a defective pixel (a pixel that outputs an abnormal pixel value different from a normal pixel) by the average value of pixel values of a plurality of pixels in the vicinity of the defective pixel. , Correction so that defective pixels do not appear in the image. Here, the defective pixel is inspected in advance, for example, at the time of production of the imaging device 1 and registered in advance in the internal memory 4 as a defective pixel address. Alternatively, even after shipment of the imaging device 1, the correlation with the peripheral pixels of each pixel in the image data obtained by photographing is calculated, and pixels that are determined to be defective pixels without correlation are May be additionally registered as a defective pixel of.

  The development processing unit 18 processes the RAW image data obtained from the imaging unit 2 or the RAW image data obtained from the imaging unit 2 as a development process on composite RAW image data obtained by combining the image combining unit 15. Perform image processing such as mosaicing, white balance adjustment, noise reduction, gamma correction, YC signal generation, and resizing. Here, the resizing process is a process for adjusting the number of pixels of the image data read from the image sensor 13 to the number of display pixels of the display unit 7. The image processing may be further performed by the development processing unit 18.

  The noise amount calculation unit 6 calculates the noise amount based on the light-shielded pixel data subjected to the image synthesis process every time image data of one frame and the light-shielded pixel data are read out from the image sensor 13. Specifically, the noise amount calculation unit 6 calculates the variance or standard deviation of the pixel values of the plurality of OB pixels 13b belonging to the light shielding pixel group as the noise amount. The noise amount calculation unit 6 further functions as an index calculation unit that calculates an image quality index representing the image quality of the composite image data from the calculated noise amount. This image quality index is a useful index for the photographer to judge the image quality of the composite image data, and some specific examples will be described later with reference to FIGS.

  The display unit 7 displays an image developed by the development processing unit 18 and various information related to the imaging device 1. For example, a TFT (Thin Film Transistor) liquid crystal, an organic EL (Electro Luminescence), etc. It has a display. As a specific configuration example of the display unit 7, a rear display unit of the imaging device 1 or an EVF (Electronic View Finder) may be mentioned. The latest composite image obtained by the image combining unit 15 is developed by the development processing unit 18 each time new image data is read out from the image sensor 13 and then displayed on the display unit 7 to obtain an image being captured. Follow up on the Further, the image quality index calculated by the noise amount calculation unit 6 described above is also displayed on the display unit 7 and can be used by the photographer to judge the image quality.

  The input IF 8 is an operation unit for the photographer to perform an input operation to the imaging device 1, and a power button for turning on / off the power of the imaging device 1, a release button for operating the start and end of shooting, etc. And a touch panel or the like for performing a touch input operation provided in association with the rear display unit or the like. The photographer operates the input IF 8 to perform various mode settings of the imaging apparatus 1 and an imaging operation instruction such as release.

  The external memory 9 is a non-volatile storage medium that is removable from the imaging device 1 or fixed inside the imaging device 1. Examples of the external memory 9 removable from the imaging device 1 include memory cards such as an SD memory card and a CF card. Image data (including composite image data subjected to development processing) or RAW image data (including composite RAW image data) subjected to development processing by the development processing unit 18 is recorded in the external memory 9, and this external memory 9 is reproduced. The recorded image data is read out from 9.

  The system control unit 10 has, for example, a CPU (Central Processing Unit), and controls the entire imaging apparatus 1 by controlling the respective units in accordance with a processing program stored in the internal memory 4. For example, when an instruction from the photographer is received via the input IF 8, the system control unit 10 performs timing control such as charge accumulation start and signal readout of the image sensor 13, opening / closing timing control of the mechanical shutter 12, and aperture control of the lens 11. And auto focus control. Further, the system control unit 10 receives image data from the image processing unit 5 and performs control of causing the display unit 7 to display an image, control of storing the image data in the external memory 9, and the like.

  Next, FIG. 3 is a flowchart showing a flow of processing for capturing and combining images of a plurality of frames in the imaging device 1. The process shown in FIG. 3 is executed by the system control unit 10 controlling the respective units in accordance with the processing program stored in the internal memory 4.

  The composition processing shown in FIG. 3 is not shown when the photographer selects the bulb imaging mode by the input IF 8 and selects the composition mode (mode for displaying the progress during exposure) in the bulb imaging mode. The main processing is executed as a subroutine or the like.

  Therefore, even if the bulb photographing mode is selected, the description of the case where the normal bulb photographing which does not display the progress during exposure is selected is omitted. What is described with reference to FIG. 3 is a process of displaying a latest combined image as a shooting progress by sequentially combining the image data read out in time series on a frame basis.

  When the photographer selects the combination mode, the photographer may further select one or more of the comparison combination mode, the comparison dark combination mode, the addition average combination mode, and the addition combination mode as more detailed combination modes. It can be done. Here, two or more combination modes can be selected at the same time, and when two or more are selected, parallel processing is performed inside the imaging device 1.

  When the process shown in FIG. 3 is started, first, the mechanical shutter 12 is opened, and the image processing unit 5 sequentially processes the image data read from the image sensor 13 at a predetermined frame rate. Live view display is performed in real time (step S1).

  When this live view display is made, the focus position is, for example, automatically adjusted. Then, the image pickup apparatus is configured so that the photographer can confirm the live view image (if the optical finder is further provided, the object image may be confirmed by the optical finder) and the subject to be photographed can be photographed. The composition is determined by adjusting the direction of 1 and the focal length (zoom) of the lens 11. In addition, even while performing this live view, the photographer should set the composition mode and more detailed composition mode via the operation button of the input IF 8 or the touch panel as necessary. Can.

  Next, the system control unit 10 determines the state of the first release switch that is turned on when the release button is pressed halfway, and continues to perform live view display in step S1 until it is turned on, and then turned on. If it is determined, processing is performed when the first release is performed (step S2).

  Here, for example, AF (automatic focus control) and AE (automatic exposure control) are performed as processing when the first release is performed. Here, when using, for example, contrast AF as AF, the contrast is extracted from the image data repeatedly read out from the image sensor 13, and drive control of the focus lens in the lens 11 is performed so that the extracted contrast becomes the maximum value. Adjust the focus position. Further, AE is processing for automatically controlling the aperture value, the ISO sensitivity, and the exposure time T so that an image repeatedly read from the image sensor 13 has a proper exposure. Note that depending on the settings of the imaging device 1, one or both of AF and AE may be turned off, and the photographer manually sets the position (focus position), aperture value, ISO sensitivity, and exposure of the focus lens via the input IF 8 or the like. The time T may be set.

  After performing the first release process, the system control unit 10 detects the state of the second release switch that is turned on when the release button is fully pressed, and determines whether the second release switch is on or not ( Step S3).

  Here, when it is determined that the second release switch is not turned on, the process returns to step S1 and the above-described processing is repeated.

  When it is determined that the second release switch is on, the system control unit 10 opens the mechanical shutter 12 and starts charge accumulation (that is, exposure) of the image of the first frame by the image sensor 13, Further, the timer incorporated in the system control unit 10 is reset to start the timing operation of the exposure time T (step S4).

  Thereafter, the system control unit 10 determines whether or not the exposure time T has elapsed from the counting result of the timer (step S5). The exposure time T is a cycle (exposure time) for reading out an image of one frame from the image sensor 13, and is automatically set by AE in step S2 or manually set by the photographer manually. Then, when the exposure time T has not elapsed, while the exposure is continued, it waits for the exposure time T to elapse.

  When it is determined that the exposure time T has elapsed, the system control unit 10 reads the image signal from the image sensor 13 and immediately after the reading of the image signal is completed (that is, the blank time when charge accumulation is not performed is minimized). Exposure of the next frame is started, and the timer is reset to start the timing operation of the exposure time T (step S6). At this time, the mechanical shutter 12 is kept open, and the electronic shutter control of the image sensor 13 starts charge accumulation of the next frame.

  For this reason, it is possible to minimize the exposure omission between the images taken in continuously, and to minimize the interruption of the movement trajectory of the subject to be captured in the finally synthesized image. In a CMOS image sensor generally used as an image sensor 13 of a digital camera, the readout and exposure start can be sequentially controlled line by line, so that the exposure omission time between successive frames is This is about the read time of one line. Since this time is very short, for example, on the order of several tens to 100 microseconds, the final composite image is hardly recognized as an image in which the movement trajectory is broken.

  The system control unit 10 also stores the image signal read from the image sensor 13 in the internal memory 4 as digital image data (step S7). At this time, in addition to the image data read from the effective pixel 13 a of the image sensor 13 shown in FIG. 1, the light-shielded pixel data read from the OB pixel 13 b is also stored in the internal memory 4.

  After storing the image data in the internal memory 4, next, the system control unit 10 determines whether the image data to be processed is the image data of the first frame (step S8).

  Here, when it is determined that the image data of the first frame is not (that is, when it is determined that the image data of the second frame or later), it is read from the image sensor 13 and the internal The image combining unit 15 performs image combining processing on the image data stored in the memory 4 and the composite image already stored in the internal memory 4 (step S9).

  For example, if it is assumed that the image data of the second frame has just been read out, then the image data of the first frame is stored as a composite image in the internal memory 4 and the image data of the second frame is further obtained in step S7. Are stored, the image combining unit 15 performs an image combining process on these image data. Also when processing the n (n ≧ 3) frame and subsequent ones, the image combining process is similarly performed on the composite image from the 1st to (n-1) th frame and the image acquired as the nth frame. Become. Here, the image combining unit 15 is a pixel at the same pixel position for both the image data read from the effective pixel group including the effective pixels 13 a and the image data read from the light blocking pixel group including the OB pixels 13 b. The image combining process for combining the values is performed in the same manner.

  After that, when the image combining process in step S9 is performed, the image data as a result of the image combining process is obtained, and when it is determined in step S8 that the image data is the first frame, it is acquired as the first frame. The obtained image data is stored in the internal memory 4 as composite image data (step S10).

  Subsequently, pixel defect correction processing is performed on the composite image data stored in the internal memory 4 by the pixel defect correction unit 17, and the composite image data after processing is separated from the original composite image data (that is, step The original combined image data obtained by the image combining process of S9 is stored in the internal memory 4 (step S11). The pixel defect correction process is similarly performed on both the image data read from the effective pixel group and the image data read from the light-shielded pixel group.

  Further, the development processing unit 18 performs the development processing as described above on the composite image data subjected to the pixel defect correction, and stores the result in the internal memory 4 (step S12). This developing process is similarly performed on both the image data read from the effective pixel group and the image data read from the light-shielded pixel group. Further, although the composite image data subjected to the development processing is stored separately from the above-described original composite image data, the composite image data after the pixel defect correction processing may be overwritten.

  Then, the noise amount calculation unit 6 numerically analyzes the OB pixel 13b of the composite image data subjected to the development processing to calculate the noise amount (step S13). Specifically, the noise amount calculation unit 6 calculates, for example, the standard deviation (or variance) of the pixel values of the plurality of OB pixels 13 b included in the light shielding pixel group. This standard deviation represents the variation of the pixel value of the OB pixel 13b, and represents the amount of noise because the OB pixel 13b is shielded from light. Therefore, the calculated standard deviation indicates that the smaller the value, the smaller the amount of noise, and the larger the value, the larger the amount of noise.

  Although the example of calculating the noise amount of the composite image data subjected to the development processing has been described here, the noise amount of the RAW composite image data before the development processing may be calculated. When image data is stored in the external memory 9, the case where image data subjected to development processing is stored, the case where RAW image data is stored, and the case where image data subjected to development processing and RAW image data are stored, is there. Since the amount of noise differs depending on whether or not development processing is performed, it is preferable to calculate the amount of noise of the image data to be stored according to which storage setting is selected.

  For example, a photographer who is familiar with photographing selects recording of a RAW image and uses the RAW image stored in the external memory 9 later using a PC (personal computer) or the like, for example, by using commercially available image processing software. It may be developed according to preference. For such photographers, it is useful to calculate the amount of noise with respect to RAW image data and to display an image quality index (this image quality index will be described later) based on the calculated amount of noise.

  On the other hand, for a general photographer who does not store the RAW image, it is sufficient to calculate the noise amount of the image after development and display the image quality index.

  Furthermore, when both image data (generally, for example, JPEG data) after development processing and RAW image data are stored in the external memory 9, both noise amounts are calculated and both image quality indexes are calculated. It may be displayed, or only the image quality index for the image data selected by the photographer as the target for displaying the image quality index among the two image data to be stored may be displayed.

  Thereafter, the noise amount calculation unit 6 calculates the image quality index based on the noise amount obtained by numerical analysis (step S14), and the display unit 7 superimposes on the developed composite image to display the image quality index (Step S15). When displaying the composite image developed here, one or both of the RAW composite image data before development processing and the image data subjected to development processing are stored in the external memory 9 as image data in the middle of progress. You may do it.

  Here, with reference to FIG. 4 to FIG. 7, several examples of the image quality index displayed together with the composite image being captured will be described.

  First, FIG. 4 is a diagram showing an example in which the ratio of the current noise amount to the noise amount at the start of shooting is displayed as the image quality index.

  In this case, the noise amount calculation unit 6 calculates, as an image quality index, a numerical value indicating a change amount of the noise amount obtained by comparing the noise amount at the start of imaging and the noise amount of the current composite image.

  Then, on the screen 7 a of the display unit 7, the synthetic image being photographed which has been developed is displayed, and the image quality index is displayed. The image quality index shown in FIG. 4 is to display numerically the ratio of the current noise amount to the noise amount at the start of shooting. Therefore, the photographer can directly judge that the amount of noise has become half, for example.

  Next, FIG. 5 is a diagram showing an example of displaying, as an image quality index, how much the current ISO sensitivity has changed to a value corresponding to the ISO sensitivity at the start of shooting.

  The ISO sensitivity corresponds to the gain for the digital output of the pixel, and a photographer who is familiar with photography generally understands that the noise is also doubled when the ISO sensitivity is doubled, that is, the noise amount is doubled. doing.

  Therefore, the noise amount calculation unit 6 calculates the ISO sensitivity as an image quality index at the start of shooting and the ISO sensitivity as an image quality index of the current composite image.

  Specifically, the image quality index shown in FIG. 5 indicates how much the current ISO sensitivity has changed to a value corresponding to the ISO sensitivity at the start of shooting.

  In the example shown in FIG. 5, the ISO sensitivity at the start of shooting is 1600. Then, it is assumed that the current noise amount of a combined image obtained by combining images of a plurality of frames is improved to 1⁄4 at the start of shooting. In this case, the photographer can intuitively grasp the noise amount by displaying 1600/4 = 400, which is the ISO sensitivity at the start of shooting multiplied by the noise change amount, as an equivalent value of the current ISO sensitivity. it can.

  Furthermore, FIG. 6 is a diagram showing an example of displaying as an image quality index how many noise levels have been improved as an EV value based on a change from the ISO sensitivity at the start of shooting to the current ISO sensitivity.

Here, the number of stages is Tv (Time Value) for shutter speed, Av (Aperture Value) for aperture value, Bv (Brightness Value) for luminance, and Sv (Speed Value) for ISO sensitivity in APEX (Additive system of Photographic Exposure). EV value when expressed EV = Bv + Sv = Tv + Av
Is the unit of Assuming that the symbol "^" is used as a symbol representing a power, if the number of steps represented by EV changes by + n, the amount related to exposure (shutter speed, aperture value, luminance, ISO sensitivity) will be 2 ^ n times When the number of steps represented by a value changes by -n steps, the amount related to exposure becomes 2 ^ (-n) times.

  In the example shown in FIG. 6, ISO1600 at the start of shooting is currently improved to a noise amount equivalent to ISO400. That is, since the ISO sensitivity is 1⁄4 = 2 ^ (− 2) times and the ISO sensitivity changes by −2 steps, “−2.0 steps” is displayed as the image quality index. . Thus, the photographer can understand that the image quality has improved by two steps when converted to the ISO sensitivity.

  Therefore, the noise amount calculation unit 6 calculates, as the image quality index, a numerical value indicating the change amount of the noise amount by comparing the noise amount at the start of shooting and the noise amount of the current composite image with the number of steps. The ISO sensitivity, which is an image quality index, and the ISO sensitivity, which is an image quality index of the current composite image, are calculated.

  And FIG. 7 is a figure which shows the example displayed as an image quality index how much the present dynamic range changed from the dynamic range at the time of imaging | photography start.

  Here, the dynamic range indicates a range from the lowest value to the highest value of the pixel value (pixel value range capable of accurately photographing from the dark part to the bright part of the subject) capable of discriminating the subject. As the dynamic range is wider, the subject information is taken without loss, and the image quality is high, which is preferable as an image.

  Here, when the amount of noise is large, the vicinity of the lowest value of the pixel value obtained as the subject information is buried in the noise level, so it can not be determined what is shown. That is, the dynamic range becomes narrower as the amount of noise is larger.

  Therefore, in the case of a photographer who takes an image by paying attention to the change of the dynamic range when performing the image combining process, as shown in FIG. 7 instead of using the magnitude of the noise amount as the image quality index. It may be displayed as an image quality index how much the current dynamic range has changed from the dynamic range at the start of shooting. In this case, the noise amount calculation unit 6 calculates a numerical value indicating the dynamic range of the image data obtained from the noise amount as an image quality index.

Here, the dynamic range can be expressed as a numerical value in units of the number of stages by the following calculation formula using the logarithm "log 2" with 2 as the base.
Dynamic range = log 2 {digital maximum value / determinable lowest pixel value}

  Here, the digital maximum value is 2 ^ n when A / D-converting an analog signal into an n-bit digital signal (Note that the dynamic range is not limited to being calculated digitally, and is calculated by analog) Is possible, but here we take digital as an example). Also, the minimum distinguishable pixel value is not particularly defined, but for example, it is defined that a signal can be determined without being buried in noise if it is equal to or more than the value of the standard deviation of noise of dark part (that is, OB pixel 13b). If this is the case, the standard deviation of noise can be used as the distinguishable lowest pixel value.

  As a specific example, when the digital signal is 12 bits, the digital maximum value is 4096 (= 2 ^ 12).

Further, it is assumed that the standard deviation of the noise of the OB pixel 13b at the start of shooting is 8. In this case,
Dynamic range at the start of shooting = log 2 {4096/8} = 9 (stage)
It becomes.

Then, it is assumed that the standard deviation of noise of the OB pixel 13b of the current composite image as a result of combining a plurality of images is 2 (that is, the amount of noise is 1/4 at the start of imaging). In this case,
Current dynamic range = log 2 {4096/2} = 11 (stage)
It becomes. Therefore, the photographer can grasp that the image has a dynamic range that is wider by two steps than at the start of imaging.

  Thus, after performing the process of step S15, the system control unit 10 detects the state of the 2nd release switch, and determines whether the 2nd release switch is off (step S16). The photographer releases the pressing of the release button when the valve shooting is ended after the valve shooting is started by pressing the release button. Therefore, by determining whether or not the second release switch is off, it is possible to determine whether or not the photographer has performed an input operation for ending the valve shooting.

  Here, if it is determined that the 2nd release switch is not off (it remains on), the process proceeds to step S5, and shooting of the next frame is repeated as described above. Thus, every time the exposure time T elapses, the latest composite image is displayed on the display unit 7 as an image indicating the progress of exposure. It can be confirmed as an image.

  On the other hand, when it is determined in step S16 that the 2nd release switch is off, the exposure of the valve shooting is ended at the timing when it is determined that the 2nd release switch is off regardless of whether the exposure time T has elapsed. Do. Therefore, the image data in the process of being charge-stored in the image sensor 13 is not read out, and reset or the like is appropriately performed.

  Then, when it is determined that the second release switch is off, the composite image data (one or both of the RAW composite image data and the composite image data subjected to development processing) stored in the internal memory 4 is determined. , And stored in the external memory 9 (step S17).

  Further, the composite image data subjected to the development processing stored in the internal memory 4 is displayed on the display unit 7 as a final recording image (step S18), and the composition processing returns to the main processing (not shown).

  According to the first embodiment, when performing the image combining process, the image combining process similar to that of the effective pixel 13a is performed also on the OB pixel 13b of the image sensor 13. Therefore, the noise amount in the OB pixel 13b is The noise amount is the same as the noise amount included in the effective pixel 13a, and the noise amount of the effective pixel 13a can be accurately estimated.

  When the variance or standard deviation of the pixel values of the OB pixel 13b is calculated as the noise amount of the combined image, there is an advantage that the processing is easy.

  In addition, since image quality indexes of various formats are calculated based on the noise amount calculated by the noise amount calculation unit 6 and superimposed on the composite image and displayed on the display unit 7, any combination method is selected. Regardless of the shooting conditions, the photographer can accurately grasp the image quality related to the noise amount of the current composite image, and can be used as a criterion for determining the timing for ending the shooting. .

  In the case of comparative bright combination, relative dark combination, and averaging, since the averaging effect of random noise is present, the noise decreases as the number of frames to be combined increases. When shooting with the aim of noise improvement and selecting these combining methods, the photographer can change the number of frames to be shot (combined) according to the intention of how much the noise should be improved. While grasping the current amount of noise or the amount of noise improvement, it is possible to finish shooting at the timing of one's own choice.

On the other hand, in the additive synthesis, noise generated for each frame is integrated, so the noise increases as the number of frames to be synthesized increases. Since the amount of noise that can be tolerated differs depending on the photographer, the photographer can finish photographing at a desired timing before reaching the amount of noise that can not be tolerated while grasping the current amount of noise.
Second Embodiment

  FIGS. 8 to 12 show a second embodiment of the present invention, and FIG. 8 is a block diagram showing a configuration of the imaging device 1.

  In the second embodiment, the same parts as those in the first embodiment described above are denoted by the same reference numerals, and the description is appropriately omitted, and only different points will be mainly described.

  In the first embodiment described above, the photographer who saw the image quality index manually terminates the photographing at a desired timing. On the other hand, in the present embodiment, the image quality of the target desired by the photographer is set in advance, and the photographing can be automatically ended when the image quality of the image currently being photographed reaches the target image quality. It has become.

  First, as shown in FIG. 8, the imaging device 1 of the present embodiment is obtained by adding a noise amount determination unit 26 to the configuration of the imaging device 1 shown in FIG. 1 of the first embodiment described above. . The noise amount determination unit 26 is connected to the system control unit 10 and the bus 3. Here, although the noise amount determination unit 26 is configured separately from the system control unit 10, the system control unit 10 may double as the noise amount determination unit 26.

  Next, FIG. 9 is a flowchart showing a flow of processing for capturing and combining images of a plurality of frames in the imaging device 1.

  First, the photographer sets a target image quality via the input IF 8 before starting bulb imaging. Therefore, the input IF 8 in the present embodiment functions as a target noise amount setting unit that sets a target noise amount.

  For example, when the selected combining method is additive average combining, the noise is improved as the number of frames to be combined increases, so that the photographer is better than the setting ISO sensitivity (for example, ISO 1600) at the start of shooting. A low ISO sensitivity (for example, equivalent to ISO 400) is set as the target image quality. The target image quality set here is converted into a noise amount and stored in the internal memory 4 or the like as the target noise amount.

  Then, the process shown in FIG. 9 is started, and the processes of steps S1 to S13 are performed in the same manner as the process shown in FIG. 3 of the first embodiment described above.

  Next, the noise amount determination unit 26 determines whether the noise amount calculated in step S13 has reached the target noise amount corresponding to the target image quality (step S21).

  Here, if it is determined that the target noise amount has not been reached, the image quality index of the target image quality is displayed together with the latest composite image and the current image quality index after performing the processing of step S14 (step S15A) ).

  FIG. 10 is a diagram showing a display example when the target image quality is set by the ISO sensitivity.

  On the screen 7a of the display unit 7 shown in FIG. 10, the ISO sensitivity at the start of shooting is 1600, and the target image quality is set to 200 ISO sensitivity, but it is displayed that the current ISO sensitivity is 400 It is an example of

  FIG. 11 is a view showing a display example when the target image quality is set by the noise improvement amount.

  In the example shown in FIG. 11, although the target noise improvement amount which is the target image quality is set to -3.0 steps, it is displayed that the current noise improvement amount has stopped at -2.0 steps. .

  Further, FIG. 12 is a diagram showing a display example when the target image quality is set by the number of stages of the improvement amount of the dynamic range.

  In the example shown in FIG. 12, the dynamic range at the start of shooting is 9 EV, and the target dynamic range improvement amount, which is the target image quality, is set to +3 EV, but the current dynamic range is 11 EV. It indicates that the amount has stopped at +2 EV.

  By performing the display as shown in these examples, the photographer can grasp the achievement degree with respect to the target image quality.

  Thereafter, the process proceeds to step S16, and the system control unit 10 detects the state of the 2nd release switch and determines whether the 2nd release switch is off, as in the first embodiment described above.

  That is, even when the target image quality is not reached, it may be sufficiently assumed that the user may want to finish shooting in the middle of shooting. For example, when the degree of achievement for the target image quality does not progress slowly and can not wait for the completion of achievement, or when it is desired to terminate the photographing halfway through for some reason. Therefore, as in the case of the first embodiment described above, it is determined that the second release switch is off, so that shooting can be ended by the photographer's intention.

  Thus, if it is determined in step S21 that the target noise amount has been reached, or if it is determined in step S16 that the second release switch is off, the processes in steps S17 and S18 are performed. Return to the main processing not shown.

  According to the second embodiment as described above, since the photographing operation is automatically ended when the set target image quality is reached while achieving substantially the same effect as the first embodiment described above, the intention of the photographer is set. It is possible to automatically perform the reflected imaging. Therefore, it is not necessary for the photographer to constantly monitor the image quality index, and it is possible to reduce the burden on the photographer when the bulb imaging is performed for a long time.

Thus, shooting can be ended at an optimal timing, and it is possible to suppress shooting failure.
Third Embodiment

  FIG. 13 shows a third embodiment of the present invention, and is a flowchart showing a flow of processing for capturing and combining images of a plurality of frames in the imaging device 1.

  In the third embodiment, the same parts as those in the first and second embodiments described above will be denoted by the same reference numerals, and the description will be appropriately omitted, and only different points will be mainly described.

  When at least one of the long exposure time (long time exposure) and the high temperature of the image sensor 13 occurs, the influence of the dark current noise generated in the PD (photodiode) of the image sensor 13 increases. Do. This dark current noise appears in the dark part of the image as a shaded or defect-like FPN, resulting in deterioration of the image quality.

  On the other hand, in the image pickup apparatus 1 such as a digital camera, the mechanical shutter 12 is opened for a long time exposure or the same time as the exposure time of the bright image photographed when the temperature of the image sensor 13 is high. Generally, an FPN cancellation process for correcting the FPN is performed by closing the mechanical shutter 12 and taking a dark image, and subtracting the dark image from the bright image for each same pixel position.

  Therefore, in the present embodiment, before the system control unit 10 opens the mechanical shutter 12 and causes the image sensor 13 to generate bright image data of a plurality of frames, the mechanical shutter 12 is closed and the dark image data is closed. Are generated by the image sensor 13. Furthermore, the FPN cancellation processing unit 16 performs FPN cancellation processing for subtracting dark image data from composite image data related to bright image data calculated by the image combining unit 15. Then, the noise amount calculation unit 6 calculates the noise amount based on the light-shielded pixel data on which the image synthesis processing is performed and the FPN cancellation processing is performed.

  That is, the FPN cancellation processing unit 16 according to the present embodiment uses the dark image data to fix the fixed pattern in the composite image data even when the bright image data is the composite image data obtained by the image combining unit 15. It is intended to correct the noise.

  Here, in the present embodiment, assuming that the exposure time T of each frame is constant and the synthesis method is comparative bright synthesis, comparative dark synthesis, or addition average synthesis, it is suitable for these synthesis methods. The FPN cancellation processing will be described.

  In addition, in the case of additive composition, not only FPN but also random noise increases as the number of frames to be combined increases, so FPN can be improved only by using one dark image taken with exposure time T. However, improvement of random noise is difficult, that is, improvement of image quality of an image showing exposure progress can not be expected so much. Therefore, here, as the image combining processing, comparative bright combining, comparative dark combining, or averaging combining is assumed.

  Under the assumption that the exposure time T of each frame described above is constant, it is possible to consider that the FPNs contained in each frame are almost equal. By performing FPN cancellation processing on an image to be displayed and displayed as an intermediate process, it is possible to improve the image quality of an image showing an exposure process.

  When the process shown in FIG. 13 is started, the processes of steps S1 to S3 are performed.

  When it is determined in step S3 that the second release switch is on, the system control unit 10 first closes the mechanical shutter 12 and takes a dark image with an exposure time T using the image sensor 13 (see FIG. Step S31) The stored dark image is stored in the internal memory 4 (step S32).

  After that, the mechanical shutter 12 is opened and the processing of steps S6 to S10 is performed, the bright image is photographed, the composite image is calculated, and stored in the internal memory 4.

  Subsequently, subtraction of the dark image from the combined image obtained by combining the bright image is performed for each same pixel position, and the FPN cancellation processing is performed (step S33). This FPN cancellation process is performed not only on the effective pixel group consisting of the effective pixels 13a but also on the light shielding pixel group consisting of the OB pixels 13b. Here, the synthesized image subjected to the FPN cancellation processing is stored in the internal memory 4 separately from the original synthesized image. Then, the composite image calculated in step S9 is an image obtained by combining the composite image not subjected to the FPN cancellation processing and the newly captured image.

  Next, the pixel defect correction process of step S11 and the development process of step S12 are performed on the composite image subjected to the FPN cancellation process.

  Furthermore, the noise amount calculation unit 6 calculates the noise amount of the composite image data subjected to the development processing after the FPN processing by performing numerical analysis of the OB pixel 13b (step S13A), and calculates the image quality index based on the noise amount. (Step S14A).

  Thereafter, the processing in step S15 and step S16 is performed, and if it is determined in step S16 that the second release switch is off, the composite image stored in the internal memory 4 when it is determined that the second release switch is off FPN cancellation processing is performed on the data (composite image data not yet subjected to FPN cancellation processing) (step S34), and the data is stored in the external memory 9 (step S17A).

  Subsequently, the process of step S18 is performed, and the combination process returns to the main process (not shown).

According to the third embodiment, substantially the same effect as the first and second embodiments described above is obtained, and each frame is acquired using the dark image captured immediately before capturing the bright image. The noise amount is calculated from the OB pixel 13b of the composite image after the FPN cancellation processing by performing the FPN cancellation processing of the composite image, and the image quality index is generated and displayed together with the composite image after the FPN cancellation processing. It is possible to shoot while grasping the image quality pertaining to.
Fourth Embodiment

  14 to 16 show Embodiment 4 of the present invention, and FIG. 14 is a timing chart showing the operation of each part in the imaging device 1, and FIG. FIG. 16 is a flowchart showing noise amount calculation processing after FPN cancellation in the imaging device 1.

  In the fourth embodiment, the same parts as those in the first to third embodiments described above will be denoted by the same reference numerals, and the description will be omitted as appropriate, and only different points will be mainly described.

  In the above-described third embodiment, the dark image is captured immediately before the bulb imaging is started, but in the present embodiment, the dark image is captured immediately after the bulb imaging is completed. Furthermore, in the third embodiment, the FPN cancellation processing of the composite image at each time point during shooting is performed using the acquired dark image, and the noise amount of the composite image after the FPN cancellation processing is calculated to display the image quality index. In the present embodiment, the residual noise (random noise) in the case of being subjected to the FPN cancellation processing from the composite image before the FPN cancellation processing at each time during photographing is estimated as the noise amount to display the image quality index It is a thing.

  First, the flow of the combining process in the present embodiment will be described with reference to FIGS. 14 and 15.

  When this process is started, the processes of steps S1 to S12 are performed as described above. As a result, the bright hour image is read each time the exposure time T elapses in step S5, and as shown in FIG. 14, the first bright hour image, the second bright hour image, and the third bright hour are sequentially from the first frame. An image, a fourth bright image,... Are obtained. Every time a new bright-time image is photographed, a composition process is performed in step S9, and a pixel defect correction process in step S11 and a development process in step S12 are performed.

  Next, the noise amount calculation unit 6 estimates and calculates the noise amount of the composite image subjected to the development processing after the FPN cancellation processing has been performed (step S41).

  That is, the noise amount calculation unit 6 is adjacent to the noise amount calculated based on the light-shielded pixel data of the composite image data and the noise amount calculated based on the light-shielded pixel data of each frame sequentially read from the image sensor 13 FPN cancellation processing every time bright image data of one frame is read from the image sensor 13 using a noise amount calculated based on subtracted image data obtained by subtracting light-shielded pixel data of two frames for each pixel position The amount of noise of the composite image data when it is assumed that The noise amount calculation processing after the FPN cancellation will be described in detail later with reference to FIG.

  Subsequently, in step S15, the composite image subjected to the development processing is displayed on the display unit 7 as a display image, as shown in FIG.

  Further, when the process of step S16 is performed and it is determined in step S16 that the second release switch is off, the system control unit 10 closes the mechanical shutter 12 and the image during the dark time of the exposure time T by the image sensor 13 Is photographed (step S42), and the photographed dark image is stored in the internal memory 4 (step S43).

  Furthermore, in step S34, FPN cancellation processing is performed on the composite image data stored in internal memory 4 using the dark image stored in internal memory 4 in step S43, and in step S17A, the composite after processing is performed. The image data (the final recorded image shown in FIG. 14) is stored in the external memory 9.

  Thereafter, the process of step S18 is performed, and the synthesis process returns to the main process (not shown).

  Next, the principle of the post-FPN cancellation noise amount calculation process of step S41 described above will be described.

First, in general, the standard deviation σ indicating the amount of noise contained in one image is a random noise amount (standard deviation: σr) and an FPN amount (standard deviation: σf), as shown in Equation 1 below. It is expressed by the square root of the sum of squares of
[Equation 1]
σ = {σr ^ 2 + σf ^ 2} ^ (1/2)

Therefore, assuming that the number of frames indicating which frame is a positive integer n, the random noise amount of the nth frame image is σr_n, and the FPN amount is σf_n, the image of the nth frame (one frame) The noise amount σ n included in) is expressed by the following equation 2.
[Equation 2]
σ n = {σ r_ n ^ 2 + σ f _ n ^ 2} ^ (1/2)

On the other hand, the noise amount σ total n of the combined image obtained by subjecting the 1st to nth frame images to image combining processing is also given by the formula given that the random noise amount of the combined image is σ r_ total n and the FPN amount is σ f_ total n According to 1, it is represented by the following equation 3.
[Equation 3]
σ combination n = {σr_combination n ^ 2 + σf_combination n ^ 2} ^ (1/2)

The amount to be obtained now is the noise amount of the composite image after the FPN cancellation processing, that is, σr_combination n. Then, Equation 3 can be transformed into Equation 4 below by transforming Equation 3 into an equation for obtaining σ r — n.
[Equation 4]
σ r_ total n = {σ total n ^ 2-σ f_ total n ^ 2} ^ (1/2)

  The noise amount σ combination n of the composite image on the right side of the equation 4 is calculated from the standard deviation of the pixel values of the OB pixel 13 b of the composite image.

  On the other hand, if the FPN amounts σf_1, σf_2, σf_3,..., Σf_n of the respective frames are known, the FPN amount σf_combination n of the composite image on the right side of Equation 4 is calculated as The random noise amount σr_combination n of the combined image according to several combining methods is also calculated as follows if the random noise amounts σr_1, σr_2, σr_3, ..., σr_n of the frames are known).

First, in the case of using additive composition, it becomes as shown in the following expressions 5 and 5A.
[Equation 5]
sigma f_ total n = sigma f 1 + sigma f 2 + sigma f 3 + ... + sigma f n
[Equation 5A]
σ r _ n n = {σ r _1 ^ 2 + σ r _2 + 2 r σ ^ 3 + 2 + ... + σ r _ n ^ 2} ^ (1/2)

Next, in the case of using the addition mean combination, the following Expression 6 and 6A are obtained.
[Equation 6]
sigma f_ total n = {sigma f_1 + sigma f 2 + sigma f 3 + ... + sigma f n / n
[Equation 6A]
σ r _ n n = {(σ r 1 ^ 2 + σ r 2 ^ 2 + σ r 3 ^ 2 + ... + σ r n 2) / 2} ^ (1/2)

Subsequently, in the case of using the comparative bright combination, if the function max giving the maximum value is used, the following formula 7 is obtained.
[Equation 7]
sigma f_ total n = max {sigma f_1, sigma f_2, sigma f_3, ... sigma f_ n}

Furthermore, in the case of using the relative dark combination, if the function min that gives the minimum value is used, the following equation 8 is obtained.
[Equation 8]
sigma f_ total n = min {sigma f_1, sigma f_2, sigma f_3, ... sigma f_ n}

  Therefore, if the FPN amounts σf_1, σf_2, σf_3,. Will be understood.

  Therefore, it is considered to calculate the FPN amount σ f — n of the image of the nth frame on the basis of the image of the (n−1) th frame and the image of the nth frame.

Here, the noise amount σ n of the image of the n th frame is expressed by Equation 2 described above, and the noise amount σ n -1 of the image of the (n-1) th frame is represented by the following Equation 9 according to Equation 1. expressed.
[Equation 9]
σ n-1 = {σ r n -1 ^ 2 + σ f n -1 ^ 2} ^ (1/2)

Here, it is assumed that the exposure time T and the ISO sensitivity at the time of shooting an image of each frame are the same. Furthermore, it is assumed that the temperature of the image sensor 13 hardly changes between the time of shooting the image of the (n-1) th frame and the time of shooting the image of the nth frame. At this time, approximation as shown in the following Equations 10 and 10A can be performed.
[Equation 10]
σr_n-1 σ σr_n
[Equation 10A]
σ f n -1 ≒ σ f n

  Here, what is described here is only that the random noise amount and the FPN amount are similar between adjacent frames in the order (frame number), so, for example, σr_n−1σσr_n, and σr_n− It should be noted that just because 2 ≒ σr_n−1 does not mean that σr_n−2 ≒ σr_n holds.

  Then, with respect to the OB pixels 13b of the (n-1) th frame image and the nth frame image of the order adjacent to each other, the pixel values of the same pixel position are subjected to subtraction processing, From the standard deviation, noise amounts σ n-1 to n included in the subtraction image are calculated. Since the FPN is canceled when this subtraction processing is performed, it can be considered that the noise amounts σ n-1 to n included in the subtraction image are noise amounts of only random noise.

Here, since random noise is noise generated independently in the + and-directions, the amount of noise when subtraction processing is performed is the amount of random noise of (n-1) frames, as in the addition and synthesis. It can be represented by the square root of the sum of squares of the random noise amount σr_n of σr_n−1 and n frames. Therefore, the noise amounts σ n-1 to n included in the subtraction image are expressed as shown in the following Equation 11.
[Equation 11]
σ n-1 to n = {σ r ^ n ^ 2 + σ r n n -1 ^ 2} ^ (1/2)

If the approximation of Equation 10 is applied to Equation 11, the following Equation 12 is obtained.
[Equation 12]
σ n-1 to n = {σ r ^ n ^ 2 + σ r n n -1 ^ 2} ^ (1/2)
{{Σr_n ^ 2 + σr_n ^ 2} ^ (1/2)
= 2 ^ (1/2) x σr_n

If Equation 12 is rewritten into an equation for obtaining σ r — n, Equation 13 is obtained as follows.
[Equation 13]
σ r n ≒ (σ n-1 to n) / (2 ^ (1/2))

Therefore, the following equation 14 is obtained by rewriting the equation 2 into the equation for finding σ f — n and applying the equation 13.
[Equation 14]
σ f n = {σ n ^ 2-σ r n ^ 2} ^ (1/2) = {σ n ^ 2-(σ n-1 to n ^ 2) / 2} ^ (1/2)

  Once the value of the FPN amount σf_n of each frame is calculated in this manner, the FPN amount σf_combination n of the composite image can be calculated by using any one of the above-mentioned formulas 5, 6, 7 and 8. As a result, the noise amount σr_combination n of the composite image after the FPN cancellation processing can be obtained by Equation 4.

  The details of the process of calculating the amount of noise after FPN cancellation in step S41 based on such a principle will be described with reference to FIG.

  When this process is started, the noise amount .sigma.n of the OB pixel 13b of the nth frame image read out from the image sensor 13 is calculated as, for example, a standard deviation, and each image of the 1st to nth frame is synthesized and obtained. For example, the noise amount σ combination n of the synthesized image is calculated as a standard deviation (step S51).

  Next, an image operation is performed to subtract the pixel value of the OB pixel 13b of the image of the (n-1) th frame from the pixel value of the OB pixel 13b of the nth frame image for each pixel position (step S52) .

  Subsequently, the noise amount σ n-1 to n of the OB pixel 13b of the subtraction image is calculated, for example, as a standard deviation (step S53).

  Further, using the noise amount σn calculated in step S51 and the noise amounts σn-1 to n calculated in step S53, the FPN amount σf_n of the n-th frame image is calculated according to Formula 14 (step S54).

  The FPN amount σ f — n calculated here is stored in the internal memory 4 (step S55). Thereby, the FPN amount of each frame is sequentially accumulated in the internal memory 4.

  Then, using the FPN amounts σf_1, σf_2, σf_3,. Calculate n (step S56).

  Thereafter, using the noise amount σ total n calculated in step S51 and the FPN amount σ f_ total n calculated in step S56, the noise amount σ r_ total n of the composite image after the FPN cancellation processing is Calculated by 4 (step S57), and return from this process.

  When shooting a dark image immediately before starting the shooting of a bright image as in the third embodiment described above, until the bright image is actually started after the second release switch is turned on by the release button For example, when shooting a firework or the like, the case where the firework you want to shoot may not appear has occurred because of a long time lag of

  On the other hand, according to the fourth embodiment, almost the same effect as the first to third embodiments described above is obtained, and the dark time image is photographed immediately after the bright time image photographing is finished. In addition, it is possible to reduce the case in which the shutter chance is missed.

  In the above description, the valve shooting is started by pressing the release button and the valve shooting is stopped by releasing the pressing button. However, the present invention is not limited to this. For example, the valve shooting is started by pressing the release button, and once the valve shooting is started, the valve shooting is continued even if the pressing is released, and the valve shooting is stopped when the release button is pressed again. Also good.

  Further, as described above, it is possible to set two or more combinations simultaneously instead of being able to set any one of comparative bright combination, comparative dark combination, addition average combination and addition combination. . Then, the display of the composite image showing the progress in the bulb photographing may be only the comparative bright composite image, only the comparative dark composite image, only the average composite image, or even simply display the addition image. good.

  Furthermore, although the imaging apparatus 1 has been mainly described above, an imaging method that performs the same process as the imaging apparatus 1 may be used, or a processing program for causing a computer to perform the same process as the imaging apparatus 1 It may be a non-transitory recording medium readable by a computer which records a program.

  Specifically, although the processing for the imaging device 1 has been described above, the present invention is not limited thereto, and image data of a plurality of frames is acquired by the imaging device 1 and the image data of the acquired plurality of frames is described above. Such image processing may be performed.

  Further, among the techniques described in the present specification, regarding the control mainly described in the flowchart, the processing program can often be executed, and the processing program may be stored in a recording medium or a recording unit. The recording method to the recording medium or the recording unit may be recorded at the time of product shipment, may use the distributed recording medium, or may be downloaded via a communication line such as the Internet.

  Furthermore, regarding the operation explanation in the claims, the specification, and the drawings (flowcharts and the like), using words expressing an order such as "first", "next", "follow", "after" etc. Even if it explains, it does not mean that it is essential to carry out in this order in the part which is not especially explained.

  The present invention is not limited to the above-described embodiment as it is, and in the implementation stage, the constituent elements can be modified and embodied without departing from the scope of the invention. In addition, various aspects of the invention can be formed by appropriate combinations of a plurality of constituent elements disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements in different embodiments may be combined as appropriate. As a matter of course, various modifications and applications are possible without departing from the scope of the invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device 2 ... Imaging part 3 ... Bus 4 ... Internal memory 5 ... Image processing part 6 ... Noise amount calculation part 7 ... Display part 7a ... Screen 8 ... Input IF
9 external memory 10 system control unit 11 lens 12 mechanical shutter 13 image sensor 13a effective pixel 13b OB pixel 15 image combination unit 16 FPN cancellation processing unit 17 pixel defect correction unit 18 development processing unit 21 ... comparison bright combining unit 22 ... comparison dark combining unit 23 ... addition combining unit 24 ... average combining unit 26 ... noise amount determination unit

Claims (13)

An imaging element having an effective pixel group for generating image data related to an optical image of a subject, and a light shielding pixel group disposed around the effective pixel group to generate light shielding pixel data by shielding the periphery of the effective pixel group;
The image combining process is performed on the plurality of frames of image data sequentially read from the effective pixel group to calculate one frame of combined image data, and the plurality of frames of frame images sequentially read from the light shielding pixel group An image combining unit that performs the same image combining processing as performed on the image data on light-shielded pixel data;
A noise amount calculation unit that calculates a noise amount based on the light-shielded pixel data subjected to the image synthesis processing;
Equipped with
An image pickup apparatus, wherein the image synthesis unit and the noise amount calculation unit perform the process each time the image data of one frame and the light-shielded pixel data are read out from the image pickup device. An index calculation unit that calculates an image quality index representing the image quality of the combined image data from the noise amount calculated by the noise amount calculation unit;
A display unit for displaying the image quality index;
The image pickup apparatus according to claim 1, further comprising:   The imaging device according to claim 1, wherein the image combining unit performs addition average combining processing as the image combining processing.   The imaging device according to claim 1, wherein the image combining unit performs comparative bright combining processing or comparative dark combining processing as the image combining processing.   The imaging device according to claim 1, wherein the image combining unit performs addition combining processing as the image combining processing. An optical shutter for controlling the arrival and non-arrival of an optical image of a subject on the imaging device by opening and closing;
A control unit for causing the image pickup device to generate dark image data by closing the optical shutter prior to generating the bright image data of a plurality of frames by opening the optical shutter;
An FPN cancellation processing unit that performs an FPN cancellation process of subtracting the dark image data from the combined image data related to the bright image data calculated by the image combining unit;
And further
The imaging apparatus according to claim 1, wherein the noise amount calculation unit calculates the noise amount based on the light-shielded pixel data on which the image synthesis processing is performed and the FPN cancellation processing is performed. An optical shutter for controlling the arrival and non-arrival of an optical image of a subject on the imaging device by opening and closing;
A control unit for causing the image sensor to generate dark image data by closing the optical shutter after the optical shutter has been opened and the image sensor has generated all bright image data of a plurality of frames;
After all the generation of the bright image data of the plurality of frames is completed, an FPN cancellation process is performed to subtract the dark image data from the combined image data related to the bright image data calculated by the image combining unit. FPN cancellation processing unit,
And further
The noise amount calculation unit is adjacent to the noise amount calculated based on the light-shielded pixel data of the composite image data and the noise amount calculated based on the light-shielded pixel data of each frame sequentially read from the imaging device. The above-mentioned FPN is used each time the bright image data of one frame is read from the image sensor using the noise amount calculated based on the subtracted image data obtained by subtracting the light-shielded pixel data of two frames for each pixel position. 2. The image pickup apparatus according to claim 1, wherein an amount of noise of the combined image data when cancellation processing is performed is calculated. A target noise amount setting unit that sets a target noise amount;
A noise amount determination unit that determines whether the noise amount of the light-shielded pixel data subjected to the image synthesis processing calculated by the noise amount calculation unit has reached the target noise amount;
A control unit that causes the image sensor to finish reading the image data when it is determined by the noise amount determination unit that the target noise amount has been reached;
The image pickup apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 2, wherein the index calculation unit calculates the image quality index at the start of shooting and the image quality index of a current composite image.   The imaging apparatus according to claim 2, wherein the index calculation unit calculates an ISO sensitivity as the image quality index.   The said index calculation part calculates the numerical value which shows the variation of the noise amount which compared the noise amount at the time of an imaging | photography start with the noise amount of the present composite image as an image quality index. Imaging device.   The imaging apparatus according to claim 2, wherein the index calculation unit calculates a numerical value indicating a dynamic range of image data obtained from a noise amount as the image quality index. Reading out image data relating to an optical image of a subject from the effective pixel group, and reading out light shielding pixel data from a light shielding pixel group arranged in a light shielding manner around the effective pixel group;
Each time the image data of one frame and the light-shielded pixel data are read out, the image combining process is performed on the image data of a plurality of frames sequentially read from the effective pixel group to calculate composite image data of one frame. And performing the same image combining processing as performed on the image data on the plurality of light shielding pixel data of a plurality of frames sequentially read out from the light shielding pixel group,
Calculating a noise amount based on the light-shielded pixel data subjected to the image combining process each time the image data of one frame and the light-shielded pixel data are read;
An imaging method comprising:

Images