JP2016225757A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus or the like capable of accurately grasping a noise amount of synthetic image data of one frame calculated by performing image composition-processing of image data of a plurality of frames during imaging.SOLUTION: An imaging device 1 includes: an image sensor 13 having an effective pixel group and a shaded pixel group; an image composition part 15 in which image composition processing is performed on image data of a plurality of frames sequentially read out from the effective pixel group to calculate synthetic image data of one frame and which performs the same image composition processing as did on image data with respect to shaded data of the plurality of frames sequentially read out from the shaded pixel group; and a noise amount calculation part 6 for calculating the noise amount on the basis of the shaded pixel data whose image composition processing is performed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup apparatus and an image pickup method for performing image composition processing on a plurality of frames of image data and calculating one frame of combined image data.

  2. Description of the Related Art Conventionally, in a single-lens reflex imaging apparatus, an object image is observed using an optical viewfinder. On the other hand, in recent years, an imaging apparatus that observes a subject image by live view display without using an optical finder has become widespread. Here, the live view is a method of displaying an image read from the image sensor on a liquid crystal monitor or the like in real time. Further, in addition to the liquid crystal monitor, an image pickup apparatus provided with an electronic viewfinder is also sold. In such an image pickup apparatus, observation through the liquid crystal monitor and observation through the electronic viewfinder can be switched. When a live view image is observed through an electronic viewfinder, observation with a sense closer to that of an optical viewfinder is possible than when a live view image is observed through a liquid crystal monitor.

  Conventionally, however, the image signal can be read from the image sensor during long exposures such as bulb photography, whether using an optical viewfinder or using an electronic viewfinder. Therefore, the exposure state of the subject could not be confirmed during the photographing, and the image was confirmed after the photographing was finished. For this reason, the photographer estimates the exposure setting such as the exposure time by the photographer himself based on the brightness of the subject, and starts the exposure and ends the exposure. For this reason, it has not been easy to obtain a desired photographed image without causing photographing failure due to underexposure or overexposure.

  Therefore, for example, in Japanese Patent Laid-Open No. 2005-117395, a pixel signal is read from an image sensor at a predetermined time interval, and an image obtained by simply accumulating each time the image signal is read from the image sensor is displayed on a liquid crystal monitor. An imaging device to be displayed is described. According to such an image pickup apparatus, the progress of the image is displayed during long-time shooting such as bulb shooting, so that shooting failures can be reduced.

  Japanese Patent No. 4148586 discloses an imaging device that continuously reads out image signals from an image sensor and generates a bulb-captured image by subjecting the read-out image signals to comparative bright combination processing. Here, the comparatively bright combination process is an image combination process in which the pixel values of the pixels constituting the image data are compared for each pixel, and the larger pixel value is selected to be the combined pixel value.

  By the way, various noises are generated as the characteristics of the image sensor. For example, in a photodiode provided in each pixel of an image sensor, dark current noise caused by dark current, dark current shot noise caused by dark current noise, and light shot noise generated when photoelectrically converting incident light Etc. occur. Further, circuit noise such as reset noise and amplifier noise is also generated in the readout circuit. These noises appear as random noise or fixed pattern noise in the image as pixel value unevenness (shading) in the image plane, with pixel values varying from pixel to pixel, line to line, and line to line.

  These noises vary in the amount of noise depending on the shooting conditions, and further, in the case of performing image composition, the amount of noise varies depending on what kind of image composition processing is used. For example, the dark current noise tends to increase as the temperature of the image sensor increases and the exposure time (shutter speed) increases. Furthermore, each noise mentioned above becomes so large that the ISO sensitivity setting of an imaging device is high.

  In addition, with respect to the image synthesis process, when addition synthesis is used, noise increases as the number of synthesized images increases, so that the amount of noise increases. In addition, in the case of using addition average synthesis, the amount of noise of fixed pattern noise included in each image to be synthesized does not basically change even after image synthesis. On the other hand, since the random noise is averaged as the number of images to be combined in the addition average composition increases, the amount of noise decreases. Further, in the case of using comparative bright synthesis or comparative dark synthesis, the solid pattern noise basically does not change as in the addition average synthesis. On the other hand, the random noise is the largest value among the pixel values that change randomly in the case of comparatively bright combination, and the smallest value among the pixel values that change randomly in the case of comparative dark combination. Since the pixel value of each pixel converges to the value, the variation of the pixel value for each pixel tends to be small.

  Further, in bulb photographing, an FPN (Fixed Pattern Noise) cancellation process may be performed to correct fixed pattern noise due to dark current. In this FPN cancellation process, an optical shutter is opened and a bright image is captured, then the optical shutter is closed and a dark image is automatically captured with the same exposure time as the bright image. This is processing for correcting fixed pattern noise by subtracting dark image data from light image data by a processing circuit. When this FPN cancellation process is performed, the fixed pattern noise is corrected, but the random noise increases because the noise in the bright image and the noise in the dark image are accumulated.

  When images with different shooting conditions (that is, with different noise amounts) are combined, not only random noise but also the fixed pattern noise amount varies from image to image, so the amount of noise changes more complicatedly. It will be.

  As described above, the noise amount of the composite image varies depending on the shooting conditions and the image composition processing.

  On the other hand, the photographer sensuously determines the noise feeling related to the image quality of the composite image when shooting. For example, in the technique described in Japanese Patent Application Laid-Open No. 2005-117395, it is possible in principle to check the image quality of an image being combined in order to display the progress of image combining on a liquid crystal monitor or the like. In a liquid crystal monitor attached to the imaging device or a smartphone wirelessly connected to the imaging device, for example, it is difficult to confirm the noise for each pixel in detail because the screen is small. In addition, since the main subjects of bulb photography are, for example, the starry sky, fireworks, fireflies, etc., bulb photography is often performed in dark places at night. Since the visibility of noise is different from that when viewing images in the daytime or in a bright room after shooting, it is possible to accurately determine whether or not the amount of noise is satisfactory for the photographer. difficult.

  The photographer who attaches importance to the amount of noise in the composite image may determine the timing to end the shooting depending on how much the amount of noise after synthesis is, but as described above, Since it is not possible to grasp the amount of noise during image capture, it is difficult to accurately determine the timing of image capture end.

  By the way, Japanese Patent Application Laid-Open No. 2005-20562 describes a technique for changing a noise correction process according to a noise amount calculated based on a pixel value output from an OB (optical black) pixel of an image sensor. . However, with the technique described in the publication, the photographer cannot grasp the amount of noise during photographing. In the publication, the amount of noise when performing various image composition processes as described above is accurately calculated. Is not considered.

JP 2005-117395 A Japanese Patent No. 4148586 Japanese Patent Laying-Open No. 2005-20562

  Thus, with the conventional technique, it has been difficult to accurately grasp the noise amount of one frame of composite image data calculated by performing image composition processing on a plurality of frames of image data during shooting.

  The present invention has been made in view of the above circumstances, and is capable of accurately grasping the noise amount of one-frame synthesized image data calculated by image synthesis processing of a plurality of frames of image data during shooting. An object is to provide an apparatus and an imaging method.

  An imaging device according to an aspect of the present invention includes an effective pixel group for generating image data related to an optical image of a subject, and a light-shielded pixel for generating light-shielded pixel data arranged around the effective pixel group in a light-shielded manner. A plurality of frames of image data sequentially read from the effective pixel group to perform image synthesis processing to calculate one frame of combined image data, and sequentially from the light-shielded pixel group An image compositing unit that performs the same image compositing process as that performed on the image data on the light-shielded pixel data of a plurality of frames read out in a frame, and a noise amount based on the light-shielded pixel data on which the image compositing process is performed The image composition unit and the noise amount calculation unit read one frame of the image data and the light-shielded pixel data from the image sensor. Performing the above-mentioned processing to.

  An imaging method according to an aspect of the present invention includes a step of reading out image data relating to an optical image of a subject from an effective pixel group, and reading out light-shielded pixel data from a light-shielded pixel group arranged in the vicinity of the effective pixel group. Each time the image data of one frame and the light-shielded pixel data are read out, image synthesis processing is performed on the image data of a plurality of frames sequentially read from the effective pixel group to calculate one frame of combined image data. In addition, a step of performing the same image composition processing as that performed on the image data with respect to the light-shielded pixel data of a plurality of frames sequentially read out from the light-shielded pixel group, the image data of one frame, and the light-shielding Each time pixel data is read, a step of calculating a noise amount based on the shaded pixel data on which the image composition processing has been performed is included.

  According to the imaging apparatus and the imaging method of the present invention, it is possible to accurately grasp the noise amount of one frame of synthesized image data calculated by performing image synthesis processing on a plurality of frames of image data during imaging.

1 is a block diagram illustrating a configuration of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram illustrating a pixel configuration of the image sensor in the first embodiment. 5 is a flowchart showing a flow of processing for photographing and combining a plurality of frames in the imaging apparatus according to the first embodiment. FIG. 6 is a diagram illustrating an example in which the ratio of the current noise amount to the noise amount at the start of shooting is displayed as an image quality index in the first embodiment. FIG. 6 is a diagram showing an example of displaying as an image quality index how much the current ISO sensitivity has changed from the ISO sensitivity at the start of shooting in the first embodiment. FIG. 6 is a diagram illustrating an example of displaying, as an image quality index, how much the noise amount has been improved as an EV value based on a change from the ISO sensitivity at the start of shooting to the current ISO sensitivity in the first embodiment. FIG. 6 is a diagram showing an example of displaying as an image quality index how much the current dynamic range has changed from the dynamic range at the start of shooting in the first embodiment. The block diagram which shows the structure of the imaging device in Embodiment 2 of this invention. 9 is a flowchart illustrating a flow of processing for capturing and combining a plurality of frames in the imaging apparatus according to the second embodiment. In the said Embodiment 2, the figure which shows the example of a display when target image quality is set with ISO sensitivity. In the said Embodiment 2, the figure which shows the example of a display when target image quality is set with the noise improvement amount. In the said Embodiment 2, the figure which shows the example of a display when target image quality is set with the step number of the improvement amount of a dynamic range. 9 is a flowchart showing a flow of processing for photographing and combining a plurality of frames in the imaging apparatus according to Embodiment 3 of the present invention. 9 is a timing chart showing the operation of each unit in the imaging apparatus according to Embodiment 4 of the present invention. 9 is a flowchart showing a flow of processing for photographing and combining a plurality of frames in the imaging apparatus according to the fourth embodiment. 10 is a flowchart illustrating a noise amount calculation process after FPN cancellation in the imaging apparatus according to the fourth embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
[Embodiment 1]

  1 to 7 show the first embodiment of the present invention, and FIG. 1 is a block diagram showing the configuration of the imaging apparatus 1. FIG. 1 mainly shows the electrical configuration of the imaging apparatus 1.

  The imaging apparatus 1 according to the present embodiment includes an imaging unit 2, a bus 3, an internal memory 4, an image processing unit 5, a noise amount calculation unit 6, a display unit 7, and an input IF (Interface) 8. And a system control unit 10. Although the external memory 9 is described in the imaging device 1 shown in FIG. 1, the external memory 9 may be a memory card that can be attached to and detached from the imaging device 1 as will be described later. The configuration need not be unique to the imaging apparatus 1.

  In the present embodiment, the image capturing apparatus 1 will be described as appropriate assuming that the image capturing apparatus 1 is, for example, a digital camera (such as a compact digital camera or a digital single-lens camera), but the image capturing apparatus 1 may be a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone, a smartphone, a personal digital assistant (PDA), a game device, or the like. Furthermore, the present invention is not limited thereto, and the imaging apparatus 1 can be widely applied as long as it is a device for photographing that can be exposed for a long time (here, “exposure” is also referred to as “exposure”).

  The imaging unit 2 includes a lens 11, a mechanical shutter 12, and an image sensor 13.

  The lens 11 forms an optical image of the subject on the image sensor 13. In general, the lens 11 includes a focus lens for performing focus adjustment and a diaphragm for adjusting the amount of light reaching the image sensor 13. When the aperture is adjusted here, the aperture value related to the exposure amount changes. In the present embodiment, the description will be made assuming that the lens 11 is configured integrally with the imaging apparatus main body. However, the lens 11 may be configured as an interchangeable lens that can be attached to and detached from the imaging apparatus main body.

  The mechanical shutter 12 is an optical shutter that controls opening and closing of the optical image of the subject from the lens 11 to the image sensor 13 by opening and closing, and also controls exposure time (shutter speed).

  The image sensor 13 is an imaging device in which a plurality of pixels are arranged in a two-dimensional manner, and an optical image of a subject formed by the lens 11 is converted into an electrical signal for each pixel to generate image data. It is configured as a CMOS image sensor, a CCD image sensor, or the like. Image data generated by the image sensor 13 is output to the image processing unit 5 and the bus 3.

  Here, FIG. 2 is a diagram showing a pixel configuration of the image sensor 13.

  The image sensor 13 is arranged to shield the effective pixel group including the effective pixel 13a for generating the image data related to the optical image of the subject as a constituent element, and to generate light-shielded pixel data arranged around the effective pixel group. A light-shielding pixel group having the OB pixel 13b as a light-shielding pixel as a constituent element.

  As shown in FIG. 2, the shading pixel group includes (horizontal) OB (Optical Black) pixels 13b on the same horizontal line as each horizontal line of the effective pixel group, for example, along the left side of the effective pixel 13a. A horizontal light-shielding pixel group arranged, and a vertical light-shielding pixel group in which (vertical) OB pixels 13b are arranged on the same vertical line as each vertical line of the effective pixel group, for example, along the upper side of the effective pixel 13a. It is equipped with.

  Here, each of the effective pixel 13a and the OB pixel 13b includes a PD (photodiode) that converts light into an electric signal. However, when the mechanical shutter 12 is in an open state, the effective pixel 13a. In this case, the light of the subject image formed by the lens 11 arrives, whereas the light shielding film is provided on the OB pixel 13b, which is a light shielding pixel, so that the subject image formed by the lens 11 The light does not reach. Therefore, the OB pixel 13b always outputs light-shielded pixel data regardless of whether the mechanical shutter 12 is opened or closed.

  The black reference signal level in the image data output from the effective pixel 13a varies depending on the temperature of the image sensor 13, the exposure time (charge accumulation time), the voltage variation of the peripheral circuit, and the like. For this purpose, the light-shielded pixel data from the OB pixel 13b output simultaneously with the image data from the effective pixel 13a is used as the black reference signal level of the image data from the effective pixel 13a.

  The bus 3 is a signal line for each unit in the imaging apparatus 1 to transmit and receive signals. For example, the imaging unit 2, the internal memory 4, the image processing unit 5, the noise amount calculation unit 6, the display unit 7, the input IF 8, the external memory 9, and the system control unit 10 are connected to the bus 3.

  The internal memory 4 stores information related to the imaging device 1 and includes a nonvolatile memory such as a flash memory or a volatile memory such as SDRAM. Here, the nonvolatile memory stores, for example, a processing program for the system control unit 10 to perform processing for controlling the imaging apparatus 1 or various setting information necessary for the operation of the imaging apparatus 1. The volatile memory temporarily stores image data during image processing.

  The image processing unit 5 performs various types of image processing on the image data output from the image sensor 13 and stored in the internal memory 4 as necessary. The image processing unit 5 and an FPN (Fixed Pattern Noise: fixed) A pattern noise) cancel processing unit 16, a pixel defect correction unit 17, and a development processing unit 18.

  The image composition unit 15 performs image composition processing on a plurality of frames of image data read out sequentially (also expressed as “time-series” or “continuously”) from the effective pixel group, thereby synthesizing one frame. The image sensor 13 calculates the image data and performs the same image composition processing as that performed on the image data on the light-shielded pixel data of a plurality of frames sequentially read from the light-shielded pixel group. This is performed each time data and light-shielded pixel data are read. Here, the image composition process performed by the image composition unit 15 is a process for composing pixel values for each corresponding pixel position.

  Here, as a specific example, a configuration in which the image composition unit 15 includes a comparative light composition unit 21, a comparative dark composition unit 22, an addition composition unit 23, and an average composition unit 24 will be described. Alternatively, instead of some of these, a unit for performing other image composition processing may be provided. In the following, processing performed by the comparatively bright combining unit 21, the comparative dark combining unit 22, the addition combining unit 23, and the average combining unit 24 respectively on the image data read from the effective pixels 13a will be described. However, as described above, the same processing is performed on the light-shielded pixel data read from the OB pixel 13b.

  The comparatively bright combination unit 21 performs the following cumulative comparative bright combination process. The image data generated based on the image data first read from the image sensor 13 (this “image data generated based on the image data read from the image sensor 13” is an image in this embodiment. Although so-called RAW image data read from the sensor 13 is assumed, the present invention is not limited to this, and image data that has been processed by the FPN cancellation processing unit 16, the pixel defect correction unit 17, the development processing unit 18, and the like. However, the same applies hereinafter) is first stored in the internal memory 4 as comparatively bright composite image data.

  Next, when image data is read out from the image sensor 13, the comparative bright combination unit 21 is stored in the internal memory 4 and pixel data constituting image data generated based on the read-out image data. The pixel data constituting the comparatively bright composite image data is compared for each corresponding pixel position. Then, the comparatively bright composite image data is reconstructed by using the larger (that is, the brighter) pixel data of the comparison results as the pixel data at the corresponding pixel position.

  By repeating such processing every time image data is read from the image sensor 13, the latest comparatively bright composite image data is stored in the internal memory 4. If such comparatively bright combination processing is performed, for example, when taking an astronomical photograph, an image of a light trail of a star in the night sky can be obtained. At this time, the brightness of the background is constant regardless of the exposure time, and the length of the light trace can be adjusted according to the exposure time.

  The comparative dark combining unit 22 performs the following cumulative comparative dark combining process. First, image data generated based on the image data read from the image sensor 13 is first stored in the internal memory 4 as comparative dark composite image data.

  Next, when image data is read out from the image sensor 13, the comparative dark combining unit 22 is stored in the internal memory 4 and pixel data constituting image data generated based on the read-out image data. The pixel data constituting the comparative dark composite image data is compared for each corresponding pixel position. Then, the comparative dark composite image data is reconstructed by using the smaller (ie, darker) pixel data of the comparison results as the pixel data at the corresponding pixel position.

  By repeating such processing every time image data is read from the image sensor 13, the latest comparative dark composite image data is stored in the internal memory 4. If such comparative dark combination processing is performed, for example, when photographing astronomical photographs, it is possible to obtain an image of only the background in which the light trails of stars moving in the night sky are erased.

  The addition synthesis unit 23 performs the following cumulative addition synthesis process. First, image data generated based on the image data read from the image sensor 13 is first stored in the internal memory 4 as addition synthesized image data.

  Next, when the image data is read from the image sensor 13, the addition / synthesis unit 23 adds the pixel data constituting the image data generated based on the read image data and the addition stored in the internal memory 4. The pixel data constituting the composite image data is added for each corresponding pixel position. Then, the added combined image data is reconstructed by using the added pixel data as the pixel data at the corresponding position.

  By repeating such processing every time image data is read from the image sensor 13, the latest addition combined image data is stored in the internal memory 4. If such addition synthesis processing is performed, for example, when photographing an astronomical photograph, the brightness of the image increases with each addition, so that the latest addition synthesis image data is displayed on the display unit 7 as needed. This makes it possible to visually check the progress of the image change during bulb photographing.

  The average synthesis unit 24 performs the following cumulative average synthesis process. First, image data generated based on the image data read from the image sensor 13 is first stored in the internal memory 4 as average synthesized image data.

  Next, when the image data is read out from the image sensor 13, the average combining unit 24 stores the pixel data constituting the image data generated based on the read-out image data and the average stored in the internal memory 4. The pixel data constituting the composite image data is averaged for each corresponding pixel position. Then, the averaged composite image data is reconstructed by using the pixel data obtained by the averaging as pixel data at the corresponding position.

  In this case, the addition average is stored in the internal memory 4 because the weight of the image data read out from the image sensor 13 at a later time becomes larger if the addition is averaged. If the average composite image data is a result of combining n (n is an integer of 1 or more) images, the average composite image data stored in the internal memory 4 is weighted by n / (n + 1), and the image It is preferable to perform a weighted average process for assigning a weight of 1 / (n + 1) to image data newly read from the sensor 13.

  By repeating such processing every time image data is read from the image sensor 13, the latest average composite image data is stored in the internal memory 4. Then, the averaged composite image data is reconstructed by using the pixel data obtained by the averaging as pixel data at the corresponding position. If such an addition average composition process is performed, for example, when taking an astronomical photograph, random noise is averaged and reduced, so that a high-quality image can be obtained with low noise.

  The FPN cancellation processing unit 16 subtracts dark image data (image data obtained by closing the mechanical shutter 12) from bright image data (image data obtained by opening the mechanical shutter 12). Thus, the fixed pattern noise is corrected.

  The pixel defect correction unit 17 replaces the pixel value of a defective pixel (a pixel that outputs an abnormal pixel value different from that of a normal pixel) with an average value of pixel values of a plurality of pixels in the vicinity of the defective pixel. The correction is made so that the defective pixel does not appear in the image. Here, the defective pixel is inspected in advance at the time of production of the imaging device 1 or the like, and is registered in advance in the internal memory 4 as a defective pixel address. Alternatively, even after the image pickup apparatus 1 is shipped, the correlation between the surrounding pixels of each pixel in the image data obtained by photographing is calculated, and a pixel that is determined to be a defective pixel without correlation is determined later. It may be additionally registered as a defective pixel.

  The development processing unit 18 performs a development process on the RAW image data obtained from the imaging unit 2 or the synthesized RAW image data obtained by synthesizing the RAW image data obtained from the imaging unit 2 by the image synthesis unit 15. Image processing such as mosaicing, white balance adjustment, noise reduction, gamma correction, YC signal generation, and resizing is performed. Here, the resizing process is a process for adjusting the number of pixels of the image data read from the image sensor 13 to the number of display pixels of the display unit 7. Note that the development processing unit 18 may further perform image compression processing or the like.

  The noise amount calculation unit 6 calculates the amount of noise based on the light-shielded pixel data on which the image composition processing has been performed every time one frame of image data and light-shielded pixel data is read from the image sensor 13. Specifically, the noise amount calculation unit 6 calculates the variance or standard deviation of the pixel values of the plurality of OB pixels 13b belonging to the light shielding pixel group as the noise amount. The noise amount calculation unit 6 further functions as an index calculation unit that calculates an image quality index representing the image quality of the composite image data from the calculated noise amount. This image quality index is a useful index for the photographer to determine the image quality of the composite image data, and some specific examples will be described later with reference to FIGS.

  The display unit 7 displays an image developed by the development processing unit 18 and various types of information related to the imaging device 1, and includes, for example, a TFT (Thin Film Transistor) liquid crystal or an organic EL (Electro Luminescence). A display device is provided. Specific examples of the configuration of the display unit 7 include a rear display unit of the imaging apparatus 1 or an EVF (electronic viewfinder). Each time new image data is read from the image sensor 13, the latest composite image obtained by the image composition unit 15 is developed by the development processing unit 18, and then displayed on the display unit 7. Can be followed. In addition, the image quality index calculated by the noise amount calculation unit 6 described above is also displayed on the display unit 7 and can be used by the photographer to determine the image quality.

  The input IF 8 is an operation unit for the photographer to perform an input operation on the image pickup apparatus 1, such as a power button for turning on / off the power of the image pickup apparatus 1, a release button for operating shooting start and end, and the like. Or a touch panel for performing a touch input operation attached to the rear display unit or the like. The photographer operates the input IF 8 to perform various mode settings of the image pickup apparatus 1 and to instruct photographing operations such as release.

  The external memory 9 is a non-volatile storage medium that can be attached to and detached from the imaging device 1 or is fixed inside the imaging device 1. Examples of the external memory 9 that can be attached to and detached from the imaging device 1 include a memory card such as an SD memory card or a CF card. The image data developed by the development processing unit 18 (including composite image data that has been developed) or RAW image data (including composite RAW image data) is recorded in the external memory 9, and the external memory is used for reproduction. 9, recorded image data is read out.

  The system control unit 10 has a CPU (Central Processing Unit), for example, and is a control unit that controls the entire imaging apparatus 1 by controlling each unit in accordance with a processing program stored in the internal memory 4. For example, when receiving an instruction from the photographer via the input IF 8, the system control unit 10 performs timing control such as charge accumulation start and signal reading of the image sensor 13, opening / closing timing control of the mechanical shutter 12, and aperture control of the lens 11. And auto focus control. The system control unit 10 also receives the image data from the image processing unit 5 and performs control for causing the display unit 7 to display an image, control for saving the image data in the external memory 9, and the like.

  Next, FIG. 3 is a flowchart showing a flow of processing for capturing and combining a plurality of frames in the imaging apparatus 1. The processing shown in FIG. 3 is executed by the system control unit 10 controlling each unit in accordance with a processing program stored in the internal memory 4.

  The composition process shown in FIG. 3 is not shown when the photographer selects the bulb photographing mode by the input IF 8 and selects the composition mode in the bulb photographing mode (the mode for displaying the progress during exposure). The main process is executed as a subroutine.

  Therefore, even when the bulb photographing mode is selected, the description of the case where the normal bulb photographing that does not display the progress during the exposure is selected is omitted. 3 will be described with reference to the progress display operation of sequentially synthesizing the image data read out in time series in frame units and displaying the latest synthesized image as the shooting progress.

  When the photographer selects the synthesis mode, the photographer may further select one or more of a comparatively bright synthesis mode, a comparative dark synthesis mode, an addition average synthesis mode, and an addition synthesis mode as more detailed synthesis modes. It can be done. Here, two or more combining modes can be selected at the same time. When two or more combining modes are selected, parallel processing is performed inside the imaging apparatus 1.

  When the processing shown in FIG. 3 is started, first, the mechanical shutter 12 is opened, and the image data read out from the image sensor 13 at a predetermined frame rate is sequentially processed by the image processing unit 5 to be displayed on the display unit 7. Live view display is performed in real time (step S1).

  When this live view display is performed, the focal position is automatically adjusted, for example. Then, the photographer can check the live view image (if an optical viewfinder is further provided, the subject image can be confirmed by the optical viewfinder), and the imaging device can capture the subject to be photographed. The composition is determined by adjusting the direction of 1 and the focal length (zoom) of the lens 11. In addition, when performing live view, the photographer can set a synthesis mode or a more detailed synthesis mode via the operation buttons of the input IF 8 or the touch panel as necessary. Can do.

  Next, the system control unit 10 determines the state of the 1st release switch that is turned on when the release button is half-pressed, and continues to perform the live view display in step S1 until the release button is turned on. If it is determined, the process when the first release is performed is performed (step S2).

  Here, for example, AF (automatic focus control) and AE (automatic exposure control) are performed as processing when the first release is performed. Here, for example, when contrast AF is used as AF, the contrast is extracted from the image data repeatedly read from the image sensor 13, and the focus lens in the lens 11 is driven and controlled so that the extracted contrast becomes the maximum value. To adjust the focus position. AE is a process of automatically controlling the aperture value, ISO sensitivity, and exposure time T so that an image repeatedly read from the image sensor 13 has proper exposure. Depending on the setting of the imaging apparatus 1, one or both of AF and AE are turned off, and the photographer manually operates the focus lens position (focus position), aperture value, ISO sensitivity, exposure via the input IF 8 or the like. Time T etc. may be set.

  After performing the first release process, the system control unit 10 detects the state of the 2nd release switch that is turned on when the release button is fully pressed, and determines whether or not the 2nd release switch is on ( Step S3).

  If it is determined that the 2nd release switch is not on, the process returns to step S1 and the above-described processing is repeated.

  When it is determined that the 2nd release switch is on, the system control unit 10 opens the mechanical shutter 12 and starts charge accumulation (that is, exposure) of the first frame image by the image sensor 13. Further, the timer built in the system control unit 10 is reset to start the operation for measuring the exposure time T (step S4).

  Thereafter, the system control unit 10 determines whether or not the exposure time T has elapsed from the timer measurement result (step S5). The exposure time T is a period (exposure time) for reading out one frame image from the image sensor 13, and is automatically set by AE in step S2 or manually set by the photographer in advance. If the exposure time T has not elapsed, the system waits for the exposure time T to elapse while continuing the exposure.

  When it is determined that the exposure time T has elapsed, the system control unit 10 reads the image signal from the image sensor 13 and immediately after the reading of the image signal ends (that is, the blank time during which charge accumulation is not performed is minimized). The exposure of the next frame is started, and the timer is further reset to start the time counting operation of the exposure time T (step S6). At this time, the mechanical shutter 12 remains open, and the charge accumulation of the next frame is started by the electronic shutter control of the image sensor 13.

  For this reason, it is possible to minimize the omission of exposure between images that are successively captured, and it is possible to minimize the interruption of the movement trajectory of the subject that appears in the finally synthesized image. In a CMOS image sensor generally used as an image sensor 13 of a digital camera, since reading and exposure start can be sequentially controlled for each line, the time of exposure omission between successive frames is It takes about one line readout time. Since this time is very short, for example, about several tens to 100 μs, for example, the final composite image is hardly visually recognized as an image in which the movement trajectory is interrupted.

  Further, the system control unit 10 stores the image signal read from the image sensor 13 in the internal memory 4 as digital image data (step S7). At this time, in addition to the image data read from the effective pixel 13 a of the image sensor 13 shown in FIG. 1, the shading pixel data read from the OB pixel 13 b is also stored in the internal memory 4.

  When the image data is stored in the internal memory 4, the system control unit 10 next determines whether or not the image data to be processed is the first frame image data (step S8).

  Here, when it is determined that the image data is not the first frame (that is, when it is determined that the image data is in the second frame or after), the image data is read from the image sensor 13 and the internal data in step S7. The image data stored in the memory 4 and the composite image already stored in the internal memory 4 are subjected to image composition processing by the image composition unit 15 (step S9).

  For example, assuming that the image data for the second frame has just been read out, the image data for the first frame is stored in the internal memory 4 as a composite image, and the image data for the second frame is further stored in step S7. Are stored, the image composition unit 15 performs image composition processing on these image data. In the case of processing the nth frame (n ≧ 3) and subsequent frames, the combined image from the first frame to the (n−1) th frame and the image acquired as the nth frame are processed in the same manner. Become. Here, the image synthesizing unit 15 applies the pixel at the same pixel position to both the image data read from the effective pixel group including the effective pixels 13a and the image data read from the light-shielding pixel group including the OB pixels 13b. The image composition process for compositing values is performed in the same way.

  Thereafter, the image data obtained as a result of the image composition process is acquired as the first frame when it is determined in step S8 that the image data is the first frame when the image composition process at step S9 is performed. The stored image data is stored in the internal memory 4 as composite image data (step S10).

  Subsequently, pixel defect correction processing is performed by the pixel defect correction unit 17 on the composite image data stored in the internal memory 4, and the processed composite image data is separated from the original composite image data (that is, step The original synthesized image data obtained in the image synthesizing process of S9 is stored in the internal memory 4 (with the original synthesized image data remaining) (step S11). This pixel defect correction processing is performed in the same manner for both the image data read from the effective pixel group and the image data read from the light-shielded pixel group.

  Further, the development processing unit 18 performs the above-described development processing on the composite image data with the pixel defect corrected, and stores it in the internal memory 4 (step S12). This development process is performed in the same manner for both the image data read from the effective pixel group and the image data read from the light-shielded pixel group. The developed composite image data is stored separately from the original composite image data described above, but the composite image data after the pixel defect correction processing may be overwritten.

  Then, the noise amount calculation unit 6 calculates the amount of noise by performing numerical analysis of the OB pixel 13b of the developed composite image data (step S13). Specifically, the noise amount calculation unit 6 calculates, for example, standard deviation (or variance) of the pixel values of the plurality of OB pixels 13b included in the light shielding pixel group. This standard deviation represents variation in the pixel value of the OB pixel 13b, and represents the amount of noise because the OB pixel 13b is shielded from light. Therefore, the calculated standard deviation indicates that the smaller the value, the smaller the noise amount, and the larger the value, the greater the noise amount.

  Although an example of calculating the noise amount of the composite image data that has been developed has been described here, the noise amount of the RAW composite image data before the development processing may be calculated. When the image data is stored in the external memory 9, there are a case where the processed image data is stored, a case where the raw image data is stored, and a case where the processed image data and the raw image data are stored. is there. Since the amount of noise varies depending on whether or not development processing is performed, it is preferable to calculate the amount of noise of image data to be stored depending on which storage setting is selected.

  For example, a photographer who is familiar with shooting chooses to record a RAW image, and later uses the PC (personal computer) or the like to store the RAW image stored in the external memory 9 by using, for example, commercially available image processing software. There are cases where development processing is performed according to preference. For such a photographer, it is useful to calculate a noise amount for the RAW image data and display an image quality index (this image quality index will be described later) based on the calculated noise amount.

  On the other hand, it is sufficient for a general photographer who does not store a RAW image to display the image quality index by calculating the noise amount of the developed image.

  Further, when both the image data after development processing (generally, for example, JPEG data) and the RAW image data are stored in the external memory 9, both noise amounts are calculated, and both image quality indices are set. It may be displayed, or only the image quality index for the image data selected by the photographer as the display target of the image quality index of the two stored image data may be displayed.

  Thereafter, the noise amount calculation unit 6 calculates an image quality index based on the noise amount obtained by numerical analysis (step S14), and the display unit 7 displays the image quality index by superimposing the developed composite image. (Step S15). When displaying the developed composite image here, either one or both of the raw composite image data before the development processing and the developed image data are stored in the external memory 9 as intermediate progress image data. You may make it do.

  Here, with reference to FIGS. 4 to 7, some examples of the image quality index displayed together with the composite image being shot will be described.

  First, FIG. 4 is a diagram illustrating an example in which the ratio of the current noise amount to the noise amount at the start of shooting is displayed as an image quality index.

  In this case, the noise amount calculation unit 6 calculates, as an image quality index, a numerical value indicating a change amount of the noise amount obtained by comparing the noise amount at the start of shooting with the noise amount of the current composite image.

  On the screen 7a of the display unit 7, a composite image that has been developed and is being captured is displayed, and an image quality index is displayed. The image quality index shown in FIG. 4 displays a numerical value indicating how much the current noise amount is relative to the noise amount at the start of photographing. Therefore, the photographer can directly determine that the amount of noise has been halved, for example.

  Next, FIG. 5 is a diagram showing an example of displaying as an image quality index how much the current ISO sensitivity has changed from the ISO sensitivity at the start of shooting.

  The ISO sensitivity corresponds to the gain for the digital output of the pixel, and a photographer who is familiar with shooting generally grasps that the noise is doubled when the ISO sensitivity is doubled, that is, the noise amount is doubled. doing.

  Therefore, the noise amount calculation unit 6 calculates ISO sensitivity as the image quality index at the start of shooting and ISO sensitivity as the image quality index of the current composite image.

  Specifically, the image quality index shown in FIG. 5 indicates how much the current ISO sensitivity has changed to a value corresponding to the ISO sensitivity at the start of shooting.

  In the example shown in FIG. 5, the ISO sensitivity at the start of shooting is 1600. Then, it is assumed that the current noise amount of the combined image obtained by combining the images of a plurality of frames has been improved to ¼ that at the start of shooting. In this case, the photographer can intuitively grasp the amount of noise by displaying 1600/4 = 400 obtained by multiplying the ISO sensitivity at the start of photographing by the amount of noise change as an equivalent value of the current ISO sensitivity. it can.

  Furthermore, FIG. 6 is a diagram illustrating an example of displaying as an image quality index how much the noise amount has been improved as an EV value based on a change from the ISO sensitivity at the start of shooting to the current ISO sensitivity.

Here, the number of stages is APX (Additive system of Photographic Exposure), shutter speed is Tv (Time Value), aperture value is Av (Aperture Value), brightness is Bv (Brightness Value), and ISO sensitivity is Sv (Speed Value). EV value when expressed EV = Bv + Sv = Tv + Av
Unit. If the symbol “^” is used as a symbol representing a power, the amount related to exposure (shutter speed, aperture value, brightness, ISO sensitivity) increases by 2 ^ n times when the number of steps represented by the EV value changes by + n steps. When the number of steps represented by the value changes by -n, the amount related to exposure becomes 2 ^ (-n) times.

  In the example shown in FIG. 6, ISO 1600 at the start of shooting is currently improved to a noise amount equivalent to ISO 400. That is, the ISO sensitivity is 1/4 = 2 ^ (− 2) times, and the ISO sensitivity has changed by −2 steps, so “−2.0 steps” is displayed as the image quality index. . As a result, the photographer can grasp that the image quality has improved by two steps when converted to ISO sensitivity.

  Therefore, the noise amount calculation unit 6 calculates, as the image quality index, a numerical value indicating the amount of change in the noise amount by comparing the noise amount at the start of shooting with the noise amount of the current composite image, and the number at the start of shooting. An ISO sensitivity that is an image quality index and an ISO sensitivity that is an image quality index of the current composite image are calculated.

  FIG. 7 is a diagram illustrating an example in which how much the current dynamic range has changed from the dynamic range at the start of shooting is displayed as an image quality index.

  Here, the dynamic range indicates a range from the lowest pixel value to the maximum pixel value (a pixel value range in which a subject can be accurately captured from a dark portion to a bright portion) in which the subject can be identified. The wider the dynamic range, the more the subject information is photographed without loss, and the higher the image quality, the better the image.

  Here, when the amount of noise is large, the vicinity of the lowest value of the pixel value obtained as subject information is buried in the noise level, so it becomes impossible to determine what is reflected. That is, the larger the noise amount, the narrower the dynamic range.

  Therefore, in the case of a photographer who takes a picture while paying attention to the change of the dynamic range when performing the image composition process, instead of using the magnitude of the noise as an image quality index, as shown in FIG. It is preferable to display as an image quality index how much the current dynamic range has changed from the dynamic range at the start of shooting. In this case, the noise amount calculation unit 6 calculates a numerical value indicating the dynamic range of the image data obtained from the noise amount as the image quality index.

Here, the dynamic range can be expressed as a numerical value with the number of stages as a unit by the following calculation formula using the logarithm “log2” with 2 as a base.
Dynamic range = log 2 {digital maximum value / distinguishable minimum pixel value}

  Here, the maximum digital value is 2 ^ n when an analog signal is A / D converted into an n-bit digital signal (note that the dynamic range is not limited to digital calculation, and is calculated in analog. It's possible, but here we take digital as an example). Further, the minimum discriminable pixel value is not particularly defined. For example, it is defined that the signal can be discriminated without being buried in the noise if it is equal to or larger than the standard deviation value of noise in the dark part (that is, the OB pixel 13b) If so, the standard deviation of noise can be used as the lowest discriminable pixel value.

  As a specific example, when the digital signal is 12 bits, the maximum digital value is 4096 (= 2 ^ 12).

It is assumed that the standard deviation of noise of the OB pixel 13b at the start of photographing is 8. In this case,
Dynamic range at the start of shooting = log 2 {4096/8} = 9 (stage)
It becomes.

Then, it is assumed that the noise standard deviation of the OB pixel 13b of the current composite image as a result of combining a plurality of images is 2 (that is, the noise amount is 1/4 of the start of shooting). In this case,
Current dynamic range = log 2 {4096/2} = 11 (stage)
It becomes. Therefore, the photographer can grasp that the image has a dynamic range that is two steps wider than that at the start of photographing.

  Thus, after performing the process of step S15, the system control unit 10 detects the state of the 2nd release switch and determines whether or not the 2nd release switch is off (step S16). The photographer releases the press of the release button when the bulb photography is finished after the bulb photography is started by pressing the release button. Accordingly, by determining whether or not the 2nd release switch is off, it is possible to determine whether or not the photographer has performed an input operation for ending bulb shooting.

  Here, if it is determined that the 2nd release switch is not off (it remains on), the process proceeds to step S5, and the next frame is repeatedly shot as described above. Thus, every time the exposure time T elapses, the latest composite image is displayed on the display unit 7 as an image indicating the exposure progress. Therefore, the photographer shows how the exposure gradually progresses in the bulb exposure. It can be confirmed as an image.

  On the other hand, if it is determined in step S16 that the 2nd release switch is off, the exposure for bulb exposure is terminated at the timing when the 2nd release switch is determined to be off regardless of whether the exposure time T has elapsed. To do. Therefore, image data in the middle of charge accumulation in the image sensor 13 is not read, and resetting or the like is performed as appropriate.

  Then, the composite image data (one or both of the RAW composite image data and the developed composite image data developed depending on the storage setting) stored in the internal memory 4 when it is determined that the 2nd release switch is turned off. Then, it is stored in the external memory 9 (step S17).

  Further, the developed combined image data stored in the internal memory 4 is displayed on the display unit 7 as a final recorded image (step S18), and the process returns from the combining process to a main process (not shown).

  According to the first embodiment, since the image composition process similar to that of the effective pixel 13a is performed on the OB pixel 13b of the image sensor 13 when performing the image composition process, the amount of noise in the OB pixel 13b is This is the same as the amount of noise included in the effective pixel 13a, and the amount of noise in the effective pixel 13a can be accurately estimated.

  When the variance or standard deviation of the pixel values of the OB pixel 13b is calculated as the noise amount of the composite image, there is an advantage that the processing is easy.

  In addition, since various types of image quality indexes are calculated based on the noise amount calculated by the noise amount calculation unit 6 and are displayed on the display unit 7 by being superimposed on the composite image, any combination method is selected. Regardless of the shooting conditions, the photographer can accurately grasp the image quality related to the noise amount of the current composite image, and can use it as a reference for determining the timing of ending the shooting. .

  In the comparative bright synthesis, comparative dark synthesis, and addition average synthesis, there is an effect of averaging random noise. Therefore, the noise decreases as the number of frames to be synthesized increases. When shooting with these synthesis methods selected to improve noise, the number of frames to be shot (synthesized) varies depending on the intention of how much noise to improve. While grasping the current amount of noise or the amount of noise improvement, shooting can be terminated at the timing of his / her preference.

On the other hand, since noise generated for each frame is accumulated in addition synthesis, the noise increases as the number of frames to be synthesized increases. Since the allowable noise amount differs depending on the photographer, the photographer can finish photographing at a desired timing before reaching the unacceptable noise amount while grasping the current noise amount.
[Embodiment 2]

  8 to 12 show the second embodiment of the present invention, and FIG. 8 is a block diagram showing the configuration of the imaging apparatus 1.

  In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  In the first embodiment described above, a photographer who viewed an image quality index manually finished photographing at a desired timing. On the other hand, in the present embodiment, the target image quality desired by the photographer is set in advance, and the shooting can be automatically terminated when the image quality of the currently captured image reaches the target image quality. It has become.

  First, as shown in FIG. 8, the imaging apparatus 1 of the present embodiment is obtained by adding a noise amount determination unit 26 to the configuration of the imaging apparatus 1 shown in FIG. . The noise amount determination unit 26 is connected to the system control unit 10 and the bus 3. Although the noise amount determination unit 26 is configured separately from the system control unit 10 here, the system control unit 10 may also serve as the noise amount determination unit 26.

  Next, FIG. 9 is a flowchart illustrating a flow of processing for capturing and combining a plurality of frames in the imaging apparatus 1.

  First, the photographer sets a target image quality via the input IF 8 before starting bulb photographing. Therefore, the input IF 8 in this embodiment functions as a target noise amount setting unit that sets a target noise amount.

  For example, when the selected combining method is addition average combining, the noise improves as the number of frames to be combined increases. Therefore, the photographer is more than the set ISO sensitivity at the start of shooting (for example, ISO 1600). A low ISO sensitivity (for example, equivalent to ISO 400) is set as the target image quality. The target image quality set here is converted into a noise amount and stored in the internal memory 4 or the like as the target noise amount.

  And the process shown in FIG. 9 is started and the process of step S1-S13 is performed similarly to the process shown in FIG. 3 of Embodiment 1 mentioned above.

  Next, the noise amount determination unit 26 determines whether or not the noise amount calculated in step S13 has reached the target noise amount corresponding to the target image quality (step S21).

  If it is determined that the target noise amount has not been reached, the image quality index of the target image quality is displayed together with the latest synthesized image and the current image quality index after performing the process of step S14 (step S15A). ).

  FIG. 10 is a diagram showing a display example when the target image quality is set by the ISO sensitivity.

  The screen 7a of the display unit 7 shown in FIG. 10 displays that the ISO sensitivity at the start of photographing is 1600 and the target image quality is set to be equivalent to ISO sensitivity 200, but the current ISO sensitivity is equivalent to 400. It is an example to do.

  FIG. 11 is a diagram illustrating a display example when the target image quality is set by the noise improvement amount.

  The example shown in FIG. 11 displays that the target noise improvement amount, which is the target image quality, is set to −3.0 steps, but the current noise improvement amount is stopped at −2.0 steps. .

  Further, FIG. 12 is a diagram showing a display example when the target image quality is set by the number of steps of the dynamic range improvement amount.

  In the example shown in FIG. 12, the dynamic range at the start of shooting is 9 EV and the target dynamic range improvement amount that is the target image quality is set to +3 EV, but the current dynamic range is 11 EV, that is, the dynamic range is improved. The amount is stopped at +2 EV.

  By performing the display as shown in these examples, the photographer can grasp the degree of achievement with respect to the target image quality.

  Thereafter, the process goes to step S16, and the system control unit 10 detects the state of the 2nd release switch and determines whether or not the 2nd release switch is off, as in the first embodiment.

  That is, even when the target image quality is not reached, it is sufficiently assumed that the user wants to end shooting in the middle of shooting. For example, there is a case where the degree of achievement with respect to the target image quality does not progress slowly and it is not possible to wait for the completion of the achievement, or when it is desired to end the shooting partway for some reason. Therefore, similarly to the first embodiment described above, the 2nd release switch is determined to be off so that the photographing can be terminated by the photographer's intention.

  Thus, if it is determined in step S21 that the target noise amount has been reached, or if it is determined in step S16 that the 2nd release switch is OFF, the processing in steps S17 and S18 is performed. Return to the main process (not shown).

  According to the second embodiment, since the effects similar to those of the first embodiment described above are obtained and the photographing is automatically terminated when the set target image quality is reached, the photographer's intention is achieved. Reflected shooting can be performed automatically. Therefore, it is not necessary for the photographer to constantly monitor the image quality index, and it is possible to reduce the burden on the photographer when bulb photographing takes a long time.

In this way, shooting can be completed at an optimal timing, and shooting failures can be suppressed.
[Embodiment 3]

  FIG. 13 shows a third embodiment of the present invention, and is a flowchart showing a flow of processing for photographing and combining a plurality of frames in the imaging apparatus 1.

  In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  If at least one of the longer exposure time (long exposure) and the temperature of the image sensor 13 occurs, the influence of dark current noise generated in the PD (photodiode) of the image sensor 13 increases. To do. This dark current noise appears in the dark part of the image as shading or defective FPN, leading to a reduction in image quality.

  On the other hand, in the imaging apparatus 1 such as a digital camera, when the exposure is performed for a long time or when the temperature of the image sensor 13 is high, the exposure time of the bright image captured by opening the mechanical shutter 12 is the same as the exposure time. In general, an FPN cancellation process for correcting the FPN is performed by closing the mechanical shutter 12 to capture a dark image and subtracting the dark image from the bright image for each same pixel position.

  Therefore, in the present embodiment, before the system control unit 10 opens the mechanical shutter 12 and causes the image sensor 13 to generate a plurality of frames of bright image data, the mechanical shutter 12 is closed and the dark image data. Is generated by the image sensor 13. Further, the FPN cancellation processing unit 16 performs FPN cancellation processing for subtracting dark image data from the combined image data related to the bright image data calculated by the image combining unit 15. Then, the noise amount calculation unit 6 calculates the noise amount based on the light-shielded pixel data that has been subjected to the image synthesis process and the FPN cancellation process.

  That is, the FPN cancellation processing unit 16 according to the present embodiment uses the dark image data to fix the fixed pattern in the composite image data even when the bright image data is the composite image data obtained by the image composition unit 15. It is intended to correct noise.

  Here, in this embodiment, assuming that the exposure time T of each frame is constant and the combining method is comparative bright combining, comparative dark combining, or addition average combining, it is suitable for these combining methods. The FPN cancellation process will be described.

  In addition, in the case of additive synthesis, as the number of frames to be synthesized increases, not only FPN but also random noise increases. Therefore, the FPN can be improved only by using a single dark image captured at the exposure time T. However, it is difficult to improve the random noise, that is, the image quality of the image showing the exposure process cannot be improved so much. Accordingly, here, as the image synthesis processing, comparatively bright synthesis, comparative dark synthesis, or addition average synthesis is assumed.

  Under the assumption that the exposure time T of each frame is constant, the FPN included in each frame may be considered to be approximately equal. Therefore, the dark image with the exposure time T is immediately before shooting the bright image. The image quality of an image showing the exposure progress can be improved by taking an image and performing the FPN cancellation processing on the image displayed as the progress in the middle.

  When the process shown in FIG. 13 is started, the processes of steps S1 to S3 are performed.

  When it is determined in step S3 that the 2nd release switch is on, the system control unit 10 first closes the mechanical shutter 12 and takes a dark image with an exposure time T by the image sensor 13 ( In step S31, the captured dark image is stored in the internal memory 4 (step S32).

  After that, the mechanical shutter 12 is opened and the processes of steps S6 to S10 are performed. The bright image is photographed, a composite image is calculated and stored in the internal memory 4.

  Subsequently, the FPN cancellation process is performed by subtracting the dark image from the combined image obtained by combining the bright image for each same pixel position (step S33). This FPN cancellation process is performed not only on the effective pixel group including the effective pixels 13a but also on the light-shielding pixel group including the OB pixels 13b. The composite image that has been subjected to the FPN cancellation processing is stored in the internal memory 4 separately from the original composite image. The composite image calculated in step S9 is an image obtained by combining the composite image that has not been subjected to the FPN cancellation process and the newly captured image.

  Next, the pixel defect correction process in step S11 and the development process in step S12 are performed on the composite image that has been subjected to the FPN cancellation process.

  Further, the noise amount calculation unit 6 calculates the noise amount of the composite image data developed after the FPN cancellation processing by performing numerical analysis of the OB pixel 13b (step S13A), and calculates the image quality index based on the noise amount. (Step S14A).

  Thereafter, the processing of step S15 and step S16 is performed, and if it is determined in step S16 that the 2nd release switch is off, the composite image stored in the internal memory 4 when it is determined that the 2nd release switch is off. FPN cancellation processing is performed on the data (composite image data that has not yet been subjected to FPN cancellation processing) (step S34) and stored in the external memory 9 (step S17A).

  Subsequently, the process of step S18 is performed, and the process returns from the synthesis process to a main process (not shown).

According to the third embodiment, the same effects as those of the first and second embodiments described above can be obtained, and at the time when each frame is acquired using the dark image captured immediately before the bright image is captured. Since the FPN cancellation processing of the composite image is performed, the noise amount is calculated from the OB pixel 13b of the composite image after the FPN cancellation processing, the image quality index is generated, and displayed together with the composite image after the FPN cancellation processing. It is possible to take a picture while more accurately grasping the image quality related to.
[Embodiment 4]

  FIGS. 14 to 16 show Embodiment 4 of the present invention. FIG. 14 is a timing chart showing the operation of each part in the imaging apparatus 1. FIG. FIG. 16 is a flowchart illustrating a noise amount calculation process after FPN cancellation in the imaging apparatus 1.

  In the fourth embodiment, the same parts as those in the first to third embodiments are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

  In Embodiment 3 described above, a dark image is taken immediately before starting bulb photography, but in this embodiment, a dark image is taken immediately after bulb photography is finished. Furthermore, in the third embodiment, FPN cancellation processing of the composite image at each time point during shooting is performed using the acquired dark image, and the noise amount of the composite image after the FPN cancellation processing is calculated and the image quality index is displayed. In this embodiment, residual noise (random noise) when FPN cancellation processing is performed from a composite image before FPN cancellation processing at each time point during shooting is estimated as a noise amount and an image quality index is displayed. Is.

  First, the flow of synthesis processing in this embodiment will be described with reference to FIGS.

  When this process is started, the processes in steps S1 to S12 are performed as described above. As a result, every time the exposure time T elapses in step S5, the bright image is read out, and as shown in FIG. 14, the first bright image, the second bright image, and the third bright image are sequentially processed from the first frame. An image, a fourth light image, etc. are obtained. Each time a new bright image is captured, a composition process is performed in step S9, and a pixel defect correction process in step S11 and a development process in step S12 are performed.

  Next, the noise amount calculation unit 6 estimates and calculates the noise amount after the FPN cancellation processing of the composite image that has been subjected to the development processing (step S41).

  That is, the noise amount calculation unit 6 is adjacent in order to the noise amount calculated based on the light-shielded pixel data of the composite image data and the noise amount calculated based on the light-shielded pixel data of each frame sequentially read from the image sensor 13. FPN cancellation processing each time one frame of bright image data is read from the image sensor 13 using the noise amount calculated based on the subtracted image data obtained by subtracting the light-shielded pixel data of two frames for each pixel position. The amount of noise of the composite image data when the process is performed is calculated. The post-FPN cancellation noise amount calculation processing will be described in detail later with reference to FIG.

  Subsequently, in step S15, the developed composite image is displayed as a display image on the display unit 7, as shown in FIG.

  Further, the process of step S16 is performed, and if it is determined in step S16 that the 2nd release switch is off, the system control unit 10 closes the mechanical shutter 12 and the image sensor 13 performs a dark image with an exposure time T. Is captured (step S42), and the captured dark image is stored in the internal memory 4 (step S43).

  Further, in step S34, FPN cancellation processing is performed on the composite image data stored in the internal memory 4 using the dark image stored in the internal memory 4 in step S43. In step S17A, post-processing composite processing is performed. Image data (final recorded image shown in FIG. 14) is stored in the external memory 9.

  Thereafter, the process of step S18 is performed, and the process returns from the synthesis process to a main process (not shown).

  Next, the principle of the noise amount calculation process after FPN cancellation in step S41 described above will be described.

First, in general, the standard deviation σ indicating the amount of noise included in one image has a random noise amount (standard deviation: σr) and an FPN amount (standard deviation: σf) as shown in the following Equation 1. And the square root of the sum of squares.
[Equation 1]
σ = {σr ^ 2 + σf ^ 2} ^ (1/2)

Therefore, if the number of frames indicating the number of frames is represented by a positive integer n, the random noise amount of the n-th frame image is σr_n, and the FPN amount is σf_n, the n-th frame image (one frame image) ) N included in () is expressed by the following mathematical formula 2.
[Equation 2]
σn = {σr_n ^ 2 + σf_n ^ 2} ^ (1/2)

On the other hand, the noise amount σ total n of the synthesized image obtained by image synthesis processing of the first to n-th frame images is also expressed as follows: 1 is represented by the following formula 3.
[Equation 3]
σgo n = {σr_go n ^ 2 + σf_go n ^ 2} ^ (1/2)

The amount to be obtained now is the noise amount of the composite image after the FPN cancellation processing, that is, σr_total n. Therefore, when Equation 3 is transformed into an equation for obtaining σr_total n, Equation 4 below is obtained.
[Equation 4]
σr_go n = {σ go n ^ 2−σf_go n ^ 2} ^ (1/2)

  The noise amount σ total n of the composite image on the right side of Equation 4 is calculated from the standard deviation of the pixel values of the OB pixel 13b of the composite image.

  On the other hand, if the FPN amount σf_1, σf_2, σf_3,. The random noise amount σr_total n of the composite image corresponding to several synthesis methods is also calculated as follows if the random noise amounts σr_1, σr_2, σr_3,.

First, when addition synthesis is used, the following formulas 5 and 5A are obtained.
[Equation 5]
σf_combined n = σf_1 + σf_2 + σf_3 +… + σf_n
[Formula 5A]
σr_go n = {σr_1 ^ 2 + σr_2 ^ 2 + σr_3 ^ 2 +… + σr_n ^ 2} ^ (1/2)

Next, in the case of using addition average synthesis, the following formulas 6 and 6A are obtained.
[Equation 6]
σf_combined n = {σf_1 + σf_2 + σf_3 + ... + σf_n} / n
[Formula 6A]
σr_go n = {(σr_1 ^ 2 + σr_2 ^ 2 + σr_3 ^ 2 +… + σr_n ^ 2) / 2} ^ (1/2)

Subsequently, in the case of using comparatively bright combination, if the function max that gives the maximum value is used, the following Expression 7 is obtained.
[Equation 7]
σf_ total n = max {σf_1, σf_2, σf_3, ..., σf_n}

Further, in the case of using comparative dark synthesis, if a function min that gives a minimum value is used, the following formula 8 is obtained.
[Equation 8]
σf_total n = min {σf_1, σf_2, σf_3, ..., σf_n}

  Therefore, if the FPN amounts σf_1, σf_2, σf_3,. Will be understood.

  Therefore, it is considered that the FPN amount σf_n of the n-th frame image is calculated based on the (n−1) -th frame image and the n-th frame image.

Here, the noise amount σ n of the n-th frame image is expressed by the above-described equation 2, and the noise amount σ n-1 of the (n−1) -th frame image is expressed by the following equation 9 according to the equation 1. expressed.
[Equation 9]
σn-1 = {σr_n-1 ^ 2 + σf_n-1 ^ 2} ^ (1/2)

Here, it is assumed that the exposure time T and the ISO sensitivity at the time of capturing an image of each frame are the same. Furthermore, it is assumed that the temperature of the image sensor 13 does not substantially change between the time when the (n-1) -th frame image is captured and the time when the n-th frame image is captured. At this time, approximation as shown in the following formulas 10 and 10A can be performed.
[Equation 10]
σr_n-1 ≒ σr_n
[Equation 10A]
σf_n-1 ≒ σf_n

  Note that what is described here is only that the random noise amount and the FPN amount are approximated between adjacent frames (frame numbers), for example, σr_n−1≈σr_n, and σr_n− Note that just because 2≈σr_n−1 does not always mean that σr_n−2≈σr_n.

  Then, with respect to the OB pixel 13b of the (n-1) -th frame image and the n-th frame image that are adjacent in order, the pixel values at the same pixel position are subtracted, and the pixel value of the OB pixel 13b in the subtraction image From the standard deviation, noise amounts σn−1 to n included in the subtracted image are calculated. When this subtraction process is performed, the FPN is canceled, so that the noise amounts σn−1 to n included in the subtracted image can be considered to be noise amounts of only random noise.

Here, since the random noise is generated regardless of the + and − directions, the amount of noise when the subtraction process is performed is the amount of random noise of (n−1) frames as in the case of addition synthesis. It can be represented by the square root of the sum of squares of σr_n−1 and the random noise amount σr_n of n frames. Therefore, the noise amounts σn−1 to n included in the subtracted image are expressed as shown in the following Expression 11.
[Equation 11]
σn-1 ～ n = {σr_n ^ 2 + σr_n-1 ^ 2} ^ (1/2)

When the approximation of Expression 10 is applied to Expression 11, the following Expression 12 is obtained.
[Equation 12]
σn-1 ～ n = {σr_n ^ 2 + σr_n-1 ^ 2} ^ (1/2)
≒ {σr_n ^ 2 +＋ σr_n ^ 2} ^ (1/2)
= 2 ^ (1/2) × σr_n

Rewriting this equation 12 into an equation for obtaining σr_n, the following equation 13 is obtained.
[Equation 13]
σr_n ≒ (σn-1 ～ n) / (2 ^ (1/2))

Therefore, when the mathematical formula 13 is applied after rewriting the mathematical formula 2 into a formula for obtaining σf_n, the following mathematical formula 14 is obtained.
[Formula 14]
σf_n = {σn ^ 2-σr_n ^ 2} ^ (1/2) = {σn ^ 2- (σn-1 to n ^ 2) / 2} ^ (1/2)

  If the value of the FPN amount σf_n of each frame is calculated in this way, the FPN amount σf_total n of the composite image can be calculated by using any one of the above-described mathematical formulas 5, 6, 7, and 8. As a result, the noise amount σr_total n of the composite image after performing the FPN cancellation process can be obtained by Expression 4.

  Details of the processing for calculating the noise amount after FPN cancellation in step S41 based on such a principle will be described with reference to FIG.

  When this processing is started, the noise amount σn of the OB pixel 13b of the n-th frame image read out from the image sensor 13 is calculated as, for example, a standard deviation, and each image of the first to n-th frames is obtained by synthesizing processing. The noise amount σ total n of the obtained composite image is calculated as, for example, a standard deviation (step S51).

  Next, for each pixel position, image calculation is performed to subtract the pixel value of the OB pixel 13b of the (n-1) -th frame image from the pixel value of the OB pixel 13b of the n-th frame image (step S52). .

  Subsequently, the noise amount σn−1 to n of the OB pixel 13b of the subtraction image is calculated as a standard deviation, for example (step S53).

  Further, using the noise amount σn calculated in step S51 and the noise amounts σn−1 to n calculated in step S53, the FPN amount σf_n of the n-th frame image is calculated by Equation 14 (step S54).

  The FPN amount σf_n calculated here is stored in the internal memory 4 (step S55). Thereby, the FPN amount of each frame is sequentially stored in the internal memory 4.

  Then, the FPN amounts σf_1, σf_2, σf_3,. n is calculated (step S56).

  Thereafter, using the noise amount σ total n calculated in step S51 and the FPN amount σ f_total n calculated in step S56, the noise amount σr_total n of the composite image after performing the FPN cancellation processing is expressed as 4 (step S57), and the process returns from this process.

  In the case of shooting a dark image immediately before starting the shooting of a bright image as in the third embodiment described above, from when the 2nd release switch is turned on by the release button until the actual shooting of the bright image is started. Because of the long time lag, for example, when shooting fireworks, there was a case where the fireworks desired to be captured were not captured.

  On the other hand, according to the fourth embodiment, the effects similar to those of the first to third embodiments are obtained, and the dark image is captured immediately after the bright image capturing is completed. In addition, it is possible to reduce cases where a photo opportunity is missed.

  In the above description, the bulb photographing is started by pressing the release button and the bulb photographing is stopped by releasing the press. However, the present invention is not limited to this. For example, bulb photography is started by pressing the release button. Once bulb photography has started, bulb photography continues even if the release is released, and when the release button is pressed again, bulb photography is stopped. Also good.

  In addition, as described above, any one of the comparative bright combination, comparative dark combination, addition average combination, and addition combination can be set, and two or more combinations may be set simultaneously. . Then, the composite image indicating the progress during bulb shooting may be displayed only for the comparatively bright composite image, only the comparative dark composite image, or only the average composite image, or may simply display the added image. good.

  Furthermore, although the imaging apparatus 1 has been mainly described above, an imaging method that performs the same processing as the imaging apparatus 1 may be used, and a processing program for causing a computer to perform the same processing as the imaging apparatus 1, the processing It may be a non-temporary recording medium that can be read by a computer that records the program.

  Specifically, in the above description, the processing for the imaging device 1 has been described. However, the present invention is not limited to this, and image data of a plurality of frames is acquired by the imaging device 1, and the above-described processing for the acquired image data of the plurality of frames is described above. Such image processing may be performed.

  Of the techniques described in this specification, the control described mainly in the flowchart is often executable by a processing program, and this processing program may be stored in a recording medium or a recording unit. The recording method for the recording medium or recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via a communication line such as the Internet.

  Further, for the explanation of the operation in the claims, the specification, and the drawings (flowcharts, etc.), for the sake of convenience, “first”, “next”, “follow”, “after”, and the like are used. Even if explained, it does not mean that it is indispensable to carry out in this order in a portion not particularly explained.

  The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. In addition, various aspects of the invention can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Imaging device 2 ... Imaging part 3 ... Bus 4 ... Internal memory 5 ... Image processing part 6 ... Noise amount calculation part 7 ... Display part 7a ... Screen 8 ... Input IF
DESCRIPTION OF SYMBOLS 9 ... External memory 10 ... System control part 11 ... Lens 12 ... Mechanical shutter 13 ... Image sensor 13a ... Effective pixel 13b ... OB pixel 15 ... Image composition part 16 ... FPN cancellation process part 17 ... Pixel defect correction part 18 ... Development process part DESCRIPTION OF SYMBOLS 21 ... Comparative light composition part 22 ... Comparative dark composition part 23 ... Addition composition part 24 ... Average composition part 26 ... Noise amount determination part

Claims (13)

An image pickup device having an effective pixel group for generating image data relating to an optical image of a subject, and a light-shielded pixel group for generating light-shielded pixel data arranged to be shielded from light around the effective pixel group;
The image data of a plurality of frames sequentially read out from the effective pixel group is subjected to image composition processing to calculate one frame of combined image data, and the plurality of frames sequentially read out from the light-shielding pixel group. An image composition unit for performing the same image composition processing as that performed on the image data with respect to the light-shielded pixel data;
A noise amount calculation unit that calculates a noise amount based on the shading pixel data subjected to the image synthesis processing;
With
The image synthesizing unit and the noise amount calculating unit perform the processing every time one frame of the image data and the light-shielded pixel data are read from the image sensor. An index calculation unit that calculates an image quality index representing the image quality of the composite image data from the noise amount calculated by the noise amount calculation unit;
A display unit for displaying the image quality index;
The imaging apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 1, wherein the image synthesis unit performs an addition average synthesis process as the image synthesis process.   The imaging apparatus according to claim 1, wherein the image composition unit performs a comparatively bright composition process or a comparative dark composition process as the image composition process.   The imaging apparatus according to claim 1, wherein the image composition unit performs addition composition processing as the image composition processing. An optical shutter that controls opening and closing of the optical image of the subject reaching and not reaching the image sensor;
Prior to generating the light image data of a plurality of frames with the optical shutter opened, the control unit causes the image sensor to generate dark image data with the optical shutter closed.
An FPN cancellation processing unit for performing an FPN cancellation process for subtracting the dark image data from the combined image data related to the bright image data calculated by the image combining unit;
Further comprising
The imaging apparatus according to claim 1, wherein the noise amount calculation unit calculates a noise amount based on the light-shielded pixel data subjected to the image synthesis process and the FPN cancellation process. An optical shutter that controls opening and closing of the optical image of the subject reaching and not reaching the image sensor;
A control unit for causing the image sensor to generate dark image data by closing the optical shutter after the optical shutter has been opened and the image sensor has generated all the light image data of a plurality of frames;
After the generation of the light image data of the plurality of frames is completed, an FPN cancellation process is performed to subtract the dark image data from the composite image data related to the light image data calculated by the image composition unit. An FPN cancellation processing unit;
Further comprising
The noise amount calculation unit is adjacent in order to the noise amount calculated based on the light-shielded pixel data of the composite image data and the noise amount calculated based on the light-shielded pixel data of each frame sequentially read from the imaging device. Each time the light image data of one frame is read from the image sensor using the noise amount calculated based on the subtracted image data obtained by subtracting the light-shielded pixel data of two frames for each pixel position, the FPN The imaging apparatus according to claim 1, wherein a noise amount of the composite image data when a cancel process is performed is calculated. A target noise amount setting unit for setting a target noise amount;
A noise amount determination unit that determines whether or not the noise amount of the light-shielded pixel data subjected to the image composition processing calculated by the noise amount calculation unit has reached the target noise amount;
When the noise amount determination unit determines that the target noise amount has been reached, the control unit causes the image sensor to finish reading the image data;
The imaging apparatus according to claim 1, further comprising:   The imaging apparatus according to claim 2, wherein the index calculation unit calculates the image quality index at the start of shooting and the image quality index of a current composite image.   The imaging apparatus according to claim 2, wherein the index calculation unit calculates ISO sensitivity as the image quality index.   The said index calculation part calculates the numerical value which shows the variation | change_quantity of the noise amount which compared the noise amount at the time of imaging | photography start with the noise amount of the present synthesized image as an image quality index | exponent, Imaging device.   The imaging apparatus according to claim 2, wherein the index calculating unit calculates a numerical value indicating a dynamic range of image data obtained from a noise amount as an image quality index. Reading out image data related to the optical image of the subject from the effective pixel group, and reading out the light-shielded pixel data from the light-shielded pixel group arranged to be shielded from light around the effective pixel group;
Each time one frame of the image data and the light-shielded pixel data are read, image synthesis processing is performed on the plurality of frames of image data sequentially read from the effective pixel group to calculate one frame of combined image data. And performing the same image composition processing as that performed on the image data with respect to the light-shielded pixel data of a plurality of frames sequentially read out from the light-shielded pixel group,
Calculating the amount of noise based on the shaded pixel data on which the image composition processing has been performed each time the image data and the shaded pixel data of one frame are read;
An imaging method characterized by comprising:

Images