JP2017028568A - Imaging device and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device and imaging method that can prevent erroneous correction in noise removing processing without increasing the scale of a correction circuit due to noise removing processing.SOLUTION: An imaging device includes an imaging element for photoelectrically converting a subject image to generate image data, an imaging optical system for forming a subject image on the imaging element, and an exposure interruption controller for interrupting image formation of the subject image on the imaging element. Normal exposure from the start of exposure to the end of exposure is controlled under the state that a subject image is formed on the imaging element by the imaging optical system (S3 to S11), shading exposure from the start of exposure to the end of exposure is controlled under the state that imaging formation of the subject image is interrupted (S13 to S19), noise extraction is performed (S21) based on shaded image data generated by the imaging element after the shading exposure, and noise correction is performed on normal exposure image data generated by the imaging element based on a noise extraction result after the normal exposure is performed.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging apparatus and an imaging method for performing noise correction processing on image data in an imaging apparatus that acquires image data using an imaging element.

  In an image pickup apparatus such as a digital camera, various noises are generated in the image pickup element. Therefore, noise correction processing for reducing these noises is known. Examples of the noise include fixed pattern noise (hereinafter referred to as “FPN”) caused by a pixel defect of the image sensor, and dark current shot noise (a kind of random noise) of the image sensor. The following method has been proposed as a method for removing FPN when shooting is performed under shooting conditions that make these noises conspicuous (for example, long exposure of several seconds or more).

(Case 1):
In one shooting, first, an image is acquired by long-time exposure shooting (this image is referred to as “image 1”), and after acquisition, the image is acquired in the same long second time in a light-shielded state. (The image acquired in this light-shielded state is referred to as “image 2”). Then, by subtracting the two acquired images, that is, by calculating “image 1” − “image 2”, an image in which pixel defect correction has been performed is obtained (see Patent Document 1).

(Case 2):
In Case 1, an image was acquired even in a light-shielded state in one shooting. However, in case 2, in one shooting, the image is acquired only by the long-time exposure shooting without performing the shooting in the light-shielded state, and the FPN is corrected using this image (see Patent Document 2). . In Case 2, FPN correction is performed by the following two correction processes, correction a and correction b.

(Correction a): Defect information (address and level of defective pixels) is recorded in the recording unit in advance under shooting conditions that cause the influence of FPN. And every time it image | photographs, defect information is read from a memory | storage part and it correct | amends with respect to a picked-up image.
(Correction b): The pixel output of the defective pixel is corrected by comparing or calculating the pixel output with the surrounding pixels.

JP 2010-118963 A JP 2012-100215 A

  In the case 1 method described above, defect correction can be performed, but random noise is increased by subtracting two images, and image quality deterioration occurs due to causes other than FPN. If the noise reduction process is strengthened to correct this increased random noise, the resolution will deteriorate.

  Further, in the correction a in the method of Case 2, since the defect information is different when the photographing condition (external temperature, ISO sensitivity, exposure time) is changed, the defect information must be recorded for each photographing condition in the storage unit. In other words, the memory capacity of the storage unit becomes enormous. For example, if exposure is performed for several seconds in a room temperature environment and about 2% of all pixels are defective, an image sensor having a number of pixels equivalent to 20M requires 40,000 defect registrations for one condition. In this case, the number of registered memories is the number of parameters × 40,000 such as ISO sensitivity, temperature, exposure time, etc., and the circuit scale of the storage unit becomes enormous.

  In the correction b in the case 2 method, the main subject is also included in the image, so that the defect detection accuracy is lowered, and small bright spots such as stars may be corrected by mistake.

  The present invention has been made in view of such circumstances, so that the correction circuit including the storage unit involved in the noise removal process does not become large-scale, and erroneous correction is prevented in the noise removal process. An object of the present invention is to provide an imaging apparatus and an imaging method.

  In order to achieve the above object, an image pickup apparatus according to a first invention has a plurality of pixels on an image pickup surface, photoelectrically converts an object image to generate image data, and the object image is applied to the image pickup element. An imaging optical system that forms an image, an exposure blocking control unit that blocks the imaging of the subject image on the imaging device, and exposure from the start of exposure in a state where the subject image is formed on the imaging device by the imaging optical system An exposure controller that controls light exposure from the start of exposure to the end of exposure in a state in which the normal exposure until the end is controlled and the image formation of the subject image is blocked by the exposure block controller; After performing the normal exposure based on the noise extraction unit that performs noise extraction based on the light-shielded image data generated by the image sensor and the noise extraction result by the noise extraction unit. The normal exposure image data generated by the imaging device comprises a noise correction unit that performs noise correction, a.

In the imaging device according to a second aspect based on the first aspect, the noise extraction unit calculates an intermediate value between each value of the pixel of the light-shielded image data and a pixel surrounding the pixel, and the intermediate value The position of the defective pixel is obtained based on the comparison between the threshold value and the threshold value.
In the imaging device according to a third aspect, in the second aspect, the noise correction unit corrects the pixel value at the position of the defective pixel by interpolation calculation using the pixel values of surrounding pixels.

According to a fourth aspect of the present invention, in the first aspect, the imaging device further includes an optical black pixel region that is a pixel region composed of a plurality of pixels that are constantly shielded from light. The noise extraction unit calculates a difference value between a maximum value and a minimum value for each block of a pixel output representative value in the optical black pixel region and a pixel of the light-shielded image data, and based on a comparison between the difference value and a threshold value. Then, depending on the distance from the center area of the imaging surface of the imaging element, the shading component, which is a change in pixel output in the light-shielded image data, or the pixel output of the light-shielded image data, and the pixels in the optical black pixel area An optical black level difference component that is a difference from the representative output value is obtained.
In the imaging device according to a fifth aspect based on the fourth aspect, the noise correction unit performs noise correction by subtracting the shading component or the optical black step component from the pixel value of the normal exposure image data. Do.

In the imaging device according to a sixth aspect of the present invention, in the third or fifth aspect, the correction amount in the noise correction unit is at least one of the conditions of ISO sensitivity, exposure time, and temperature during normal exposure or light-shielding exposure. Change according to.
In the imaging device according to a seventh aspect of the present invention, in the third or fifth aspect of the invention, the correction amount in the noise correction unit is obtained when a plurality of sheets are normally exposed under at least one of the conditions of ISO sensitivity, exposure time, and temperature. Different at each exposure.

In the imaging apparatus according to an eighth aspect based on the first aspect, the exposure control unit does not depend on an exposure time when performing the light-shielding exposure on an exposure time when performing the normal exposure.
In the imaging apparatus according to a ninth aspect based on the first aspect, the exposure control unit sets the photographing condition at the time of the light shielding exposure, the photographing condition at the time of normal exposure photographing, or the photographing condition set by an external operation. Or, it is determined according to the number of shots at the time of multiple exposure.

  The image pickup apparatus according to a tenth aspect of the present invention is the image pickup apparatus according to the first aspect, further comprising an additional light-shielding photographing determination unit that determines whether or not to perform additional light-shielding photographing based on a noise extraction result by the noise extraction unit. As a result of the determination by the additional light shielding photographing determination unit, when additional light shielding photographing is performed, noise extraction is performed on the light shielding image data acquired by the additional light shielding photographing, and the noise correction is performed.

  An image pickup method according to an eleventh aspect of the invention is an image pickup device that photoelectrically converts a subject image to generate image data, a photographing optical system that forms the subject image on the image pickup device, and an image pickup device that captures the subject image on the image pickup device. And an exposure block control unit that blocks image formation, and controls normal exposure from the start of exposure to the end of exposure in a state where the subject image is formed on the image sensor by the imaging optical system. In addition, in the state where the image formation of the subject image is blocked by the exposure blocking control unit, the light blocking exposure from the start of exposure to the end of exposure is controlled, and the light blocking generated by the image sensor after the light blocking exposure is performed. Based on the image data, noise extraction is performed, and based on the noise extraction result, the normal exposure image data generated by the image sensor after the normal exposure is performed, Perform the size correction.

  According to the present invention, it is possible to provide an image pickup apparatus and an image pickup method capable of preventing erroneous correction in a noise removal process without increasing the scale of a correction circuit including a storage unit associated with the noise removal process. it can.

1 is a block diagram mainly showing an electrical configuration of an imaging apparatus according to a first embodiment of the present invention. 3 is a flowchart illustrating a main operation of the imaging apparatus according to the first embodiment of the present invention. 3 is a flowchart illustrating an operation of noise detection processing of the imaging apparatus according to the first embodiment of the present invention. 3 is a flowchart showing an operation of noise correction processing of the imaging apparatus according to the first embodiment of the present invention. It is a figure for demonstrating a filter process in the imaging device in 1st Embodiment of this invention. It is a figure explaining performing noise correction by calculating | requiring the difference of image data in the imaging device in 1st Embodiment of this invention. It is a figure explaining the image block average process in the case of the light shielding filter process in the imaging device in 1st Embodiment of this invention. It is a figure explaining the example of correction of a defective pixel in the imaging device in a 1st embodiment of the present invention. It is a figure explaining the correction example of a block offset in the imaging device in 1st Embodiment of this invention. It is a block diagram which mainly shows the electric constitution of the imaging device which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the imaging device which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the light-shielding imaging condition calculation process of the imaging device which concerns on 2nd Embodiment of this invention. It is a figure which shows the example of the menu screen of dark condition setting in the imaging device which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the imaging device which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the main operation | movement of the imaging device which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of the additional light-shielding imaging condition judgment process of the imaging device which concerns on 3rd Embodiment of this invention. It is a graph explaining block offset correction coefficient calculation in the imaging device concerning a 3rd embodiment of the present invention.

  Hereinafter, an example applied to a digital camera as an imaging apparatus according to an embodiment of the present invention will be described. The imaging apparatus includes an imaging unit, and converts the subject image into image data by the imaging unit, and displays the subject image on a display unit disposed on the back of the main body based on the converted image data. . The photographer determines the composition and shutter timing by observing the live view display. During the release operation, image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

  In addition, in the case of long-time exposure photography such as bulb photography, this imaging apparatus repeatedly performs exposure at predetermined time intervals, and noise generated in the image data using a plurality of photographed image data acquired by this exposure. Correction processing is performed so as to reduce.

  First, a first embodiment of the present invention will be described with reference to FIGS. In the present embodiment, the normal exposure photographing and the light shielding photographing are performed with a predetermined exposure time, and noise extraction (separation of defective pixel components and normal pixels) is performed from the light shielding image. Next, noise correction of the normal exposure photographed image is performed based on the normal exposure and shading photographing conditions and the calculation result obtained by extracting the noise.

  FIG. 1 is a block diagram mainly showing an electrical configuration of the imaging apparatus 1 according to the present embodiment. The imaging device 1 includes an imaging unit 2, an internal memory 3, an image processing unit 4, an external memory 5, a display unit 6, an input unit 7, a system control unit 8, a bus 9, and a temperature detection unit 10. The imaging unit 2 includes a photographing lens 2a, a diaphragm 2b, a shutter 2c, and an imaging element 2d. Note that the lens barrel including the imaging optical system 2a and the aperture stop 2b in the imaging unit 2 may be configured integrally with the imaging apparatus main body, or may be configured as an interchangeable lens barrel.

  The imaging optical system 2a includes a plurality of optical lenses such as a zoom lens and a focus lens, and forms an optical image of a subject on the imaging surface of the imaging element 2d. It has a drive mechanism that moves the zoom lens and the focus lens along the optical axis direction, and performs focus adjustment by controlling the position of the focus lens based on the output of the system control unit 8 described later. The photographing optical system 2a functions as a photographing optical system that forms the subject image on the image sensor.

  The diaphragm 2b is disposed in the optical path of the photographic optical system 2a and is configured to be openable and closable. The aperture diameter changes in order to adjust the amount of photographic light flux incident on the image sensor 7. The aperture diameter of the diaphragm 2b is controlled by the system control unit 8. That is, the system control unit 8 calculates subject brightness based on the image data from the image sensor 2d, and automatically controls the aperture value so as to achieve proper exposure. Alternatively, the aperture value is manually set by the photographer.

  The shutter 2c is disposed in the optical path of the photographic optical system 2a, and is configured to switch the imaging surface of the imaging element 2d to a light shielding state or an exposure state. The exposure time of the image sensor 2d is adjusted by the shutter 2c. The opening time (exposure time, shutter speed) of the shutter 2c is controlled by the system control unit 8. That is, the system control unit 8 calculates the subject brightness based on the image data from the image sensor 2d, and automatically controls the exposure time so as to achieve proper exposure. Alternatively, the exposure time is manually set by the photographer. Further, as will be described later, the shutter 2c is controlled so as to be in a light-shielding state during light-shielding photographing. The shutter 2c functions as an exposure blocking control unit that blocks the imaging of the subject image on the image sensor.

  The imaging element 2d has an imaging surface on which an imaging light beam from the subject condensed by the imaging optical system 2a is imaged. A plurality of pixels are two-dimensionally arranged on the imaging surface of the imaging device 2d, and a color filter is provided on the incident surface side. Each pixel of the image pickup device 2d converts a light image (subject image) corresponding to a photographing light beam formed on the image pickup surface into an analog image signal corresponding to the light amount. The analog image signal is converted into a digital signal (a digital signal corresponding to the analog image is simply referred to as “image data”) by an A / D converter (not shown), and is output to the bus 9. The image sensor 2d functions as an image sensor that photoelectrically converts a subject image to generate image data.

  The bus 9 is a bus line for transmitting and receiving various data and control signals. An internal memory 3, an image processing unit 4, an external memory 5, a display unit 6, an input unit 7, a system control unit 8, and a temperature detection unit 10 are connected to the bus line.

  The internal memory 3 is configured by a non-volatile memory such as a flash memory or a volatile memory such as a DRAM (Dynamic Random Access Memory). The internal memory 3 temporarily stores various setting information necessary for camera operation and an intermediate image during image processing.

  The image processing unit 4 performs desired image processing on the image data read from the image sensor 2d. The image processing unit 4 includes a noise calculation processing unit 4a, a noise correction processing unit 4b, and an image development processing unit 4c.

  The noise calculation processing unit 4a calculates the amount of noise using the image data read from the image sensor 2d or the light-shielded image data stored in the internal memory 3. The noise calculation processing unit 4a functions as a noise extraction unit that performs noise extraction based on the light-shielded image data generated by the image sensor after the light-shielding exposure. The noise correction processing unit 4b calculates a correction amount based on the noise amount calculated above and corrects the image data that has been captured with normal exposure. The noise correction unit 4b functions as a noise correction unit that performs noise correction on the normal exposure image data generated by the image sensor after performing normal exposure based on the noise extraction result by the noise extraction unit. The image development processing unit 4c performs development processing for performing various image processing such as exposure correction, WB gain correction, contour enhancement, and false color correction on the image data subjected to noise correction.

  In the present embodiment, the image processing unit 4 exemplifies a dedicated image processing processor configured by an arithmetic circuit that performs image processing calculation corresponding to each part. However, the present invention is not limited to this configuration, and for example, a configuration may be employed in which image processing calculation is developed on a signal processor such as a DSP based on an image processing program.

  The external memory 5 is an electrically rewritable nonvolatile memory such as an SD card or a CF card, and is detachable with respect to the imaging apparatus 1. The external memory 5 records the image data developed by the image development processing unit 4 c and can read the recorded image data from the external memory 5.

  The display unit 6 is an LCD arranged on the back surface of the imaging device 1 or an EVF (electronic viewfinder) built in the imaging device 1 and observable through an eyepiece, and is developed by the image development processing unit 4c. Display the image.

  The input unit 7 has operation members such as buttons and a touch panel for instructing photographing operations such as various operation mode settings and release based on the operation of the photographer, and detects the operation state of these operation members, The data is output to the system control unit 8 via the bus 9. When the release button is operated, an exposure operation is started.

  The system control unit 8 includes a CPU (Central Processing Unit) and its peripheral circuits, and comprehensively controls each unit in the imaging apparatus 1 according to a program stored in the internal memory 3. Specifically, the exposure amount is controlled at the time of shooting, the imaging timing is controlled, and the imaging controller has a function as an imaging control unit. In addition, A / D conversion control is also performed in outputting image data from the image sensor 2d.

  Further, the system control unit 8 performs focus drive control on the drive mechanism of the photographic optical system 2a so that the photographic optical system 2a reaches the in-focus position based on the image data. Further, the diaphragm drive mechanism is controlled so that the diaphragm 2b has a predetermined opening diameter. Further, drive control such as exposure time of the shutter 2c and light shielding operation is performed. The system control unit 8 controls normal exposure from the start of exposure to the end of exposure in a state where the subject image is formed on the image sensor by the photographing optical system. Also, it functions as an exposure control unit that controls light-blocking exposure from the start of exposure to the end of exposure in a state where the subject image formation is blocked by the exposure block control unit.

  The temperature detection unit 10 is disposed in the vicinity of the image sensor 2d, detects the temperature of the image sensor 2d, and outputs the detection result to the system control unit 8 via the bus 9. As will be described later, the system control unit 8 controls noise correction processing performed by the image processing unit 4 in accordance with the temperature detected by the temperature detection unit 10.

  Next, an outline of the operation of the imaging apparatus 1 will be described. The shooting operation in the imaging apparatus 1 is as follows. When receiving a main shooting instruction from the photographer via the input unit 7, the system control unit 8 performs an imaging control operation for performing timing control such as exposure start and signal readout on the image sensor 2d. Further, with respect to the stop 2b, control of the aperture of the stop so as to obtain a predetermined incident light flux, automatic exposure control (AE control) for calculating an exposure time in the shutter 2c so as to achieve proper exposure, On the other hand, autofocus control (AF control) is performed to move the focus lens of the photographing optical system 2a so that the focus is in focus.

  Furthermore, the system control unit 8 opens and closes the shutter 2c so that the calculated exposure time is reached. While the shutter 2c is open, an electrical signal obtained by photoelectrically converting the optical image is generated in the image sensor 2d. After the exposure time ends, an image signal read operation is performed on the image sensor 2d.

  Image data read from the image sensor 2 d and subjected to AD conversion is temporarily stored in the internal memory 3. The image data read from the internal memory 3 is subjected to desired image processing by the image processing unit 4, developed, and recorded in the external memory 5. The image data that has been subjected to the predetermined image processing by the image processing unit 4 is displayed on the display unit 6 after being resized to an image size necessary for display output on the display unit 6.

  Next, the operation in this embodiment will be described with reference to the flowcharts shown in FIGS. When the CPU in the system control unit 8 is executed according to a program stored in the internal memory 3, the operation according to this flowchart is realized.

  The flow shown in FIG. 2 will be described assuming that the stage of shooting preparation for subject shooting is an initial state. In the initial state, it is possible to set not only single-image shooting but also multiple-image combination shooting that combines a plurality of images such as HDR (High Dynamic Range Imaging) and super-resolution shooting. The setting of the multiple-image composite shooting mode is performed by a menu button or the like of the input unit 7.

  When the flow shown in FIG. 2 is started, exposure preparation is first performed (S1). When the photographer operates the release button of the input unit 7, preparation for exposure is performed. As the exposure preparation operation, the system control unit 8 performs AF control and AE control. As the AF control, focus information is acquired based on image data for live view display generated based on a subject image that enters the imaging surface of the image sensor 2d through the imaging optical system 2a, and based on the focus information. Then, drive control of the position of the focus lens to the in-focus position is performed.

  As AE control, subject luminance information is acquired based on the above-described image data for live view display, and based on the acquired luminance information, an aperture value set in the aperture mechanism of the aperture 2b, and a shutter 2c. Calculate the exposure time to be set. Then, drive control of the aperture stop position is performed so that the calculated aperture value is obtained. The exposure time by the shutter 2c can be set manually by the photographer via the input unit 7 in addition to being automatically set by AE control.

  When the AE control and the AF control in the exposure preparation are completed, exposure shooting is started (S3). Here, the system control unit 8 controls to reset the charge accumulated in the image sensor 2d once, and opens the shutter 2c. When the shutter 2c is opened, the subject image formed on the photographing optical system 2a is incident on the imaging surface of the imaging element 2d. Then, the image sensor 2d performs control to cancel the reset, and starts accumulating charges that are photoelectrically converted according to the luminance of the incident subject.

  Once exposure shooting is started, it is next determined whether or not the set shooting time has been reached (S5). Here, it is determined whether or not the exposure time calculated in step S1 has elapsed, or whether or not the end of exposure has been instructed by the photographer operating the release button or the like during long time exposure. If the result of this determination is No, the process returns to step S5 again, and this determination is repeated.

  If the result of determination in step S5 is that the set shooting time has been reached, exposure shooting is terminated (S7). Here, the system control unit 8 performs control so that the shutter 2c blocks light.

  When the exposure shooting is finished, exposure image data is acquired (S9). Here, exposure image data obtained by the current exposure is acquired from the image sensor 2d. The acquired exposure image data is stored in the internal memory 3. This stored image data is Bright_Image [n]. Bright_Image [n] means n-th exposure image data sequentially obtained by exposure, n = 1, 2,..., And n has an initial value of 1.

  Once the exposure image data has been acquired, it is next determined whether or not the exposure shooting has been completed (S11). In the initial state, an exposure shooting mode is set in advance, and it is determined whether or not shooting is finished according to the set exposure shooting mode. For example, in the case of multiple exposure modes such as HDR and super-resolution in which a plurality of images are acquired and the plurality of images are combined, the set number of shots is reached (the determination result in step S11 is No). Returning to step S3, exposure photographing is repeated.

  On the other hand, if the result of determination in step S11 is that exposure exposure has been completed, after step S13, image data is acquired in a light-shielded state, noise correction processing is performed using this image data, and then composition processing is performed. Perform image processing.

  First, light shielding preparation is performed (S13). Here, the system control unit 8 performs control so that the shutter 2c is shielded from light. In preparation for light-shielding photography, control is performed to reset the charge of the image sensor 2d once. As the photographing time in the light-shielded photographing, the lightest photographing condition is set to the longest exposure time among the photographing conditions (ISO sensitivity, exposure time) of the image data Bright_Image [n] acquired in step S9.

  Once the light shielding preparation is made, the light shielding photographing is started (S15). When the reset of the image sensor 2d is released, light-shielding shooting starts. Since the shutter 2c is in a light-shielding state, no subject light is incident on the image pickup device 2d, and charges generated due to defective pixels, shot noise, dark current, and the like are accumulated in this state. .

  When the shaded shooting is started, it is determined whether or not the set shooting time has been reached (S17). Here, in step S13, determination is made based on whether or not the set shooting time at the time of shading shooting has elapsed. If the result of this determination is that the shooting time during shaded shooting has not elapsed, the system waits for this shooting time to be reached.

  As a result of the determination in step S17, when the set shooting time is reached, the image is read from the image sensor and stored in the internal memory (S19). Here, the shaded image data obtained by the current exposure is acquired from the image sensor 2d, and the obtained shaded image data is stored in the internal memory 3. This stored image data is image data Dark_Image [1]. Dark_Image [1] means image data obtained by shading.

  Once the shaded image data is read out and stored in the internal memory, next, noise detection processing is performed (S21). Here, in step S19, the light-shielded image data stored in the internal memory 3 is read, and the noise detection processing unit 4a performs noise detection processing using the light-shielded image data. In the noise detection process, a block offset that appears as shading is detected due to a difference in pixel output during light shielding according to the position of the defective pixel that appears as noise and the area of the imaging surface. Detailed operation of this noise detection processing will be described later with reference to FIG.

  Once the noise detection process is performed, next, a noise correction process is performed (S33). Here, the exposure image data Bright_Image [n] is read from the internal memory 3, and the noise correction processing unit 4b performs noise correction processing using the result of the noise detection processing. The image data that has undergone this correction processing is referred to as image data Raw_Image [n]. This Raw_Image [n] means image data obtained by performing noise correction processing on n-th exposure image data Bright_Image [n] sequentially obtained by exposure. Note that n = 1, 2, 3,..., And the initial value is 1. Detailed operation of this noise correction processing will be described later with reference to FIG.

  When the noise correction process is performed, an image development process is performed (S35). Here, the image development processing unit 4c performs development processing on the image data subjected to the noise correction processing in step S33. In addition, when the HDR mode or the super-resolution mode is set, image synthesis is performed according to the mode in which a plurality of pieces of image data are set.

  Once image development processing has been carried out, next, image data is stored in an external memory (S37). Here, the image data processed in step S35 is recorded in the external memory 5. Subsequently, image data is displayed (S39). Here, the image data is displayed on the display unit 6 for a predetermined time based on the image data recorded in the external memory 5. When the HDR mode or the super-resolution mode is set, an image synthesized according to these modes is displayed. When display is performed based on the image data in step S39, the flow for photographing shown in FIG.

  As described above, in the shooting flow in the present embodiment, a plurality of exposure image data is acquired (S3 to S11), and when a predetermined number of images are acquired, the light-shielded image data is acquired by performing the light-shielding shooting ( S13 to S19), noise correction is performed using the light-shielded image data (S21, S33).

  Next, the noise detection process in step S21 is demonstrated using the flowchart shown in FIG.

  When the flow of the noise detection process is entered, first, a light shielding image filter process is performed (S41). Here, the shaded image data Dark_Image [1] stored in the internal memory 3 in step S19 is read out, and the filter process is performed. As the filter processing, for example, as shown in FIG. 5, the output for each pixel such as a defect is smoothed. In this example, smoothing is performed by median processing. Specifically, the pixel values at the star positions in the figure are arranged in descending order of the pixel output levels at A to D and the star positions in the figure, and the pixel output levels at the star positions are The pixel output level is replaced with an intermediate value (the third highest value in this example). Here, the pixel array is a Bayer pattern, and the pixels A to D have the same color pattern as the star-marked pixels. All the pixels are replaced with intermediate values while sequentially shifting the stars. The image data that has been subjected to the filter processing is defined as Filter_Image [1].

  In the example shown in FIG. 5, median processing is performed with five pixels. However, the median processing is not limited to five pixels, but may be performed with other numbers of pixels. Further, the smoothing process is not limited to the median process, and other smoothing processes may be used.

  If shading image filter processing is performed in step S41, next, difference extraction processing is performed (S43). Here, difference processing, that is, Dark_Image [1] −Filter_Image [1], is calculated for the shading image data Dark_Image [1] stored in step S19 and the shading filter processing image data Filter_Image [1] obtained in step S41. To do. The image data calculated by the difference process is set as image data Sub_Image [1].

  FIG. 6 illustrates the difference process in step S43. FIG. 6A shows the pixel output of the shaded image data Dark_Image [1] acquired in the shaded shooting in steps 15 and 17 and stored in the internal memory 3 in step S19. The image sensor 2d has defective pixels, and in the example shown in FIG. 6, N1 to N4 are pixel outputs of defective pixels.

  FIG. 6B shows a pixel output of the shading filter processed image data Filter_Image [1] that has been subjected to the shading image filter processing in step S41. As a result of smoothing by the filter processing, the pixel outputs (N1 to N4) of the defective pixels as shown in FIG.

  FIG. 6C shows a pixel output of the image data Sub_Image [1] that has been subjected to the difference output process in step S43. By calculating Dark_Image [1] −Filter_Image [1], only the pixel output (N1 to N4) of the defective pixel has a value, and the output of the pixel that is not the defective pixel becomes 0 or a value close to 0.

  If difference extraction processing is performed in step S43, then defective pixel determination processing is performed (S45). Here, it is determined whether or not the threshold value is greater than or equal to the threshold set for each pixel of the difference processed image data Sub_Image [1] obtained in step S43. This threshold is determined based on each shooting condition (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], shooting temperature Bright_Tem [n]) when acquiring the image data Bright_Image [n] in step S9. The threshold value is expressed as Th_defect [n]. The threshold value may be stored in advance in the internal memory 3 as a table indicating the threshold value for each shooting condition, or may be obtained by an arithmetic expression.

When determining the threshold value by an arithmetic expression, for example, when the exposure time for normal exposure (light image) is 1 minute or more on the condition of the exposure time for normal shooting (light image), the following formula Calculate from (1).
Th_defect [n] = (Bright_ISO [n] / Bright_ISO_min) × ((Bright_Tem [n] −A _{Th_defect} ) / B _{Th_defect} ) × Bright_TV [n] × C _{Th_defect} (1)

However, Bright_Tem [n] in the case of less than A _{Th_defect,} clipped to A _{Th_defect.} A _{Th_defect} , B _{Th_defect} , and C _{Th_defect} are coefficients, and Bright_ISO_min is the minimum value in Bright_ISO [n]. For example, these coefficients (A _{Th_defect} , B _{Th_defect} , C _{Th_defect} ) are the temperature change (A _{Th_defect} , B _{Th_defect} ) of the defective pixel output characteristic determined depending on the circuit structure of the image sensor, and the time change or constant (C _{Th_defect} ) Constant.

Further, when the exposure time of normal exposure photography (light image) is less than 1 minute, it is obtained from the following formula (2).
Th_defect [n] = D _{Th_defect} (2)
Here, D _{Th_defect} is a coefficient. This coefficient (D _{Th_defect} ) is a constant constant (D _{Th_defect} ) when the bright exposure time to the image sensor is short, since the characteristic change (temperature, time) of the defective pixel output of the image sensor is small. Th_defect [n] means a defect determination threshold for the exposure image data Bright_Image [n], n = 1, 2,..., And the initial value is 1.

  If defective pixel determination processing is performed in step S45, next, if the pixel value of the difference processed image data Sub_Image [1] is greater than or equal to the threshold as a result of the determination processing, the pixel address is set to defect_add [x] [y] [ n] is stored in the internal memory 3. Here, defect_add [x] [y] [n] means defect address XY coordinate data for the exposure image data Bright_Image [n], and n, x, y = 1, 2,... Is 1.

  If the address of the defective pixel is stored in the internal memory in step S47, next, block average processing of the shading filter processed image is performed (S49). Here, an average value Block_AVE [m] for each block is obtained for the filtered image data Filter_Image [1] calculated in step S41. The average value for each block is, for example, as shown in FIG. 7, an optical black pixel that is a pixel that is physically shielded in the effective pixel blocks 1 to 9 and the image sensor 2. The average value Block_AVE [m] for each block is calculated by dividing the total of 10 blocks of OB, which is the pixel area corresponding to (OB pixel), adding the pixel value for each block, and dividing by the number of pixels. Block_AVE [m] means m-th average value data obtained sequentially for each block, m = 1, 2,..., And the initial value is 1.

  In the example shown in FIG. 7, the division number of the block average process is 1 for the OB block and 9 for the effective pixel block. However, the division number and the division method are not limited to this example. Further, the shading filter processed image data Filter_Image [1] may be left without performing the averaging process.

  If block average processing of the shading filter processed image is performed in step S49, then block offset determination processing is performed (S51). Here, the minimum value is subtracted from the maximum value of the average value Block_AVE [m] calculated in step S49, and it is determined whether the subtraction value is equal to or greater than the set threshold.

  The threshold at the time of the determination process performed in step S51 is the shooting conditions (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], shooting temperature of the exposure image data Bright_Image [n] acquired in step S9 (see FIG. 2). Bright_Tem [n]) to determine the value. The threshold value may be stored in advance in the internal memory 3 for each photographing condition, or may be obtained by an arithmetic expression.

When the threshold value is determined by an arithmetic expression, for example, when the exposure time for normal exposure (light image) is 1 minute or more on the condition of the exposure time for normal shooting (light image), the following formula ( 3) Obtain from
Th_block [n] = (Bright_ISO [n] / Bright_ISO_min) × ((Bright_Tem [n] −A _{Th_block [n]} ) / B _{Th_block [n]} ) × Bright_TV [n] × C _{Th_block [n]} (3 )

However, Bright_Tem [n] in the case of less than A _{Th_block [n]} is clipped to A _{Th_block [n].} A _{Th_block [n]} , B _{Th_block [n]} , and C _{Th_block [n]} are coefficients. Also, (Bright_ISO_min is the minimum value in Bright_ISO [n]. For example, these coefficients (A _{Th_block [n]} , B _{Th_block [n]} , C _{Th_block [n]} ) depend on the circuit structure of the image sensor. _{These are constants} of temperature change (A _{Th_block [n]} , B _{Th_block [n]} ) of the shading output characteristic determined by the above, and time change or constant (C _{Th_block [n]} ).

Further, when the exposure time of normal exposure photography (light image) is less than 1 minute, the following equation (4) is used.
Th_block [n] = D _{Th_block [n]} (4)
Here, D _{Th_block [n]} is a coefficient. For example, this coefficient (D) has a constant constant (D _{Th_block [n]} ) because the characteristic change (temperature, time) of the shading output of the image sensor is small when the bright exposure time to the image sensor is short. Yes. Th_block [n] means a block offset determination threshold for the exposure image data Bright_Image [n], n = 1, 2,..., And the initial value is 1.

  Once the block offset determination process is performed in step S51, the result of the determination process for each block is stored in the internal memory (S53). Here, if the subtraction value obtained by subtracting the minimum value from the maximum value of the average value Block_AVE [m] is equal to or greater than the set threshold as a result of the determination process, Block_AVE [m] [n] is stored in the internal memory 3. . 0 if less than the threshold. Block_AVE [m] [n] means m-th average value data sequentially obtained for each block with respect to the exposure image data Bright_Image [n]. m, n = 1, 2,..., and the initial value is 1.

  When the determination processing result of each block is stored in the internal memory in step S53, the flow of the noise detection processing is terminated and the processing returns to the original flow.

  Next, the noise correction processing in step S33 will be described using the flowchart shown in FIG.

  In the noise correction process flow, first, defect correction is performed (S61). Here, defect correction is performed on the pixel value located at the pixel address defect_add [x] [y] [n] stored in step S47 for the exposure image data Bright_Image [n] acquired in step S9. The image data after defect correction is defined as Bright_Image_defect [n].

  Defect correction will be described using the example shown in FIG. In the figure, the position with a star is the address of the defective pixel in the internal memory in step S47. In this case, the pixel values of the four pixels A to D in the figure are added and divided by the number of pixels (here, 4) (that is, (A + B + C + D) ÷ 4 is calculated), and this divided value is represented by an asterisk. Replace with the pixel value at the position of. If the positions A to D are recorded in the internal memory as the positions of defective pixels, the average value is calculated by excluding the pixel values of the defective pixels. For example, when there is a defective pixel at the position A, the addition average value is calculated by (B + C + D) ÷ 3.

  In the example of the defect correction processing shown in FIG. 8, the average value of four pixels around the defective pixel is calculated. However, the number of pixels and the method for calculating the average value are limited to this example. Not. Further, similar pixels (or blocks) may be detected from the surroundings instead of the average and may be interpolated. In the example shown in FIG. 8, the pixel array is a Bayer pattern, and the pixels A, B, C, and D have the same color pattern as the star-marked pixels.

  By performing defect correction, it is possible to correct noise caused by defective pixels that differ for each image sensor.

  If defect correction is performed in step S61, next, block offset correction is performed (S63). Block offset correction is performed on the image data Bright_Image_defect [n] corrected in step S61 using Block_AVE [m] [n] stored in step S53. The image data subjected to the block offset correction is set as image data Raw_Image [n]. Raw_Image [n] means image data obtained by performing noise correction processing on n-th exposure image data Bright_Image [n] sequentially obtained by exposure, and n = 1, 2,... The initial value is 1.

The block offset correction will be described with reference to FIG. FIG. 9A shows the image data Bright_Image_defect [n] corrected in the defect in step S61, and is a set of pixel values corresponding to each address position. FIG. 9B shows a set of values obtained by multiplying the block αAVE [m] [n] stored in step S53 by the coefficient α [n] Block_AVE and adding the coefficient β [n] Block_AVE. Here, a value is determined for each OB block and effective pixel block. These coefficients (α [n] _{Block_AVE} , β [n] _{Block_AVE} ) are constants determined by the temperature change and time change of the shading output characteristics determined depending on the circuit structure of the image sensor.

  The block offset correction is performed by subtracting a value determined for each block shown in FIG. 9B for each pixel of the image data Bright_Image_defect [n] subjected to the defect correction shown in FIG. A set of subtraction values performed for each pixel becomes image data Raw_Image [n] from the image data subjected to the block offset correction shown in FIG.

  By performing the block offset correction, it is possible to obtain image data from which the influence of shading included in the image data output from the image sensor 2d is removed.

  As described above, in this embodiment, normal exposure photography (S3 to S11 in FIG. 2) and light-shielding photography with a predetermined exposure time (S13 to S19 in FIG. 2) are obtained and acquired with a predetermined exposure time. Noise extraction is performed from the light-shielded image (S21 in FIG. 2).

  Here, in the present embodiment, the predetermined exposure time described above is the first exposure time and the second exposure time when the exposure time for normal exposure photography is the first exposure time and the exposure time for light-shielding photography is the second exposure time. The exposure time is the same. However, the present invention is not limited to this, and the second exposure time may be varied according to the first exposure time.

The above noise extraction is
(1) In order to extract a defective pixel, an output level for each pixel or a difference from surrounding pixels is calculated (see S43 in FIG. 3 and FIG. 6).
(2) Filter processing such as median or averaging is performed for each pixel block, and block components (filtered values for each pixel block), shading components (filtered values for each row or column), OB step components (effective pixels) (The difference between the filtered value of the block and the filtered value of the OB block) (see S49 in FIG. 3, FIG. 7),
By doing that.

  Further, in the present embodiment, noise correction is performed on the normal exposure photographed image using the above-described calculation result obtained by noise extraction (see S33 in FIG. 2 and FIG. 4).

Noise correction
(3) In the case of a defective pixel, interpolation is performed from surrounding pixels (blocks). Specifically,
Correction amount X = Σsurrounding pixel (block) calculation value / number of surrounding pixels (block) (5)
(See FIG. 8),
(4) Addition / subtraction of block component, shading component, and OB step component is performed. Specifically,
Correction amount X = target pixel−the above component value (6)
(See FIG. 9).

  In the present embodiment, the correction amount is changed in accordance with the normal exposure and light-shielding shooting conditions and the noise extraction result.

The correction amount is changed by changing the correction coefficients (α, β) according to ISO sensitivity, exposure time, and temperature in the case of shooting conditions of normal exposure or light shielding. The correction amount Y in this case can be expressed by the following equation (7).
Correction amount Y = αX + β (7)
Here, X is a correction amount before the change.

In addition, when multiple images (such as HDR and super-resolution) are normally exposed and combined, the respective correction coefficients (α1, β1) are selected in accordance with the ISO sensitivity, exposure time, and temperature at the time of shooting. , Α2, β2,... Αn, βn). The correction amounts Y1, Y2,... Yn in this case can be expressed by the following formulas (8), (9), and (10).
Correction amount Y1 = α1X + β1 (8)
Correction amount Y2 = α2X + β2 (9)
...
Correction amount Yn = αnX + βn (10)
Note that X is a correction amount before the change, and n indicates the number of images taken when a plurality of normal exposures are performed.

In changing the correction amount according to the noise extraction result, the correction coefficients (α, β) of the defective pixel, the block component, the shading component, and the OB step component are changed according to the respective threshold values.
Correction amount Y = αX + β (11)
Here, X is a correction amount before the change.

  Next, a second embodiment of the present invention will be described with reference to FIGS. Also in the present embodiment, as in the first embodiment, normal exposure shooting and light-shielding shooting are performed, noise is extracted from the light-shielded image, and the normal exposure photographed image is calculated from the normal exposure and light-shielding shooting conditions and the calculation result of noise extraction. Perform noise correction. In the first embodiment, the light-shielding shooting condition for the light-shielding shooting is the same as the shooting condition for the normal exposure shooting. In the second embodiment, the light-shielding shooting condition is calculated when the light-shielding shooting is performed. Shoot under different shooting conditions from normal exposure shooting.

  The configuration of the present embodiment is the same as the configuration of the block diagram shown in FIG. 1 except that a light-shielding shooting condition calculation processing unit 11 is added and the other configurations are the same.

  The light-shielding shooting condition calculation processing unit 11 connected to the bus 9 inputs the normal exposure shooting conditions (ISO sensitivity, exposure time) calculated by the system control unit 8, the temperature detected by the temperature detection unit 10, and the photographer. The light-shielding shooting condition is calculated based on the setting value set via the unit 7 or the setting value stored in the internal memory 3 in advance (see S12 in FIG. 11 and FIG. 12). To instruct. The system control unit 8 controls light-shielded photographing according to the light-shielded photographing conditions (see S15 and S17 in FIG. 11).

  Next, the operation in the present embodiment will be described with reference to the flowcharts shown in FIGS. When the CPU in the system control unit 8 is executed according to a program stored in the internal memory 3, the operation according to this flowchart is realized.

  Compared with the flowchart shown in FIG. 2 according to the first embodiment, the operation in the present embodiment adds the light-shielding shooting condition calculation process in step S12, and performs light-shielding in steps S13 to S17 according to the light-shielding shooting conditions calculated here. The difference is that the shooting is controlled. Since the processing in the other steps is the same, the differences will be mainly described.

  The initial state in the present embodiment is the same as that in the first embodiment, and the stage where the subject is prepared for shooting is set as the initial state of the start state. In the process of step S1, as in the first embodiment, the photographer performs a release operation and starts the exposure start process.

  The processes from step S1 to S11 are the same as in the first embodiment. When the exposure shooting is completed as a result of the determination in step S11, a shading shooting condition calculation process is performed (S12). Here, each photographing condition (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], photographing temperature Bright_Tem [n]) of the image data Bright_Image [n] acquired in step S9 and the photographer through the input unit 7 Based on the setting value set in advance or the setting value stored in the internal memory 3 in advance, the light-shielding shooting conditions (ISO sensitivity, exposure time) are obtained and set as Dark_ISO and Dark_TV.

  When the light-shielding shooting condition is calculated in step S12, the light-shielding shooting is performed in steps S13 to S17 using the calculated light-shielding shooting condition (ISO sensitivity, exposure time). When the shaded shooting is finished, steps S19 to S39 are executed, and this flow is finished.

  Next, the shading imaging condition calculation process in step S12 will be described using the flowchart shown in FIG.

  When entering the flow of the light-shielding shooting condition calculation process, first, an input setting condition determination process is performed (S71). Here, the dark exposure time coefficient Dark_TV_coe is determined based on a setting value set by the photographer via the input unit 7 or a setting value stored in the internal memory 3 in advance. For the shading shooting shooting conditions, obtain shooting conditions such as ISO sensitivity and exposure time during shading shooting according to the exposure time during normal shooting and the temperature during shooting, and multiply the obtained shooting conditions by the dark exposure time coefficient Dark_TV_coe. To calculate.

  FIG. 13 shows an example of a dark condition setting menu screen when the photographer makes settings via the input unit 7. The photographer calls a dark condition setting menu screen as shown in FIG. 13 before starting photographing, and selects “ON” by a touch operation or an input button operation or the like to reduce long-time noise. If “1/2” is further selected by a touch operation or an input button operation, the dark exposure time coefficient Dark_TV_coe becomes 1/2 (0.5). By this operation, the photographer can shorten the exposure time in the dark.

  If the input setting condition determination process is performed in step S71, then the exposure photographing condition determination process is performed (S73). Here, the respective exposure conditions (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], shooting temperature Bright_Tem [n]) of the image data Bright_Image [n] acquired in step S9 and the dark exposure determined in step S71. The dark ISO sensitivity coefficient Dark_ISO_coe is determined based on the time coefficient Dark_TV_coe.

For Dark_ISO_coe, a table may be stored in the internal memory 3 in advance, or may be obtained by an arithmetic expression at the time of shooting. For example, the following equation (12) is used as this arithmetic expression.
Dark_ISO_coe = (1 / Dark_TV_coe) x ((Bright_Tem [n]-A _{Dark_ISO_coe} ) / B _{Dark_ISO_coe} ) x C _{Dark_ISO_coe} (12)
Incidentally, Bright_Tem [n] is the time less than A _{Dark_ISO_coe} clipped to A _{Dark_ISO_coe.}

In the above equation (12), A _{Dark_ISO_coe} , B _{Dark_ISO_coe} , and C _{Dark_ISO_coe} are coefficients. These coefficients (A _{Dark_ISO_coe} , B _{Dark_ISO_coe} , C _{Dark_ISO_coe} ) are, for example, temperature variations (A, B) of defective pixel output characteristics and shading output characteristics determined depending on the circuit structure of the image sensor, and constant (C _{Dark_ISO_coe} ) Is a constant. Bright_ISO [n] is, for example, the ISO sensitivity under the condition that the exposure time is the longest (shortest) among the shooting conditions of Bright_Image [n]. Bright_Tem [n] is, for example, a shooting temperature under the Bright_ISO [n] shooting condition.

  If the ISO sensitivity coefficient at the time of light-shielded photographing is obtained by the exposure photographing condition determination processing in step S73, next, light-shielded photographing condition calculation is performed (S75). Here, the dark exposure time coefficient Dark_TV_coe determined in step S71 from among the shooting conditions (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n]) of the image data Bright_Image [n] acquired in step S9, From the dark ISO sensitivity coefficient Dark_ISO_coe determined in step S73, the light-shielding shooting conditions (ISO sensitivity, exposure time) are obtained, and these light-shielding shooting conditions are set to Dark_ISO and Dark_TV.

The shading imaging conditions Dark_ISO and Dark_TV may be stored in advance in a table in the internal memory 3 or may be obtained by an arithmetic expression at the time of shooting. As the arithmetic expression, for example, the following expressions (13) and (14) are used.
Dark_ISO = Dark_ISO_coe × Bright_ISO [n] (13)
Dark_TV = Dark_TV_coe × Bright_TV [n] (14)
Bright_ISO [n] is the ISO sensitivity under the condition that the exposure time is the longest (shortest) among the shooting conditions of Bright_Image [n]. Bright_TV [n] is the condition with the longest (shortest) exposure time among the shooting conditions of Bright_Image [n].

  When the light-shielding shooting condition calculation is performed, the flow of the light-shielding shooting condition calculation is ended and the process returns to the original flow. When returning to the original flow, as described above, the system control unit 8 controls the light-shielding photographing using the light-shielding photographing conditions (ISO sensitivity, exposure time) calculated in this flow.

  As described above, also in this embodiment, the normal exposure shooting (S3 to S11 in FIG. 11) and the light-shielded shooting with the predetermined exposure time (S12 to S19 in FIG. 11) are performed, and the light-shielded image with the predetermined exposure time. Noise extraction is performed (S21 in FIG. 11).

  Here, in the first embodiment, the above-described predetermined exposure time is the first exposure time when the exposure time for normal exposure photography is the first exposure time and the exposure time for light-shielding photography is the second exposure time. Second, the exposure time was the same. On the other hand, in the present embodiment, the second exposure time is not dependent on the first exposure time (see FIG. 12). That is, in the present embodiment, the exposure time for the light-shielding shooting is shorter than that for the normal exposure shooting. For this reason, it is possible to shorten the photographing time including the light-shielding photographing, and it is possible to suppress an increase in the photographing time accompanying the noise removal processing.

  In the present embodiment, the shooting condition setting (ISO sensitivity, exposure time) for shaded shooting is determined according to the shooting conditions for normal exposure or the conditions set by an external operation. That is, shooting condition settings (ISO sensitivity, exposure time) for light-shielded shooting are determined based on shooting conditions (ISO sensitivity, exposure time, temperature) for normal exposure shooting. Also, shooting condition settings (ISO sensitivity, exposure time) for light-shielded shooting are determined by external operation (camera settings) (see S71 in FIG. 12, FIG. 13).

  Note that the shooting condition settings (ISO sensitivity, exposure time) for shaded shooting are not limited to the shooting conditions for normal exposure shooting and settings by external operation, for example, multiple exposures such as multiple exposure (HDR and super-resolution) are used for normal exposure. In some cases, the shooting condition setting (ISO sensitivity, exposure time) for each light-shielded shooting corresponding to the number of shots and the number of shots may be determined.

  Next, a third embodiment of the present invention will be described with reference to FIGS. Also in the present embodiment, normal exposure shooting and light-shielding shooting are performed, noise is extracted from the light-shielded image, and noise correction of the normal exposure photographed image is performed from the normal exposure and light-shielding shooting conditions and the calculation result obtained by noise extraction. In the present embodiment, in addition to the processing in the first and second embodiments, additional light-shielding photographing condition calculation is performed after light-shielding photographing, and it is determined whether or not to perform light-shielding photographing additionally.

  In the configuration of the present embodiment, an additional light shielding photographing determination processing unit 12 is added to the block diagram shown in FIG.

  The additional light shielding photographing determination processing unit 12 connected to the bus 9 determines whether or not to perform additional light shielding photographing based on the result calculated by the noise calculation processing unit 4a. If necessary, the light shielding photographing condition different from the previous time is again determined. The system control unit 8 is instructed to take a shaded image. The additional light-shielding shooting determination processing unit 12 functions as an additional light-shielding shooting determination unit that determines whether or not to perform additional light-shielding shooting based on the noise extraction result by the noise extraction unit.

  That is, in the first and second embodiments, when exposure shooting is completed (S11 Yes in FIGS. 2 and 11), light-shielding shooting is performed once (S13 to S19 in FIG. 2, S12 to S19 in FIG. 11), and thereafter Thus, noise detection and noise correction processing were performed (S2 and S21 and S33 in FIG. 11). In these embodiments, the noise correction processing may not be performed properly under the light-shielded shooting conditions of the light-shielded shooting. Therefore, in the present embodiment, the additional light-shielding photographing determination processing unit 12 changes the light-shielding photographing conditions after the light-shielding photographing, and determines whether or not to perform additional light-shielding photographing again (FIG. 15 S23, see FIG. 16).

  Next, the operation in the present embodiment will be described using the flowcharts shown in FIGS. 15 and 16. When the CPU in the system control unit 8 is executed according to a program stored in the internal memory 3, the operation according to this flowchart is realized.

  Compared with the flowchart shown in FIG. 11 according to the second embodiment, the operation in this embodiment is different in that steps S23 to S29 are added and the noise correction process in step S33 is changed to the noise correction process in step S33a. . Since the processing in the other steps is the same, the differences will be mainly described.

  The initial state in the present embodiment is the same as in the first and second embodiments, and the stage where the subject is prepared for shooting is set as the initial state of the start state. In the process of step S1, as in the first embodiment, the photographer performs a release operation and starts the exposure start process.

  The processing from step S1 to S21 is the same as in the second embodiment. If it is determined in step S11 that the exposure shooting has been completed, the first light-shielding shooting is performed in steps S12 to S19. In the noise detection process in step S21, the address of the defective pixel is stored in the internal memory (see S41 to S47 in FIG. 3), the block offset determination process for removing the influence of shading is performed, and the determination result is stored in the internal memory. (See S41 to S53 in FIG. 3).

  If noise detection processing is performed in step S21, then additional light-shielding shooting condition determination processing is performed (S23). Here, using the result of the noise detection process in step S21, it is determined whether or not to perform additional light-shielding shooting using the number of defective pixels and the number of block offsets. The shooting conditions are calculated.

  Detailed operation of this additional light-shielding shooting condition determination process will be described with reference to FIG. If the flow of the additional light-shielding shooting condition determination process is entered, first, the defect number determination process is performed (S81). Here, the threshold determination is performed using the number of defect addresses defect_add [x] [y] [n] stored in step S21, and the additional dark shooting flag Add_Black_flg is set to “1” when additional shading shooting is necessary. To do.

As the threshold determination for comparing the number of defective addresses and the threshold value, for example, a determination is made according to whether the following equation (15) is satisfied.
Number of defects <(1 / Dark_TV_coe) × ((Bright_Tem [n] −A _{number of defects} ) / B _{number of defects} ) × C _{number of defects} (15)
When this equation (15) is established, Add_Black_flg = 1.

In Expression (15), when the value of Bright_Tem [n] is less than A, it is clipped to A. Further, the _{number of} A _{defects, the number} of B _{defects, and the number} of C _defects are coefficients. For example, these coefficients ( _{number of} A _{defects, number} of B _{defects, number of} C _defects ) are determined according to the circuit structure of the image sensor, and the pixel output characteristics ( _{number of} A _{defects, number} of B _defects ) and constant ( _{number of} C _defects ) ) Constant. The additional dark shooting flag Add_Black_flg is set to 1 when the number of defects is small.

  In step S81, if the defect count determination process is performed, next, a block offset determination process is performed (S83). Here, when the level of the block offset value Block_AVE [m] [n] stored in step S21 is not less than the threshold and not more than the threshold, the additional dark shooting flag Add_Black_flg is set to “1”.

Here, the threshold determination is performed according to whether, for example, the following equation (16) is satisfied.
Block_AVE [m] [n] <(1 / Dark_TV_coe) × ((Bright_Tem [n] −A _{Block_AVE [m] [n] 1} ) / B _{Block_AVE [m] [n] 1} ) × C _{Block_AVE [m] [n ] 1} ... (16)
When this equation (16) is established, Add_Black_flg = 1.

These coefficients (A _{Block_AVE [m] [n] 1} , B _{Block_AVE [m] [n] 1,} C _{Block_AVE [m] [n] 1} ) are shading outputs that are determined depending on the circuit structure of the image sensor. It is a characteristic (A _{Block_AVE [m] [n] 1} , B _{Block_AVE [m] [n] 1} ) and a constant (C _{Block_AVE [m] [n] 1} ).

In addition, the threshold determination is performed according to whether, for example, the following formula (17) is satisfied.
Block_AVE [m] [n]> (1 / Dark_TV_coe) x ((Bright_Tem [n] -A _{Block_AVE [m] [n] 2} ) / B _{Block_AVE [m] [n] 2} ) x C _{Block_AVE [m] [n ] 2} ... (17)
When this equation (17) is established, Add_Black_flg = 1.

For example, these coefficients (A _{Block_AVE [m] [n] 2} , B _{Block_AVE [m] [n] 2,} C _{Block_AVE [m] [n] 2} ) are determined depending on the circuit structure of the image sensor. The shading output characteristics (A _{Block_AVE [m] [n] 2} , B _{Block_AVE [m] [n] 2} ) are constants (C _{Block_AVE [m] [n] 2} ).

  When the above equation (16) holds, it is determined that additional dark imaging is necessary when shading is not detected. Further, when the equation (17) is established, a shading characteristic is detected. When this characteristic is non-linear, it is determined that additional dark imaging is necessary. Expression (16) and Expression (17) are determined based on the image data Bright_Image [n] imaging conditions (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], imaging temperature Bright_Tem [n]), It is determined based on the shading output characteristics determined depending on the circuit structure of the image sensor.

  If block offset discrimination processing is performed in step S83, then additional light-shielding shooting condition calculation processing is performed (S85). Here, if Add_Black_flg = 1 in step S81 or S83, additional dark imaging conditions are calculated. Based on the shading imaging conditions Dark_ISO and Dark_TV determined in S75 of FIG. 12, additional shading conditions Dark_Add_ISO and Dark_Add_TV are determined.

The additional shading conditions Dark_Add_ISO and Dark_Add_TV may be stored in advance in the internal memory 3 or may be obtained by an arithmetic expression. As an arithmetic expression, you may calculate by following (18) (19), for example.
Dark_Add_ISO = α _{Dark_Add_ISO} × Dark_ISO + β _{Dark_Add_ISO} (18)
Dark_Add_TV = α _{Dark_Add_TV} × Dark_TV + β _{Dark_Add_TV} (19)
In the above equations (18) and (19), α _{Dark_Add_ISO} , β _{Dark_Add_ISO,} α _{Dark_Add_TV} , and β _{Dark_Add_TV} are coefficients. For example, these coefficients (α _{Dark_Add_ISO} , β _{Dark_Add_ISO} , α _{Dark_Add_TV} , β _{Dark_Add_TV} ) are constants depending on defective pixel output characteristics and shading output characteristics determined depending on the circuit structure of the image sensor.

  Returning to FIG. 15, if the additional light-shielding photographing condition determination process is performed in step S23, it is next determined whether or not additional light-shielding photographing is performed (S24). In the additional light-shielding shooting condition determination process in step S23, when additional light-shielding shooting is required, “1” is set in Add_Black_flg (see S81 and S83 in FIG. 16 described later), and determination is made based on this flag. .

  If the result of determination in step S24 is to perform shading shooting with use, preparation for shading is made (S25). Here, similarly to the light shielding preparation in step S13, the system control unit 8 controls the shutter 2c to shield the light. In preparation for light-shielding photography, control is performed to reset the charge of the image sensor 2d once. The shooting conditions in the additional light-shielding shooting follow the additional light-shielding shooting conditions (ISO sensitivity, exposure time) calculated in the additional light-shielding shooting condition determination process in step S23 (see S85 in FIG. 16).

  When the light-shielding preparation is completed, the light-shielding photographing is started (S26). Here, similarly to step S15, in a state where light is shielded by the shutter 2c, the reset of the image sensor 2d is released, and light-shielded photographing is started.

  When shaded shooting is started, it is determined whether or not the next set shooting time has been reached (S27). Here, in step S25, the determination is made based on whether or not the set shooting time at the time of additional light-shielding shooting has elapsed. If the result of this determination is that the imaging time for additional light-shielding imaging has not elapsed, the system waits for this imaging time to be reached.

  As a result of the determination in step S27, when the set shooting time is reached, an image is read from the image sensor and stored in the internal memory (S28). Here, the light-shielded image data obtained by the additional light-shielded photographing is acquired from the image sensor 2 d, and the acquired light-shielded image data is stored in the internal memory 3. The stored image data is image data Dark_Image [2]. Dark_Image [2] means image data obtained by additional light shielding.

  Once the shaded image data is read and stored in the internal memory, a noise detection process is performed (S29). Here, in step S28, the shaded image data Dark_Image [2] stored in the internal memory 3 is read, and the noise detection processing unit 4a performs noise detection processing using the shaded image data. Similarly to step S21, the noise detection process detects the position of the defective pixel that appears as noise and the block offset that appears as shading (see FIG. 3).

  If noise detection processing is performed in step S29 or if the result of determination in step S24 is that additional light-shielding shooting is not performed, next, noise correction processing is performed (S33a). When the additional light-shielding shooting is not performed, when the additional light-shielding photographing is performed based on the noise detection process at step S21 performed based on the result of the light-shielding photographing performed at steps S12 to S19. Noise correction is performed based on the noise detection processing in FIG.

  Whether or not additional light-shielding shooting has been performed is determined based on the aforementioned flag Add_Black_flg. The case of Add_Black_flg = 0 is a case where no additional light-shielding photographing is performed. In this case, since it is the same as the noise correction processing in step S33 described above, detailed description is omitted.

  On the other hand, when Add_Black_flg = 1, this is a case where additional light-shielding shooting is performed. In this case, noise correction processing is performed in step S33a based on the image data at the time of additional light-shielding shooting stored in step S28 and the noise detection result performed in step S29.

  The noise correction process in step S33a is substantially the same as the noise correction process shown in FIG. 4, but differs in detail. When the noise correction process shown in FIG. 4 is entered, first, defect correction is performed (S61 in FIG. 4). Here, the defective pixel address defect_add [x] [y] [n] detected and stored in step S29 based on Dark_Image [2] stored in step S28 for the image data Bright_Image [n] acquired in step S9. Based on the above, defect correction is performed. The image data after defect correction is defined as Bright_Image_defect [n]. Since the defect correction method is the same as that of the first embodiment, the description thereof is omitted.

  If defect correction is performed in step S61, then block offset correction is performed (S63 in FIG. 4). Here, the block offset correction is performed using the Block_AVE [m] [n] stored in Step S29 for Bright_Image_defect [n] of the image data corrected in Step S61. The image data subjected to the block offset correction is set as image data Raw_Image [n]. Raw_Image [n] means image data obtained by performing noise correction processing on the nth exposure image data Bright_Image [n] sequentially obtained by exposure. n = 1, 2,..., and the initial value is 1. Since the block offset correction method is the same as that in the first embodiment, the description thereof is omitted.

Block offset correction is performed (S63). In the first and second embodiments, block offset correction is performed on the image data Bright_Image_defect [n] corrected in step S61 by using Block_AVE [m] [n] stored in step S53. In the present embodiment, the Block_AVE [m] [n] coefficients (α [n] _{Block_AVE} , β [n] _{Block_AVE} ) used in the block offset correction are calculated using the Block_AVE [m] [n] , Each shooting condition (ISO sensitivity Bright_ISO [n], exposure time Bright_TV [n], shooting temperature Bright_Tem [n]) of image data Bright_Image [n] at the time of exposure shooting, and image data Dark_Image [ Calculated based on each shooting condition (ISO sensitivity Dark_ISO, exposure time Dark_TV, shooting temperature Dark_Tem) and image data Dark_Image [2] shooting conditions (ISO sensitivity Dark_Add_ISO, exposure time Dark_Add_TV, shooting temperature Dark_Add_Tem) To do. In addition, the coefficients of block_AVE [m] [n] (α [n] _{Block_AVE} , β [n] _{Block_AVE} ) used in block offset correction are calculated by shading output characteristics determined depending on the circuit structure of the image sensor, You may utilize for the process of block offset correction | amendment in the process of S63.

  FIG. 17 shows an image diagram of block offset correction coefficient calculation. In FIG. 17, the horizontal axis indicates shooting conditions when performing light-shielded shooting, and comprehensively shows exposure time, ISO sensitivity, and shooting temperature. Specifically, as a photographing condition, a temperature related to the temperature specification of the image sensor 2d with respect to an exposure amount (product of exposure time (charge accumulation time) and ISO sensitivity) with respect to the imaging surface of the image sensor 2d when light is blocked A value obtained by weighting the coefficient is shown. For example, the horizontal axis represents the relationship where the exposure time is long, the ISO sensitivity is high, or the imaging temperature is high when the imaging conditions on the right side of the coordinates are met.

  The vertical axis represents the average value (Block_AVE [m] [n]) for each block. As can be seen from FIG. 17, first, Dark_Image [1] obtained at the first shading shooting (shooting conditions when acquiring Dark_Image [1]) and image data Dark_Image [2] at the second additional shading shooting (Shooting conditions when acquiring Dark_Image [2]), the average value for each block (Block_AVE [m] [n] of Dark_Image [1] is set to Block_AVE [m] [n] -1. Dark_Image [2 ] Is calculated as Block_AVE [m] [n] -2.

Next, an average value (Block_AVE [m] [n] −3) for each block in the image data at the time of light-shielded photographing under the same exposure condition as the exposure time is predicted from the change of these two values. Specifically, this prediction is performed by first calculating the average value for each block (Block_AVE [m] [n] -1, Blok_AVE [m] [n]) calculated from the image data at the first and second shaded shooting. -2) Calculate the coefficient α [n] _{Block_AVE} for the linear change.

Next, a coefficient β [n] _{Block_AVE} corresponding to the shading output characteristic is determined. Finally, Block_AVE [m] [n] -2 is multiplied by coefficient α [n] _{Block_AVE} , and coefficient β [n] _{Block_AVE} is added to calculate Block_AVE [m] [n] -3. As in FIG. 9, image data (Raw_Image [n]) with corrected block offset can be obtained by subtracting Block_AVE [m] [n] -3 from Bright_Image_defect [n].

The reason why the offset increase (β [n] _{Block_AVE} ) needs to be estimated is that the average value for each block in the image data at the time of the light-shielded photographing is not necessarily linear with respect to the exposure time, ISO sensitivity, and photographing temperature. This is because it does not always change.

  If noise correction processing is performed in step S33a, steps S35 to S39 are executed. When display is performed based on the image data in step S39, the flow for photographing shown in FIG. 15 is terminated.

  As described above, also in the present embodiment, the normal exposure shooting (S3 to S11 in FIG. 15) and the light-shielding shooting with the predetermined exposure time (S12 to S19 in FIG. 15) are performed, and the light-shielded image with the predetermined exposure time. Noise extraction is performed (S21 in FIG. 15).

  Also in the present embodiment, as in the second embodiment, the shooting condition setting (ISO sensitivity, exposure time) for light-shielded shooting is determined according to the shooting conditions for normal exposure or the conditions set by external operation. (S12 in FIG. 15). That is, shooting condition settings (ISO sensitivity, exposure time) for light-shielded shooting are determined based on shooting conditions (ISO sensitivity, exposure time, temperature) for normal exposure shooting.

  In the present embodiment, noise is extracted from the light-shielded image, and it is determined whether additional light-shielding shooting is performed based on the extraction result (S23 in FIG. 15). For this reason, when noise detection and noise correction are not sufficient by the image data acquired by shading after the exposure shooting is completed, additional shading shooting can be performed to obtain image data that has been subjected to noise correction with high accuracy. .

  As described above, in each embodiment of the present invention, normal exposure from the start of exposure to the end of exposure is controlled in a state where the subject image is formed on the image sensor by the photographing optical system (for example, S3 in FIG. 2). To S11), and in the state where the image formation of the subject image is blocked by the exposure blocking control unit, the light blocking exposure from the start of exposure to the end of exposure is controlled (for example, S13 to S19 in FIG. 2), and the light blocking exposure is performed. Noise extraction is performed based on the shading image data generated later by the image sensor (for example, S21 in FIG. 2), and the normal exposure image generated by the image sensor after performing normal exposure based on the noise extraction result. Noise correction is performed on the data (for example, see S9 in FIG. 2) (for example, S33 in FIG. 2).

  For this reason, it is not necessary to record defect information for each photographing condition in advance, the memory capacity does not increase, and the correction circuit associated with the noise removal process does not become large. Further, since only the pixel value of the detected defective pixel position is corrected, the resolution does not deteriorate due to the correction of random noise, and no erroneous correction is performed in the noise removal process.

  In each embodiment of the present invention, detection of a defective pixel position and offset correction are performed based on light-shielded image data. However, the present invention is not limited to this, and noise is extracted based on light-shielded image data. As long as noise correction is performed based on the above, the present invention is not limited to the embodiment. Further, the noise correction is not limited to that described in the embodiment of the present invention, and other correction methods may be used.

  In each embodiment of the present invention, the noise calculation processing unit 4a and the noise correction processing unit 4b are configured separately from the system control unit 8 as the image processing unit 4, but all or a part of each unit is software. Of course, it may be configured by software and executed by a CPU and a program in the system control unit 8. Similarly, the shading imaging condition calculation processing unit 11 and the additional shading imaging determination processing unit 12 may be entirely or partially configured by software and executed by the CPU and program in the system control unit 8. Of course it doesn't matter.

  In the present embodiment, the digital camera is used as the photographing device. However, the camera may be a digital single-lens reflex camera or a compact digital camera, and may be used for moving images such as video cameras and movie cameras. It may be a camera, and may be a camera built in a mobile phone, a smart phone, a personal digital assistant (PDA), a personal computer (PC), a tablet computer, a game machine, or the like. In any case, the present invention can be applied to any device that has an imaging unit and performs processing such as displaying and recording image data from the imaging unit.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Imaging part, 2a ... Imaging optical system, 2b ... Aperture, 2c ... Shutter, 2d ... Imaging element, 3 ... Internal memory, 4. Image processing unit, 4a ... Noise calculation processing unit, 4b ... Noise correction processing unit, 4c ... Image development processing unit, 5 ... External memory, 6 ... Display unit, 7 ... Input unit, 8 ... system control unit, 9 ... bus, 10 ... temperature detection unit, 11 ... shading imaging condition calculation processing unit, 12 ... additional shading imaging judgment processing unit

Claims (11)

An image sensor that has a plurality of pixels on the imaging surface and photoelectrically converts a subject image to generate image data;
A photographing optical system for forming the subject image on the image sensor;
An exposure blocking control unit that blocks image formation of the subject image on the image sensor;
A state in which normal exposure from the start of exposure to the end of exposure is controlled while the subject image is formed on the image sensor by the photographing optical system, and the image formation of the subject image is blocked by the exposure cutoff control unit And an exposure control unit for controlling light-shielding exposure from the start of exposure to the end of exposure,
A noise extraction unit that performs noise extraction based on the light-shielded image data generated by the image sensor after the light-shielding exposure;
Based on the noise extraction result by the noise extraction unit, a noise correction unit that performs noise correction on the normal exposure image data generated by the imaging device after performing the normal exposure;
An imaging apparatus comprising:   The noise extraction unit calculates an intermediate value between each value of the pixel of the shaded image data and pixels around the pixel, and obtains the position of the defective pixel based on a comparison between the intermediate value and a threshold value. The imaging apparatus according to claim 1.   The imaging apparatus according to claim 2, wherein the noise correction unit corrects the pixel value at the position of the defective pixel by interpolation calculation using pixel values of surrounding pixels. The imaging device further includes an optical black pixel region that is a pixel region composed of a plurality of pixels that are constantly shielded from light,
The noise extraction unit calculates a difference value between a maximum value and a minimum value for each block of a pixel output representative value in the optical black pixel region and a pixel of the light-shielded image data, and based on a comparison between the difference value and a threshold value. Then, depending on the distance from the center area of the imaging surface of the imaging element, the shading component, which is a change in pixel output in the light-shielded image data, or the pixel output of the light-shielded image data, and the pixels in the optical black pixel area The imaging apparatus according to claim 1, wherein an optical black step component that is a difference from a representative output value is obtained.   The imaging apparatus according to claim 4, wherein the noise correction unit performs noise correction by subtracting the shading component or the optical black level difference component from a pixel value of the normal exposure image data.   6. The imaging apparatus according to claim 3, wherein the correction amount in the noise correction unit is changed according to at least one of the conditions of ISO sensitivity, exposure time, and temperature during normal exposure or light-shielding exposure. .   6. The imaging according to claim 3, wherein the correction amount in the noise correction unit varies at least one condition of ISO sensitivity, exposure time, and temperature at each exposure time when a plurality of normal exposures are performed. apparatus.   The imaging apparatus according to claim 1, wherein the exposure control unit does not depend on an exposure time when performing the light-shielding exposure depending on an exposure time when performing the normal exposure.   The exposure control unit determines a shooting condition at the time of the light-shielded exposure according to a shooting condition at the time of normal exposure shooting, a shooting condition set by an external operation, or a number of shots at the time of multiple exposure. The imaging apparatus according to claim 1. Furthermore, based on the noise extraction result by the noise extraction unit, it has an additional light shielding shooting determination unit for determining whether to perform additional light shielding shooting,
As a result of the determination by the additional light shielding photographing determination unit, when additional light shielding photographing is performed, noise extraction is performed on the light shielding image data acquired by the additional light shielding photographing, and the noise correction is performed. The imaging device according to claim 1. An image sensor that photoelectrically converts a subject image to generate image data, a photographing optical system that forms the subject image on the image sensor, and an exposure block controller that blocks image formation of the subject image on the image sensor In the imaging method of the imaging device having
A state in which normal exposure from the start of exposure to the end of exposure is controlled while the subject image is formed on the image sensor by the photographing optical system, and the image formation of the subject image is blocked by the exposure cutoff control unit And controls the shading exposure from the start of exposure to the end of exposure,
Based on the shading image data generated by the image sensor after performing the shading exposure, noise extraction is performed,
Based on the extraction result of the noise, noise correction is performed on the normal exposure image data generated by the image sensor after performing the normal exposure.
An imaging apparatus characterized by that.