JP2019153866A - Imaging apparatus and imaging method - Google Patents

Abstract

To provide an imaging apparatus and an imaging method, capable of detecting defective pixels in a short time in consideration of an occurrence frequency of blinking defects.SOLUTION: The imaging method of an imaging apparatus having an imaging device that has a plurality of pixels and photoelectrically converts incident light to generate an image signal, includes steps of: obtaining image signals from the imaging device at different times; adding the obtained plurality of image signals for each pixel (ST1 to ST9); performing comparative light synthesis on the added image signals for each pixel (ST11 to ST17); and detecting image signals exceeding a predetermined threshold among pixel signals included in the comparatively brightly synthesized image signals to store a position of the detected image signals (ST19).SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging apparatus and an imaging method capable of detecting a defective pixel in an imaging element and recording the defective pixel.

  A large number of pixels are two-dimensionally arranged on the image sensor. Among many pixels, there are blinking defective pixels. If this blinking defective pixel exists, the pixel output may be a normal value or an abnormal value. When the blinking defective pixel outputs an abnormal value, a bright spot appears in the blinking defective pixel position in the image. This blinking defect is also called RTS (Random Telegraph Signal) noise, and becomes a noticeable bright spot even after image processing. Therefore, it has been proposed to detect defective pixels by shortening the accumulation time when generating an image signal and increasing the number of times of photographing used for detection (see, for example, Patent Document 1). Further, there is a technique for calculating a σ value of each pixel from a plurality of images and extracting a target blinking defect from the σ value.

JP 2010-74240 A

  In the technique proposed in Patent Document 1, the occurrence frequency of blinking defects is not taken into consideration. For this reason, it is difficult to detect defective pixels according to the occurrence frequency. Further, in the method of detecting a defective pixel from the σ value, it takes time to calculate the σ value, so that the processing time becomes long.

  The present invention has been made in view of such circumstances, and an object thereof is to provide an imaging apparatus and an imaging method capable of detecting defective pixels in a short time in consideration of the occurrence frequency of blinking defects.

  In order to achieve the above object, an imaging apparatus according to a first invention has a plurality of pixels, photoelectrically converts incident light to generate an image signal, and the image signal from the imaging element at different times. Each of the acquired image signals is added to each pixel, an addition unit for adding the acquired image signals for each pixel, a comparatively bright combining unit for comparing and brightening the added image signals for each pixel, and the comparatively bright combined image signal. A detection unit that detects a pixel signal that exceeds a predetermined threshold among the included pixel signals, and a storage unit that stores the position of the detected pixel signal are provided.

An imaging apparatus according to a second invention is the addition number determination unit that determines the addition number of the image signals to be added based on the characteristics of the plurality of image signals or the characteristics of the imaging element in the first invention. Have
In the imaging device according to a third aspect, in the second aspect, the storage unit stores the addition number.

An imaging apparatus according to a fourth invention includes a correction unit that corrects a pixel signal at a position stored in the storage unit among the image signals generated by the imaging unit in the third invention, The correction unit corrects the pixel signal based on the stored addition number.
According to a fifth aspect of the present invention, there is provided an imaging apparatus according to the second aspect, further comprising a comparatively bright composite number determining unit that determines the composite number of image signals to be comparatively brightly combined based on the addition number.

In the imaging device according to a sixth aspect of the present invention based on the fifth aspect, the storage unit stores the composite number.
An image pickup apparatus according to a seventh invention includes a correction unit that corrects a pixel signal at a position stored in the storage unit among the image signals generated by the image pickup unit in the sixth invention, The correction unit corrects the pixel signal based on the stored composite number.

  An image pickup apparatus according to an eighth aspect of the present invention includes a plurality of pixels, photoelectrically converts incident light to generate an image signal, and obtains the image signal from the image sensor at different times, respectively. A comparatively bright combining unit that compares and synthesizes the plurality of image signals for each pixel, a detection unit that detects a pixel signal that exceeds a predetermined threshold among pixel signals included in the comparatively brightly combined image signal, A storage unit for storing the position of the detected pixel signal.

  An image pickup apparatus according to a ninth invention has a plurality of pixels, photoelectrically converts incident light to generate an image signal, and obtains the image signal from the image sensor at different times, respectively. An addition unit that adds the plurality of image signals for each pixel; a detection unit that detects a pixel signal that exceeds a predetermined threshold among the pixel signals included in the added image signal; and the detected pixel signal And a storage unit for storing the positions of these.

  An image pickup apparatus according to a tenth aspect of the present invention includes a plurality of pixels, an image sensor that photoelectrically converts incident light to generate an image signal, and a pixel that is detected as a blinking defect among the plurality of pixels. A storage unit that stores a signal position; and a correction unit that corrects a pixel signal at a position stored in the storage unit among the image signals.

In the imaging device according to an eleventh aspect according to the tenth aspect, the storage unit stores the frequency of the blinking defect together with the position of the pixel signal, and the correction unit corrects the frequency according to the frequency. Select a pixel.
In the imaging device according to a twelfth aspect based on the eleventh aspect, the correction unit selects a pixel to be corrected in accordance with the ISO sensitivity or the shooting mode in addition to the frequency.
In the imaging device according to a thirteenth aspect based on the tenth aspect, the storage unit stores, as the frequency, the number of times of comparatively bright combination performed at the time of detecting the blinking defect or the number of times of addition.

  An image pickup method according to a fourteenth aspect of the present invention is the image pickup method in an image pickup apparatus having a plurality of pixels, photoelectrically converting incident light, and generating an image signal. The obtained plurality of image signals are added for each pixel, the added image signal is subjected to comparatively bright synthesis for each pixel, and among the pixel signals included in the comparatively brightly synthesized image signal, The pixel signal exceeding a predetermined threshold is detected, and the position of the detected pixel signal is stored.

  According to the present invention, it is possible to provide an imaging apparatus and an imaging method capable of detecting defective pixels in a short time in consideration of the occurrence frequency of blinking defects.

1 is a block diagram mainly showing an electrical configuration of a digital camera according to an embodiment of the present invention. 4 is a graph showing the occurrence frequency of blinking defect noise and the pixel level in the digital camera according to the embodiment of the present invention. It is a figure which shows the process sequence in the case of detecting a blinking defect pixel by adding 2 sheets in the digital camera which concerns on one Embodiment of this invention. It is a figure which shows the process sequence in the case of detecting a blinking defective pixel by adding 5 sheets in the digital camera which concerns on one Embodiment of this invention. It is a flowchart which shows the operation | movement of the blinking defective pixel detection of the digital camera in one Embodiment of this invention. It is a flowchart which shows the operation | movement of the threshold value determination of the digital camera in one Embodiment of this invention. 6 is a chart showing the presence or absence of defect correction depending on the blinking frequency and ISO sensitivity conditions in the digital camera of one embodiment of the present invention. It is a flowchart which shows the ISO sensitivity pixel defect correction process of the digital camera in one Embodiment of this invention. 5 is a chart showing the presence / absence of defect correction according to the frequency and shooting mode conditions in the digital camera of one embodiment of the present invention. It is a flowchart which shows the imaging | photography mode and pixel defect correction process of the digital camera in one Embodiment of this invention. 6 is a chart showing presence / absence of defect correction depending on conditions of a photographing mode and a comparatively bright combination number in a digital camera according to an embodiment of the present invention. It is a flowchart which shows the imaging | photography mode and pixel defect correction process of the digital camera in one Embodiment of this invention.

  Hereinafter, as an embodiment of the present invention, an example in which the present invention is applied to a digital camera (hereinafter referred to as “camera”) will be described. This camera has an imaging unit, and the imaging unit converts the subject image into image data, and displays the subject image on a display unit arranged on the back of the main body based on the converted image data. The photographer determines the composition and shutter timing by observing the live view display. During the release operation, image data is recorded on the recording medium. The image data recorded on the recording medium can be reproduced and displayed on the display unit when the reproduction mode is selected.

  This camera measures the blinking defect for each pixel at the time of shipment from the factory, and the position (address information) of the pixel having the blinking defect and the frequency information on which the blinking defect occurs are registered in the internal memory. When the user uses the camera, if image data is output from the imaging unit, defective pixel correction is performed using the registered address information and frequency information. Even after shipment from the factory, the blinking defective pixel position and frequency information may be measured again and re-registered in the internal memory.

  FIG. 1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. The camera in the present embodiment includes an imaging unit 1, an image processing unit 10, a system control unit 20, a bus 31, and each unit connected thereto.

  The imaging unit 1 includes a lens 2, a mechanical shutter 3, an image sensor 4, and a temperature sensor 5. In the present embodiment, description will be made assuming that the lens is configured integrally with the camera body. However, the lens may of course be configured to be detachable from the camera body as an interchangeable lens.

  The lens 2 is an optical system that forms a subject optical image (subject optical image) on the image sensor 4. The lens 2 moves in the direction of the optical axis of the lens 2 based on a control signal from the system control unit 20, and focus adjustment is performed. The lens 2 has an optical aperture inside. The size of the aperture of the optical aperture changes according to the aperture value, and the amount of light on the image sensor 4 is adjusted. The aperture value is a control parameter for adjusting the exposure amount.

  The mechanical shutter 3 is disposed on the optical axis of the lens 2 and on the front side of the image sensor 4. The mechanical shutter 3 controls the time (shutter speed) for guiding the photographing light flux from the lens 2 onto the image sensor 4. If, for example, a focal plane shutter is used as the mechanical shutter 3, the photographing light flux from the lens 2 can reach the image sensor 4 by opening and closing the shutter front curtain and the rear curtain, or the photographing light flux reaches the image sensor 4. The light can be shielded from reaching it. At this time, the time from when the shutter front curtain starts to travel until the shutter rear curtain starts to travel is the shutter speed. The mechanical shutter 3 is controlled by the shutter opening time based on the shutter speed calculated by the system control unit 20.

  The image sensor 4 includes an image sensor such as a CMOS image sensor or a CCD image sensor. The image sensor has a plurality of pixels arranged two-dimensionally and converts each pixel into an electrical signal. The subject light image formed by the lens 2 is converted into an electric signal by each pixel of the image sensor, and image data is generated. The generated image data is output to the image processing unit 10 and the bus 31. The image sensor 4 has a plurality of pixels and functions as an image sensor that photoelectrically converts incident light to generate an image signal.

  The temperature sensor 5 outputs an electrical signal corresponding to the temperature to the system control unit 20. Since the dark current and the like of the image sensor 4 change according to the temperature, the system control unit 20 corrects the image data from the image sensor 4 according to the output of the temperature sensor 5.

  The bus 31 is a signal line for transmitting and receiving signals between the respective units in the camera. In the example illustrated in FIG. 1, an imaging unit 1, an image processing unit 10, a system control unit 20, an internal memory 33, an external memory 36, a display unit 37, an input IF 38, and a condition setting unit 39 are connected to the bus 31. Yes. Although the external memory 36 is also illustrated in FIG. 1, the external memory 31 may be configured by, for example, a memory card that is detachable from the camera, or may not be a configuration unique to the camera. Absent.

  The image processing unit 10 includes an image processing circuit, and performs various image processes on the image data output from the image sensor 4. The image processing unit 10 includes an image composition unit 11, a defect determination unit 14, an addition number determination unit 15, a comparatively bright combination number determination unit 16, a defect correction unit 17, and a development processing unit 18.

  The image composition unit 11 includes an image composition circuit, and composes a plurality of pieces of image data to generate at least one piece of composite image data. The image composition unit 11 includes a comparatively bright composition unit 12 and an image addition unit 13.

  The comparatively bright combining unit 12 includes a comparatively bright combining circuit and performs comparatively bright combining processing. The comparatively bright combination process is a process for generating the comparatively bright combined image by roughly comparing pixel data of corresponding pixels of two images and replacing them with the brighter pixel data. Specifically, the following processing is performed. The comparative bright combination unit 12 compares pixel data of pixels arranged at the same position of the image sensor 4 with respect to the image data read from the image sensor 4 and a plurality of pieces of image data stored in the internal memory 33. . Then, the comparatively bright combination unit 12 generates a comparatively bright combined image using the larger one of the pixel data comparison results, that is, the brighter pixel data. By performing such processing for all the pixel positions, each pixel data of the comparatively bright composite image is composed of only the brightest pixel data.

  The comparatively bright combining unit 12 functions as a comparatively bright combining unit that performs comparatively bright combining of the added image signals for each pixel (for example, see ST13 in FIG. 5). Further, the comparatively bright combining unit 12 functions as a comparatively bright combining unit that acquires the image signals from the image sensor at different times, and performs comparatively bright combining of the acquired image signals for each pixel (for example, FIG. 5). ST13).

  The image addition unit 13 includes an image addition circuit, performs addition synthesis processing, and generates an addition synthesis image. Specifically, the image addition unit 13 adds pixel data arranged at the same position of the image sensor 4 to a plurality of pieces of image data stored in the internal memory 33, and generates an added composite image. The image adding unit 13 functions as an adding unit that acquires image signals from the image sensor at different times and adds the acquired plurality of image signals for each pixel (see, for example, ST5 in FIG. 5). The image adder 13 functions as an adder that acquires the image signals from the image sensor at different times and adds the acquired image signals for each pixel (see, for example, ST5 in FIG. 5).

  The addition number determination unit 15 determines the number of image data to be combined in the image addition unit 13 when the defect detection image is generated (see ST9 in FIG. 5). The number of objects to be determined in the composition process is the number of image data read from the image sensor 4 or the number of image data stored in the internal memory 33. For example, when detecting pixels with a high occurrence frequency of blinking, the number of additions is small, and when detecting pixels with a low occurrence frequency of blinking, a large number of additions is set. When the set number is reached, the image addition is completed. Considerations for setting the added number will be described later in step ST9 of FIG. The addition number determination unit 15 functions as an addition number determination unit that determines the addition number of image signals to be added based on the characteristics of a plurality of image signals or the characteristics of the image sensor.

  The comparatively bright composite number determination unit 16 determines the number of comparatively bright composites in the comparative bright composite unit 12 when generating the defect detection image (see ST17 in FIG. 5). When the preset number of comparatively bright composites is reached, the comparatively bright composite is completed. The comparatively bright composite image data generated here is stored as defect detection image data in the internal memory 33 (see the defect detection image DI in FIGS. 3 and 4 and ST19 in FIG. 5).

  The comparatively bright composite number determination unit 16 functions as a comparatively bright composite number determination unit that determines the composite number of image signals to be comparatively brightly combined (for example, see ST17 in FIG. 5). This combination number may be determined based on the addition number.

  The defect determination unit 14 determines defective pixels in the defect detection image data stored in the internal memory 33. For example, the pixel position is recorded in the internal memory 33 with the upper tens of thousands from the larger pixel data as defects. As another determination method, pixel data of a predetermined threshold or more is used as a defective pixel, and the pixel position is recorded in the internal memory 33. The operation in the defect determination unit 14 will be described later with reference to FIG. The defect determination unit 14 functions as a detection unit that detects a pixel signal that exceeds a predetermined threshold among the pixel signals included in the image signal that has been relatively brightly synthesized (for example, see ST19 in FIG. 5 and FIG. 6). The defect determination unit 14 functions as a detection unit that detects a pixel signal exceeding a predetermined threshold among the pixel signals included in the added image signal (see, for example, ST19 in FIG. 5 and FIG. 6).

  The defect correction unit 17 corrects the pixel data at the defective pixel position recorded in the internal memory 33. That is, as a result of adjustment at the time of factory shipment or the like, if the defect determination unit 14 determines that the pixel is defective, the position of the pixel is stored in the internal memory 33. At the time of photographing by the user, the defect correction unit 17 reads the position of the defective pixel stored in the internal memory 33 and corrects the pixel data corresponding to this position.

  As a method of correcting the pixel value of the defective pixel by the defect correcting unit 17, for example, there is a method of replacing with an average value of four pixels around the same color. However, the correction method is not limited to this. The image corrected by the defect correcting unit 17 is stored in the internal memory 33 as a final composite image. When registering defective pixels, information on the number of added images and the number of comparatively bright composite images is also recorded, so that it is possible to select an optimal value for the correction method and the number of corrections according to the shooting conditions. The operation in the defect correction unit 17 will be described later with reference to FIGS.

  The defect correction unit 17 functions as a correction unit that corrects the pixel signal at the position stored in the storage unit among the image signals generated by the imaging unit (see, for example, FIGS. 7 to 12). The correction unit corrects the pixel signal based on the stored addition number (for example, see FIG. 9). The correction unit corrects the pixel signal based on the stored composite number. The correction unit selects a pixel to be corrected according to the frequency. The correction unit selects a pixel to be corrected according to the ISO sensitivity or the shooting mode in addition to the frequency (see, for example, FIGS. 7 to 12).

  The development processing unit 18 has a development processing circuit, and performs various development processes on the generated RAW image data. Examples of development processing include demosaicing processing, white balance adjustment processing, noise reduction processing, YC signal generation processing, gamma correction processing, resizing processing, and image compression processing. The resizing process is a process of converting the number of pixels of the image data read from the image sensor 4 into another number of pixels, and this process can be adjusted to the number of display pixels of the display unit 37, for example.

  The display unit 37 includes a display panel such as a TFT (Thin Film Transistor) liquid crystal or an organic EL (Electro Luminescence). The display unit 37 is arranged as a rear display unit provided on the rear surface of the camera body, an EVF (Electronic View Finder) that observes the display through the eyepiece unit, or the like. The display unit 37 displays the image developed by the development processing unit 18.

  The input IF is an operation unit for performing various settings and instructions for the camera in response to an operation of the photographer. The input IF 38 includes, for example, operation members such as a power button for turning on / off the power of the camera or a release button for operating shooting start and end, and is provided on the front surface of the rear display unit or the like. A touch panel is provided. When a touch panel is provided, settings and instructions can be input by a touch operation.

  The condition setting unit 39 sets shooting conditions of the camera and conditions regarding processing of the shot image on the menu screen displayed on the display unit 37.

  The internal memory 33 stores image data read from the image sensor 4 or image data being processed by the image processing unit 10. The internal memory 33 is a storage unit that also stores various processing programs and setting values necessary for the operation of the camera. The CPU in the system control unit 20 controls each unit in the camera according to the processing program stored in the internal memory 33. The internal memory 3 is configured by, for example, a non-volatile memory such as a flash memory or SDRAM, a volatile memory, or a combination thereof.

  The internal memory 33 functions as a storage unit that stores the position of the detected pixel signal as a pixel signal that exceeds a predetermined threshold among the pixel signals included in the image signal. This storage unit stores the addition number. Note that the number of additions for the addition process may be stored in this memory by providing a memory in the addition number determination unit 15 in addition to storing it in the internal memory 33. In addition, the storage unit stores a composite number for comparatively bright composite. Note that the number of composites for comparatively bright combination may be stored in this memory by providing a memory in the comparatively bright composite number determination unit 16 in addition to storing in the internal memory 33. The internal memory 33 functions as a storage unit that stores the position of a pixel signal detected as a blinking defect among a plurality of pixels. The storage unit stores the frequency of the blinking defect together with the position of the pixel signal (see ST23, ST27, ST31, ST35, ST39, etc. in FIG. 6). The storage unit stores, as a frequency, the number of times of comparatively bright combining performed when the blinking defect is detected or the number of times of addition (for example, see FIGS. 9 to 12).

  The external memory 36 is a memory card or the like that can be loaded into the camera body, or a non-volatile storage medium fixed inside the camera body. The external memory 36 records the image data developed in the development processing unit 18, and the recorded image data is read out during reproduction. Image data read from the external memory 36 can be output to the outside of the camera.

  The system control unit 20 is a controller and includes a CPU (Central Processing Unit), peripheral circuits, and the like. The system control unit 20 controls each unit in the camera according to a processing program stored in the internal memory 33.

Next, the occurrence frequency of blinking defects and the pixel level will be described with reference to FIG. In FIG. 2, the pixel level of each blinking defective pixel is uniform, and the pixel level when 10 images are added is shown for each occurrence frequency. When 10 images are acquired, a blinking defect occurs in one image with a pixel with an occurrence frequency of 10% (the pixel becomes a bright spot), and a defective pixel with an occurrence frequency of 50% results in 5 images. A flashing defect occurs. As the images are added, the pixel level of a pixel having a higher occurrence frequency becomes higher. In the example shown in FIG. 2, when the occurrence frequency is 10%, the pixel level is O ₁ , and when the occurrence frequency is 50%, the pixel level is O ₅ .

  Using the above-described properties, the pixels can be flexibly divided according to the frequency of occurrence of blinking defects. If it is desired to detect only pixels with a high occurrence frequency, detection is performed with a small number of additions. If a pixel with low occurrence frequency is to be detected, detection may be performed with an increase in the number of additions. For the same added number of sheets, if the pixel level is high, the occurrence frequency is high, and if the pixel level is low, the occurrence frequency is low. In the flowchart shown in FIG. 6 to be described later, the occurrence frequency of the blinking defective pixel is classified into five stages.

  Next, a processing sequence in the comparative bright combination unit 12 and the image addition unit 13 will be described with reference to FIG. The example shown in FIG. 3 is a case where the number of added sheets in the image adding unit 13 is set to two. In the flowchart shown in FIG. 5 to be described later, when “2” is set as the addition number for determination in step ST9, the processing sequence shown in FIG. 3 is performed. The horizontal axis direction of the drawing shows the passage of time.

  When image data is first read from the image sensor 4, it is stored in the internal memory 33 as a dark image_1. Thereafter, every time a time corresponding to one frame elapses, dark image_2, dark image_3, dark image_4,... Dark image_N are read from the image sensor 4 and temporarily stored in the internal memory 33. Is remembered. The dark image refers to an image acquired by the image sensor 4 when the mechanical shutter 3 is closed, or an image acquired in a light-shielded state where the mechanical shutter 3 is open but there is no light in the surrounding environment.

  When the dark image_2 is read at time T1, the image adding unit 13 adds the levels of the dark image_1 and the dark image_2 for each pixel level to generate one added composite image S1. Next, when the dark image_3 is read out and the dark image_4 is read out at time T2, the image adding unit 13 adds the levels of the dark image_3 and the dark image_4 for each pixel level. An addition composite image S2 is generated. Thereafter, every time two dark images are read out, the image adding unit 13 adds two dark images for each pixel level to generate one added composite image.

  In addition, when an addition composite image is generated by the image addition unit 13, the comparatively bright combination unit 12 performs comparatively bright combination processing. At time T2, the comparatively bright composite unit 12 performs comparatively bright composite of the additive composite image S1 and the additive composite image S2, and generates a comparatively bright composite image S10. The generated comparatively bright composite image S10 is temporarily stored in the internal memory 33. Subsequently, at time T3, the comparatively bright combination unit 12 performs comparatively bright combination of the addition combined image S3 generated by the image adding unit 13 and the comparatively bright combined image S10 temporarily stored in the internal memory 33. A composite image S11 is generated. Thereafter, each time an additive composite image is generated, the comparatively bright composite unit 12 performs comparatively bright composite processing of the additive composite image and the comparatively bright composite image generated immediately before.

  When the last dark image_N is read and a comparatively bright composite image is generated, the generated comparatively bright composite image becomes the defect detection image DI. The defect detection image DI is temporarily stored in the internal memory 33. Using this defect detection image DI, the pixel position (address information) and frequency information of the defective pixel are detected and registered in the internal memory 33 (see ST19 in FIG. 5 and FIG. 6).

  In the example shown in FIG. 2, the addition / synthesis unit 13 adds two dark images. However, the number of added sheets is not limited to this, and may be 1 to an unlimited number. When the number of added sheets is 1, the addition process by the image addition unit 13 is not performed, and the comparatively bright combination unit 12 performs the comparatively bright combination process every time an image is read from the image sensor 4.

  Next, another example of the processing sequence in the comparative bright combination unit 12 and the image addition unit 13 will be described with reference to FIG. This example is different from the example shown in FIG. 3 in that the image adding unit 13 adds five images. In step ST9 of the flowchart shown in FIG. 5, when “5 sheets” is set as the additional number for determination, the processing sequence shown in FIG. 4 is performed.

  Also in the example shown in FIG. 4, the horizontal axis direction of the drawing indicates the passage of time. When image data is first read from the image sensor 4, it is temporarily stored in the internal memory 33 as a dark image_1. Thereafter, every time a time corresponding to one frame elapses, dark image_2, dark image_3, dark image_4,... Dark image_N are read from the image sensor 4 and temporarily stored in the internal memory 33. Is remembered.

  When the dark image_2 is read at time T11, the image adding unit 13 adds the levels of the dark image_1 and the dark image_2 for each pixel level, and generates one added composite image S1a. Subsequently, when the dark image_3 is read at time T12, the dark image_3 and the added composite image S1a are level-added for each pixel level to generate one added composite image S2a. Thereafter, every time a dark image is read out from the image sensor 4, the dark image and the added composite image synthesized immediately before are level-added for each pixel level to sequentially generate one added composite image.

  At time T13, the image addition unit 13 generates an addition composite image S4a. This addition composite image S4a is an addition composite image of five images of dark image_1 to dark image_5. At time T14, the image adding unit 13 generates an added composite image S4b. This addition composite image S4b is an addition composite image of five images of dark image_6 to dark image_10. Thereafter, every time five dark images are read out, an addition composition for comparatively bright composition is generated.

  Further, when the image adding unit 13 generates an addition combined image using substantially five dark images, the comparatively bright combining unit 12 performs comparatively bright combining processing. At time T14, the comparatively bright composite unit 12 performs comparatively bright composite of the additive composite image S4a and the additive composite image 4b to generate a comparatively bright composite image S10a. The generated comparatively bright composite image S10a is temporarily stored in the internal memory 33. Subsequently, at time T <b> 16, the comparatively bright combination unit 12 performs comparatively bright combination of the addition combined image S <b> 4 c generated by the image adding unit 13 and the comparatively bright combined image S <b> 10 a temporarily stored in the internal memory 33. A composite image S10b is generated. Thereafter, each time an additive composite image corresponding to a preset number of images is generated, the comparatively bright composite unit 12 performs comparative bright composite processing of the additive composite image and the comparatively bright composite image generated immediately before.

  When the last dark image_N is read and a comparatively bright composite image is generated, the generated comparatively bright composite image becomes the defect detection image DI also in the example shown in FIG. The defect detection image DI is temporarily stored in the internal memory 33. Using this defect detection image DI, the pixel position (address information) and frequency information of the defective pixel are detected and registered in the internal memory 33 (see ST19 in FIG. 5 and FIG. 6).

  Next, the defect detection image generation operation according to the present embodiment will be described with reference to the flowchart shown in FIG. This flow is realized by the CPU in the system control unit 20 controlling each unit in the camera according to a program stored in the internal memory 33. This flow is executed before factory shipment, and is executed when a blinking defective pixel is detected. However, even after shipment from the factory, the user can execute this flow on the menu screen, etc., detect the blinking defective pixel, and update the address information and frequency information of the defective pixel registered in the internal memory 33. it can.

  When the flow shown in FIG. 5 is started, first, photographing is performed (ST1). In this step, the image sensor 4 captures an image for a predetermined exposure time in a light shielding state with the mechanical shutter 3 closed. When a predetermined exposure time elapses, image data is read from the image sensor 4 and stored in the internal memory 33.

  Once shooting is performed in step ST1, it is next determined whether or not the number of shots is the first (ST3). As described with reference to FIGS. 3 and 4, the addition process is performed after the second and subsequent shots. As a result of this determination, in the case of shooting the first picture, the addition process is not performed, so step ST5 is skipped.

  If the result of determination in step ST3 is that the number of shots is not the first, addition processing is performed on the image for addition (ST5). In this step, as described with reference to FIGS. 3 and 4, the image addition unit 13 performs an addition process every time image data is read from the image sensor 4.

  If addition processing is performed in step ST5, or if the result of determination in step ST3 is that one image has been shot, then the shot image of ST1 or the added image of ST5 is held as an addition image (ST7). ). Here, when the addition process is performed in step ST <b> 5, the generated addition composite image is temporarily stored in the internal memory 33. If it is determined at step ST3 that the image is the first image, the image data acquired at step ST1 is temporarily stored in the internal memory 33.

  If the image is held in step ST7, it is next determined whether or not the added number has reached a predetermined number (ST9). In this step, the addition number determination unit 15 determines whether or not the number of images subjected to addition processing has reached a predetermined number. For example, in the example illustrated in FIG. 3, it is determined that the timing at which the addition composite image S1 is acquired at time T1 and the timing at which the addition composite image S2 is generated at time T2 have reached a predetermined number. In the example shown in FIG. 4, it is determined that the timing at which the added composite image S4a is acquired at time T13 and the timing at which the added composite image S4b is generated at time T14 have reached a predetermined number.

  The predetermined number in step ST9 may be set as appropriate in accordance with the target blink occurrence frequency in detecting the blink defective pixel. The predetermined number here may be set in the addition number determination unit 15 or may be changed as appropriate according to conditions.

The predetermined number for determining the number of added sheets may be determined in consideration of the following factors.
(A1) Flashing frequency: The added number is changed according to the flashing frequency to be corrected. When it is desired to correct a blinking defect that occurs twice in succession, it is better to reduce the number of additions.
(A2) Pixel size: The generation level and frequency change according to the pixel size. Depending on the imaging characteristics of the image sensor 4, the added number is made variable.
(A3) Pixel addition condition: In the case of driving with a large number of pixel additions, the blinking level is lowered by averaging. In such a case, it is preferable to increase the number of added sheets to facilitate extraction.

(A4) Pixel addition method: When pixel signals from the image sensor 4 are mixed, the flashing level changes depending on whether digital mixing or analog mixing is performed. Therefore, the number of added sheets is set according to the level.
(A5) Usage time (deterioration over time): In the image sensor 4, pixels that generate blinking defects change due to subsequent reasons such as exposure to ultraviolet rays and radiation. Depending on the reason for the occurrence of the subsequent flashing defect, the number of additions may be changed as appropriate.
(A6) Usage method: The blinking frequency that requires correction differs depending on the application (photographing mode) of a still image, a moving image, a through image, or the like. You may make it change the number of additions according to the specification of a camera.

  The above conditions (a5) and (a6) may be considered when the user wants to update the blinking defect correction information (pixel position (address information) and frequency information of defective pixels) after shipment from the factory. If the result of determination in step ST9 is that the added number has not reached the predetermined number, processing returns to step ST1, an image for addition is acquired, and the addition process is repeated.

  As a result of the determination in step ST9, when the number of added sheets reaches a predetermined value, it is determined whether or not the number of added sheets is the first sheet (ST11). When a predetermined number of addition processes are performed in steps ST3 to S9, a comparatively bright combination process is performed next. As described with reference to FIGS. 3 and 4, the comparatively bright combination process is performed after the second combined image. If the result of this determination is that the number after addition is the first, the comparatively bright combination process is not performed, so step ST13 is skipped.

  If the result of determination in step ST11 is that the number after addition is not the first, comparatively bright composite image processing is performed (ST13). In this step, as described with reference to FIGS. 3 and 4, the comparatively bright combination unit 12 performs the comparatively bright combination process every time a predetermined number of addition processes are performed by the image addition unit 13.

  If comparatively bright combining processing is performed in step ST13 or if the result of determination in step ST11 is the first number after addition, the image is then stored as a comparatively bright combining image (ST15). Here, if it is determined in step ST11 that it is the first image, the added image generated in step ST7 is temporarily stored in the internal memory 33. For example, in the example shown in FIG. 3, when the added composite image S1 is generated at time T1, the added image S1 is temporarily stored in the internal memory 33. When the comparatively bright composite process is performed in step ST13, the generated comparatively bright composite image is temporarily stored in the internal memory 33. For example, in the example illustrated in FIG. 3, when the relatively bright composite image S <b> 10 is generated at time T <b> 2, the generated comparative bright composite image S <b> 10 is temporarily stored in the internal memory 33.

  If the image is stored as a comparatively bright composite image in step ST15, it is next determined whether or not the comparatively bright composite number has reached a predetermined number (ST17). In this step, the comparatively bright composite number determination unit 16 determines whether or not the number of images that have undergone comparatively bright composite processing has reached a predetermined number. The predetermined number here may be set in the comparatively bright composite number determination unit 16 or may be changed as appropriate according to the conditions.

The predetermined number for determining the comparatively bright composite number may be determined in consideration of the following factors.
(B1) Flashing frequency: When the flashing frequency is low, the comparatively bright composite number is increased.
(B2) Processing time: It changes depending on whether it is an adjustment process performed in the manufacturing process or an in-camera adjustment performed by the user.
(B3) Additional number of sheets: The number of comparatively brightly combined images changes according to the added number of sheets processing before comparatively bright combination.

  If the result of determination in step ST17 is that the number of comparatively bright composites has not reached the predetermined number, processing returns to step ST1, an image is acquired, and the addition processing and comparatively bright composite processing are repeated.

  As a result of the determination in step ST17, when the number of images that have undergone comparatively bright combination processing reaches a predetermined number, a threshold determination is performed (ST19). In step ST17, when a predetermined number of comparative bright combinations are completed, a defect detection image DI (see FIGS. 3 and 4) is generated. As described with reference to FIG. 2, the pixel level varies depending on the occurrence frequency. That is, the occurrence frequency can be estimated from the pixel level. In order to perform correction according to the occurrence frequency at the time of shooting, the frequency is determined for each pixel position using the defect detection image DI, and the determination result is stored in the internal memory 33. Detailed operation of this threshold determination will be described with reference to FIG.

  Next, the threshold determination operation in step ST19 will be described using the flowchart shown in FIG. If it is determined in step ST17 that the number of comparatively bright composite images has reached a predetermined number, a defect detection image DI is generated. In the threshold determination flow, the pixel level determination is performed for each pixel using the defect detection image DI. In executing this flow, a set value is determined for frequency determination. This set value may be a level that can be regarded as a blinking defect.

  In this flow, when pixel data is sequentially read out from the image data of the defect detection image DI, the process is executed from step ST21. In steps ST23, ST27, ST31, ST35, and ST39, registration is performed in association with the frequency of the pixel address. When this series of processing is completed, the next pixel data is read out and repeated from ST21. When the threshold determination flow is executed for all the pixel data of all the defect detection images DI, the flow returns to the original flow.

  When the threshold determination flow shown in FIG. 6 starts, first, it is determined whether or not the pixel level is higher than five times the set value (ST21). In this step, the pixel level of the pixel data read from the defect detection image DI is compared with a value five times the set value. As a result of this determination, if the pixel level is larger than 5 times the set value, it is registered in the memory together with the pixel address as frequency 5 (ST23). Here, the frequency 5 is set and registered in the internal memory 33 in the address of the pixel to be determined in step ST21.

  If the result of determination in step ST21 is that the pixel level is not greater than 5 times the set value, it is next determined whether or not the pixel level is greater than 4 times the set value (ST25). In this step, the pixel level of the pixel data read from the defect detection image DI is compared with a value four times the set value. As a result of this determination, if the pixel level is greater than four times the set value, it is registered in the memory together with the pixel address as frequency 4 (ST27). Here, the frequency 4 is set as the set address of the pixel to be determined in step ST25 and registered in the internal memory 33.

  If the result of determination in step ST25 is that the pixel level is not greater than 4 times the set value, it is next determined whether or not the pixel level is greater than 3 times the set value (ST29). In this step, the pixel level of the pixel data read from the defect detection image DI is compared with a value three times the set value. As a result of this determination, if the pixel level is greater than three times the set value, it is registered in the memory together with the pixel address as frequency 3 (ST31). Here, the frequency 3 is set and registered in the internal memory 33 in the address of the pixel to be determined in step ST29.

  If the result of determination in step ST29 is that the pixel level is not greater than three times the set value, it is next determined whether or not the pixel level is greater than twice the set value (ST33). In this step, the pixel level of the pixel data read from the defect detection image DI is compared with a value twice the set value. If the pixel level is greater than twice the set value as a result of this determination, it is registered in the memory together with the pixel address as frequency 2 (ST35). Here, in step ST33, the frequency 2 is set as the address of the pixel to be determined and registered in the internal memory 33.

  If the result of determination in step ST33 is that the pixel level is not greater than twice the set value, it is next determined whether or not the pixel level is greater than the set value (ST37). In this step, the pixel level of the pixel data read from the defect detection image DI is compared with the set value. If the result of this determination is that the pixel level is greater than the set value, it is registered in the memory together with the pixel address as frequency 1 (ST39). Here, in step ST37, the frequency 1 is set as a pixel address to be determined and registered in the internal memory 33.

  If the result of determination in step ST37 is that the pixel level is not greater than the set value, no pixel defect is registered (ST41). When the pixel level is smaller than the set value, the pixel defect is not registered because the level is not a blinking defect.

  When the processes of steps ST21 to ST41 are executed, the pixel data of the next pixel is read from the internal memory 33, and the processes of steps ST21 to ST41 are executed again. When the process is executed for the pixel data of all the pixels, the threshold determination flow is terminated and the process returns to the original flow.

  As described above, by performing the flow shown in FIGS. 5 and 6 at the time of adjustment before shipment from the factory or at the time of adjustment by the user, the internal memory 33 stores the position of the defective pixel (defective address information). ) And the frequency (frequency information) at which the pixel has a blinking defect. The frequency indicates the probability of occurrence of a blinking defect. The frequency 5 is more likely to cause the blinking defect than the frequency 1.

  In the flowchart shown in FIG. 6, when registering in the internal memory 33, the frequency information is registered together with the defect address information for each pixel. However, the present invention is not limited to this method, and for example, a defective address may be registered by allocating memory areas for each frequency in advance. Further, in the example of the flowchart illustrated in FIG. 6, the frequency is divided into five frequencies assuming that five images are added. However, the frequency need not be the same as the added number, and the added number need not be five.

  Next, blinking defect correction performed at the time of shooting will be described with reference to FIGS. As described above, in the internal memory 33, defect address information and frequency information of defective pixels are registered. FIG. 7 shows conditions for performing defect correction of pixel data using this registered information. You may make it correct | amend about all the blinking defects registered. However, when the number of pixels to be corrected is large, the correction processing time and resolution are affected. For this reason, there is a merit when the number of corrections is small.

  The example shown in FIG. 7 is a correction example of a defective pixel in the case where the process of repeating simple addition is performed. When adding a plurality of images, the values of the same pixel address of the images of the same size are added. By this addition processing, the same effect as that of long-time shooting can be obtained. However, if the pixel has a blinking defect, the defect level increases as the blinking frequency increases by performing addition processing. Further, the higher the set ISO sensitivity, the higher the defect level of the pixel defect. Therefore, a method of determining the presence or absence of correction depending on the ISO sensitivity and the content of processing is preferable.

  For example, the presence / absence of defect correction may be determined according to the blinking frequency and ISO sensitivity conditions (correction conditions) as shown in FIG. In the table shown in FIG. 7, ◯ indicates a condition for correction, and x indicates a condition for no correction. Note that frequencies 1 to 5 indicate frequencies set in the threshold determination of FIG. In the example of FIG. 7, when the set ISO sensitivity is 100, no correction is performed on pixels with any frequency from frequency 1 to frequency 5. When the set ISO sensitivity reaches 200, correction is performed only for pixels with frequency 5. As the set ISO sensitivity increases, correction is performed even for pixels that are less frequent. When the set ISO sensitivity is 3200 or higher, correction is performed for all pixels having frequencies 1 to 5. As described above, the pixel having the frequency 5 is most likely to cause the blinking defect.

  Further, the distribution may be made only by the processing contents. For example, when shooting a simple single image, only frequency 5 having a high frequency may be always set as a correction target, and frequency 1 to frequency 5 may be set as a correction target at the time of comparative bright combination processing or addition processing.

  The flowchart shown in FIG. 8 shows a processing procedure when correction is performed in accordance with the correction conditions shown in FIG. This flowchart is executed at the time of photographing. At the time of shooting, image data is read from the image sensor 4 and temporarily stored in the internal memory 33. The defect correction unit 17 corrects the image data temporarily stored in the internal memory 33 by using the defect address information and frequency information of the defective pixels registered in the internal memory 33 according to the set ISO sensitivity. I do.

  In this flow, when image data temporarily stored in the internal memory 33 is read out, processing is executed from step ST51. In steps ST51, ST55, ST59, ST63, and ST67, the set ISO sensitivity is determined, and the correction is performed according to the determined ISO sensitivity and frequency (however, if the conditions of ST67 are not satisfied, the correction is not performed). ). When this series of processing is completed, the original flow is restored.

  When the ISO sensitivity pixel defect correction processing flow starts, it is first determined whether or not the ISO sensitivity is 3200 or higher (ST51). Here, the determination is made based on the ISO sensitivity that is currently set manually or automatically by the camera (the same applies to ST55, ST59, ST63, and ST67 described later). As a result of this determination, if the ISO sensitivity is 3200 or higher, defects with frequencies 1 to 5 are corrected (ST53). As described above, in the internal memory 33, defect address information and frequency information of defective pixels are registered. The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 1 to 5. As described with reference to the chart of FIG. 7, when the ISO sensitivity is 3200 or more, even if the pixel level is relatively small, the pixel level is easily amplified and manifested, so that the frequency 1 to 5 Correction is performed on all of the pixels.

  There are various methods for correcting defective pixels. For example, there is a method of replacing pixel data of registered defective pixels with an average value of four pixels around the same color of defective pixels. In addition to this, other pixel interpolation methods such as nearest neighbor interpolation may be used.

  If the result of determination in step ST51 is that ISO sensitivity is not 3200 or higher, it is next determined whether or not ISO sensitivity is 1600 or higher (ST55). If the ISO sensitivity is 1600 or higher as a result of this determination, defects having a frequency of 2 to 5 are corrected (ST57). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 2 to 5.

  If the result of determination in step ST55 is that the ISO sensitivity is not 1600 or higher, it is next determined whether or not the ISO sensitivity is 800 or higher (ST59). As a result of this determination, if the ISO sensitivity is 800 or more, defects having a frequency of 3 to 5 are corrected (ST61). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 3 to 5.

  If the result of determination in step ST59 is that the ISO sensitivity is not 800 or higher, it is next determined whether or not the ISO sensitivity is 400 or higher (ST63). As a result of the determination, if the ISO sensitivity is 400 or more, defects with a frequency of 4 to 5 are corrected (ST65). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 4 to 5.

  If the result of determination in step ST63 is that the ISO sensitivity is not 400 or higher, it is next determined whether or not the ISO sensitivity is 200 or higher (ST67). If the ISO sensitivity is 200 or more as a result of this determination, the defect with frequency 5 is corrected (ST69). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which the defect address information is registered with the frequency information as the frequency 5.

  If the result of determination in step ST67 is that the ISO sensitivity is not 200 or higher, defective pixel correction is not performed (ST71). As described with reference to the chart of FIG. 7, when the ISO sensitivity is less than 200, it is difficult to manifest as a defective pixel when the pixel level of the blinking defect is small. Do not do.

  If correction is not performed in step ST71, or correction is performed in steps ST53, ST57, ST61, ST65, and ST69, the ISO sensitivity pixel defect correction process is terminated.

  As described above, in the ISO sensitivity pixel defect correction processing flow, the target of defective pixel correction is changed according to the ISO sensitivity and frequency. That is, if the ISO sensitivity is high, the influence of the blinking defect is likely to be manifested. Therefore, correction is performed even for a pixel having a low frequency. On the other hand, if the ISO sensitivity is low, the influence of the blinking defect is difficult to be manifested. Correction is performed only for pixels with a high. When there are many pixels to be corrected, the processing time becomes long, and when the pixels to be corrected are narrowed down, a blinking defect that is not corrected becomes conspicuous. In this regard, in this flow, the number of pixels to be corrected can be optimized.

  Next, another example of correction of the blinking defect performed at the time of shooting will be described with reference to FIGS. 9 and 10. In the example shown in FIGS. 7 and 8, defective pixel correction is performed according to ISO sensitivity information and frequency. In the example shown in FIGS. 9 and 10, defective pixel correction is performed according to the shooting mode and frequency.

  FIG. 9 is a chart showing the presence / absence of defect correction depending on the shooting mode and frequency conditions. In FIG. 9, the shooting mode is shown in the vertical column, and the frequency is shown in the horizontal column. As the shooting mode, a still image mode, a moving image / through image mode, a multiple image addition process, and a comparatively bright combination process are shown.

  In FIG. 9, in the case of the still image mode, only one image is used, and the possibility that a blinking defect will occur when this one image is taken is extremely low overall. Therefore, only pixels with a high occurrence frequency are targeted for defective pixel correction. The multiple sheet addition process is a process of adding corresponding pixels of an image. Since several to several tens of images are added during this process, pixels that are less frequently generated than still images are also subject to correction.

  In addition, a moving image is recorded by sequentially capturing a plurality of images. In the case of a roux image, a plurality of images are sequentially captured and displayed. Since moving images / through images are shot at several tens to several hundreds of frames per second, flickering is a cause of flickering. For this reason, in the moving image / through image, a single-image shooting operation is performed, but pixels that occur less frequently than still images are also subject to correction.

  The comparatively bright combination process is a process in which corresponding pixels of an image are compared and replaced with pixel data of bright pixels. In this process, hundreds to tens of thousands of image synthesis are performed, so that there is a high possibility that a blinking defect will occur during the image synthesis process. Therefore, pixels with a low occurrence frequency are also subjected to defective pixel correction.

  The defective pixel correction may be selected from various correction methods as in the case of the defective pixel correction in FIGS. If the target pixel is a normal pixel and defective pixel correction is performed, overcorrection occurs. Therefore, it is preferable to perform correction according to the conditions shown in FIG.

  The flowchart shown in FIG. 10 shows a processing procedure when correction is performed in accordance with the correction conditions shown in FIG. Also in this process, as in the case of FIG. 7 and FIG. 8, image data is read from the image sensor 4 and temporarily stored in the internal memory 33 at the time of shooting. The defect correcting unit 17 corrects the image data temporarily stored in the internal memory 33 using the defect address information and frequency information of the defective pixels registered in the internal memory 33 according to the shooting mode. Do.

  In this flow, when the image data temporarily stored in the internal memory 33 is read out, the process is executed from step ST81. In steps ST81, ST85, ST89, and ST93, the shooting mode is determined, and correction is performed according to the determined shooting mode and frequency (however, correction is not performed if the conditions of ST93 are not satisfied). When this series of processing is completed, the original flow is restored.

  When the flow of the shooting mode pixel defect correction process is started, it is first determined whether or not it is a comparatively bright combination process (ST81). When the user wants to generate an image by comparatively bright combination processing, this combination mode is selected by the input IF 38. In this step, it is determined whether or not a comparatively bright combination processing mode is set. If the result of this determination is that a comparatively bright combination processing mode has been set, defects with frequencies 1 to 5 are corrected (ST83). As described above, in the internal memory 33, defect address information and frequency information of defective pixels are registered. The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 1 to 5. As described with reference to the chart of FIG. 9, when the comparatively bright combination process is performed, the number of images to be combined is large, and thus a blinking defect often occurs during this period. Therefore, correction is performed on all the pixels having the frequencies 1 to 5 including the pixels having a low frequency.

  If the result of determination in step ST81 is that the shooting mode is not comparatively bright combining processing, it is next determined whether or not the shooting mode is processing for adding a plurality of images (ST85). If the result of this determination is that the shooting mode is multiple-sheet addition processing, defects with a frequency of 3 to 5 are corrected (ST87). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 3 to 5.

  If the result of determination in step ST85 is that the shooting mode is not multiple-sheet addition processing, it is next determined whether or not the shooting mode is moving image (ST89). As a result of the determination, in the case of the moving image mode, the defect having the frequency of 4 to 5 is corrected (ST91). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which defect address information is registered with frequency information of frequencies 4 to 5.

  If the result of determination in step ST89 is that the shooting mode is not moving image mode, it is next determined whether or not the shooting mode is still image mode (ST93). If the result of this determination is that the shooting mode is still image mode, defects with frequency 5 are corrected (ST95). The defect correction unit 17 reads the temporarily stored image data, and performs defect pixel correction on the pixel data for each pixel in which the defect address information is registered with the frequency information as the frequency 5.

  If the result of determination in step ST93 is that the shooting mode is not a still image, defective pixel correction is not performed. If the result of determination in step ST93 is not the still image mode, or if correction is performed in steps ST83, ST87, ST91, and ST95, the shooting mode pixel defect correction processing ends.

  As described above, in the shooting mode pixel defect processing flow, the target of defective pixel correction is changed according to the shooting mode and frequency. That is, as in the comparatively bright combination process, the number of images to be processed is large, and the defect when the blinking defect occurs due to the characteristics of the process does not disappear. . On the other hand, when the number of images to be processed is small (one in the still image mode) as in the still image mode, the blinking defect is unlikely to occur, so that correction is performed only for pixels with high frequency. I have to.

  In the example illustrated in FIGS. 9 and 10, only four of the comparatively bright combination processing, the multiple image addition processing, the moving image, and the still image have been described as the shooting modes. However, the present invention is not limited to this. For example, pixel defect correction processing is performed according to the frequency for other combining processing such as depth combining processing, panoramic combining, HDR processing, addition averaging, and multiple exposure. Also good. Further, the number of frequency classifications is not limited to five.

  Next, another example of correction of the blinking defect performed at the time of shooting will be described with reference to FIGS. 11 and 12. In the examples shown in FIGS. 7 to 11, the predetermined number for determining the number of comparatively bright composites for detecting the frequency is a fixed value (see ST17 in FIG. 5). In the example shown in FIGS. 11 and 12, a plurality of predetermined numbers for determination at the time of frequency measurement (adjustment) are prepared, and the frequency for each pixel is registered. At the time of shooting, the frequency is set according to the number of comparatively bright composites used at the time of adjustment according to the shooting mode.

  A description will be given of registering the frequency by changing the number of comparatively bright composites during frequency measurement (during adjustment). Since correction of defective pixels is often replaced by interpolation processing or the like, it is preferable to minimize the correction target. When the number of added images is the same at the time of adjustment (when the flow of FIG. 5 is executed) (see ST9 of FIG. 5), the higher the number of comparatively bright composites (see ST17 of FIG. 5), the higher the detection accuracy and the shooting. It is possible to correct a pixel to be corrected sometimes with high accuracy. In particular, in the case of a flashing defect with a low occurrence frequency, it may be possible to detect the flashing defect at the time of shooting by increasing the number of times of comparatively bright combination at the time of adjustment.

  For example, if the number of additions at the time of adjustment is set to 10, only those having a frequency of blinking defects of 10% can be extracted. However, 10 images are added to generate one added image, and 10 additional images are used to perform comparatively bright combination 10 times, thereby extracting a flashing defect with an occurrence frequency of 1%. Can do. When 100 images are simply added, noise components are also added, and a blinking defect with an occurrence frequency of 1% cannot be extracted. Therefore, when measuring the frequency of the blinking defect, it is preferable to perform not only the addition process but also the comparatively bright combination process.

  Associating the number of comparatively bright composites at the time of frequency measurement with the occurrence frequency, and as in FIG. 9, it becomes possible to select a pixel to be corrected depending on the shooting mode, and to perform accurate correction. Can do. An example is shown in FIG.

  FIG. 11 is a chart showing the presence or absence of defect correction depending on the shooting mode and frequency conditions. The shooting mode is shown in the vertical column of FIG. 11, and the number of times of comparatively bright combination at the time of extracting the blinking defect (during frequency measurement) is shown in the horizontal column. Since only a pixel with a high occurrence frequency is to be corrected, the still image is for a pixel that is generated even with a small number of comparatively bright combinations. In a moving image or a through image, a defect causes flickering. Therefore, a pixel whose occurrence frequency is lower than that of a still image (extracted by increasing the number of times of comparison light combination) is set as a correction target. In the multiple image addition process, several to several tens of images are added, and therefore pixels whose occurrence frequency is lower than that of a still image or a moving image (extracted by increasing the number of times of comparison light combination) are to be corrected. In the comparatively bright combination process, since hundreds to tens of thousands of images are combined, pixels extracted from a comparatively brightly combined number of images are set as correction targets.

  The flowchart shown in FIG. 12 shows a processing procedure when correction is performed in accordance with the correction conditions shown in FIG. In this example, a plurality of comparative bright combination counts for determination in ST17 of FIG. 5 are prepared at the time of frequency measurement, and the address information of pixels exceeding the threshold is registered in the internal memory 33 for each comparative bright combination count. deep. At the time of shooting, image data is read from the image sensor 4 and temporarily stored in the internal memory 33. The defect correction unit 17 performs defect address information and frequency information (number of times of comparatively bright combination) of defective pixels registered in the internal memory 33 according to the shooting mode with respect to the image data temporarily stored in the internal memory 33. Is used to correct.

  In this flow, when image data temporarily stored in the internal memory 33 is read out, processing is executed from step ST101. In steps ST101, ST105, ST109, and ST113, the shooting mode is determined, and the correction is performed according to the determined shooting mode and the number of times of comparatively bright combination (however, if the condition of ST113 is not satisfied, the correction is performed). Do not do). When this series of processing is completed, the original flow is restored.

  When the flow of the shooting mode pixel defect correction process starts, it is first determined whether or not it is a comparatively bright combination process (ST101). When the user wants to generate an image by comparatively bright combination processing, this combination mode is selected by the input IF 38. In this step, it is determined whether or not a comparatively bright combination processing mode is set. If the result of this determination is that a comparatively bright combination processing mode has been set, pixel defects extracted at a comparatively bright combination 1000 times or less are corrected (ST103). As described above, in the internal memory 33, defect address information and frequency information (comparative bright combination number information) of defective pixels are registered. The defect correction unit 17 reads the temporarily stored image data, and performs defective pixel correction on the pixel data for each pixel extracted as having frequency information with the number of times of comparatively bright combination being 1000 or less and having a pixel defect.

  If the result of determination in step ST <b> 101 is that the shooting mode is not comparatively bright combining processing, it is next determined whether or not the shooting mode is processing for adding multiple images (ST <b> 105). If the result of this determination is that the shooting mode is a multiple image addition process, pixel defects extracted in 100 or less comparatively bright combinations are corrected (ST107). The defect correction unit 17 reads out the temporarily stored image data, and performs defective pixel correction on the pixel data for each pixel extracted as having frequency information of which the number of comparatively bright combination is 100 or less and there is a pixel defect.

  If the result of determination in step ST105 is that the shooting mode is not multiple-sheet addition processing, it is next determined whether or not the shooting mode is moving image (ST109). If the result of this determination is that the shooting mode is moving image, pixel defects extracted in 10 or less comparatively bright composites are corrected (ST111). The defect correction unit 17 reads the temporarily stored image data, and performs defective pixel correction on the pixel data for each pixel extracted as having frequency information of which the number of times of comparative bright combination is 10 or less and there is a pixel defect.

  If the result of determination in step ST109 is that the shooting mode is not video, it is next determined whether or not the shooting mode is a still image (ST113). If the result of this determination is that the shooting mode is still image, pixel defects extracted in 5 or less comparatively bright composites are corrected (ST115). The defect correction unit 17 reads the temporarily stored image data, and performs defective pixel correction on the pixel data for each pixel that is extracted as a frequency information when the number of comparatively bright combination is 5 or less and there is a pixel defect.

  If the result of determination in step ST113 is that the shooting mode is not a still image, defective pixel correction is not performed. If the result of determination in step ST113 is not the still image mode, or if correction is performed in steps ST103, ST107, ST111, and ST115, the shooting mode pixel defect correction processing is terminated.

  In this way, in the shooting mode pixel defect processing flow shown in FIG. 12, the target of defective pixel correction is changed according to the shooting mode and the frequency (number of times of comparatively bright combination). That is, the number of times of comparatively bright combination is used in detecting the occurrence frequency of the blinking defect according to the characteristics of the shooting mode. In the case of a shooting mode in which the influence of the blinking defect on the output image is not large, a number of comparisons are performed in the case of the shooting mode in which the influence of the blinking defect on the output image is large on the pixel extracted with a small number of comparatively bright combination Pixel defect correction is performed on the pixels extracted by the number of times of bright synthesis.

  In the example shown in FIG. 11 and FIG. 12, only four of the comparative bright combination processing, the multiple image addition processing, the moving image, and the still image have been described as the shooting modes. However, the present invention is not limited to this. For example, the pixel defect correction processing may be performed according to the frequency in the same manner for other combining processing such as depth combining processing, panoramic combining, HDR processing, addition averaging, and multiple exposure. Good. Further, the number of frequency classifications is not limited to five.

  As described above, the imaging device according to an embodiment of the present invention has a plurality of pixels, and has an imaging element (see image sensor 4) that photoelectrically converts incident light and generates an image signal. Then, image signals are acquired from the image sensors at different times, the acquired image signals are added for each pixel (for example, see ST1 to ST9 in FIG. 5), and the added image signals are compared for each pixel. Bright synthesis is performed (for example, see ST11 to ST17 in FIG. 5). Among the pixel signals included in the comparatively brightly synthesized image signal, a pixel signal exceeding a predetermined threshold is detected, and the position of the detected pixel signal is stored (for example, ST19 in FIG. 5, FIG. 6). reference). For this reason, the defective pixel can be detected in a short time in consideration of the occurrence frequency of the blinking defect.

  Further, in one embodiment of the present invention, comparatively bright combination is performed at the time of defect detection (see ST13 in FIG. 5), and an arbitrary level or higher is determined as a blinking defect with respect to the combined image (ST19 in FIG. 5). FIG. 6). At this time, by performing addition processing on a plurality of images before comparatively bright combination (see ST5 in FIG. 5), the level of the blinking pixel is increased to facilitate extraction of defective pixels. In particular, by performing the addition process, the pixel value (level) increases although the blinking frequency is high, so that it is easy to extract the blinking defect only by simple processing.

  In one embodiment of the present invention, it is possible to change the correction target pixel according to the application (shooting mode) and the blinking frequency. For example, during comparatively bright combination, even if the blinking frequency is low, it is a correction target (see, for example, ST103 in FIG. 12). For example, see ST111 in FIG. 12). In addition to the purpose (photographing mode), the blinking frequency varies depending on the conditions of the image sensor and the driving method, so that correction processing according to the blinking frequency is possible.

  In one embodiment of the present invention, when performing addition synthesis processing or comparatively bright synthesis processing at the time of frequency measurement, the pixel information extracted under each condition is registered in the internal memory according to the condition. (See, for example, FIG. 6). Then, at the time of shooting, it is determined whether or not to correct according to shooting conditions (see, for example, FIGS. 8, 10, and 12). For this reason, it is possible to perform correction according to shooting conditions and blinking frequency.

  In one embodiment of the present invention, detection image data is generated by performing addition processing and comparatively bright combination processing on a plurality of image data. However, the comparatively bright combination process may be omitted and the added image may be used as detection image data, or the addition process may be omitted and the comparatively bright combined image may be used as detection image data.

  Further, in one embodiment of the present invention, when adjustment is performed in a manufacturing process or the like, the number of times of comparatively bright synthesis is increased by taking time (see ST9 and ST17 in FIG. 5), and the camera is operated as a user operation. In the case of carrying out the above, the number of times of comparatively bright synthesis or the like may be reduced to shorten the processing time. The extraction accuracy can be increased as the number of times of performing comparatively bright combination or the like during the adjustment is larger. However, considering the adjustment time, the number of times of processing such as comparatively bright composition may be set as appropriate.

  In one embodiment of the present invention, the defect determination unit 14, the added number determination unit 15, the comparatively bright composite number determination unit 16, and the like are configured separately from the system control unit 20. However, all or a part of each unit may be configured by software and executed by the CPU in the control unit 20 as a matter of course. In addition to the hardware circuit, each unit in the image processing unit 10 may have a hardware configuration such as a gate circuit generated based on a programming language written in Verilog, and a DSP ( A hardware configuration using software such as a digital signal processor may be used. Of course, these may be combined appropriately.

  In this embodiment, a digital camera is used as the imaging device for shooting. However, the camera may be a digital single lens reflex camera, a mirrorless camera, or a compact digital camera. It may be a camera for moving images such as a mobile phone, a smartphone, a personal digital assistant (PC), a personal computer (PC), a tablet computer, a camera built in a game machine, a medical camera, a scientific camera such as a microscope. A camera, an on-vehicle camera, or a surveillance camera may be used. In any case, the present invention can be applied to any device equipped with an image sensor that causes a blinking defect.

  Of the techniques described in this specification, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording method for the recording medium and the recording unit may be recorded at the time of product shipment, may be a distributed recording medium, or may be downloaded via the Internet.

  Further, in one embodiment of the present invention, the operation in the present embodiment has been described using a flowchart. However, the processing procedure may be changed in order, or any step may be omitted. Steps may be added, and specific processing contents in each step may be changed.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Imaging part, 2 ... Lens (aperture), 3 ... Mechanical shutter, 4 ... Image sensor, 5 ... Temperature sensor, 10 ... Image processing part, 11 ... Image composition , 12... Comparative bright combination unit, 13... Image addition unit, 14... Defect determination unit, 15. Defect correction unit 18 ... Development processing unit 20 ... System control unit 31 ... Bus 33 ... Internal memory 36 ... External memory 37 ... Display unit 38 ...・ Input IF, 39 ... Condition setting section

Claims (14)

An imaging device having a plurality of pixels, photoelectrically converting incident light, and generating an image signal;
An adder that obtains the image signals from the image sensor at different times and adds the obtained image signals for each pixel;
A comparatively bright combining unit that performs comparatively bright combining of the added image signal for each pixel;
A detection unit for detecting a pixel signal exceeding a predetermined threshold among the pixel signals included in the comparatively brightly synthesized image signal;
A storage unit for storing the position of the detected pixel signal;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, further comprising: an addition number determination unit that determines an addition number of image signals to be added based on characteristics of the plurality of image signals or characteristics of the imaging element.   The imaging apparatus according to claim 2, wherein the storage unit stores the addition number. Among image signals generated by the imaging unit, a correction unit that corrects a pixel signal at a position stored in the storage unit,
The imaging apparatus according to claim 3, wherein the correction unit corrects a pixel signal based on the stored number of additions.   The imaging apparatus according to claim 2, further comprising a comparatively bright composite number determining unit that determines a composite number of image signals to be subjected to comparatively bright combination based on the addition number.   The imaging apparatus according to claim 5, wherein the storage unit stores the composite number. Among image signals generated by the imaging unit, a correction unit that corrects a pixel signal at a position stored in the storage unit,
The imaging apparatus according to claim 6, wherein the correction unit corrects a pixel signal based on the stored composite number. An imaging device having a plurality of pixels, photoelectrically converting incident light, and generating an image signal;
A comparatively bright combining unit that acquires the image signals from the image sensor at different times, and performs comparatively bright combining of the acquired plurality of image signals for each pixel;
A detection unit for detecting a pixel signal exceeding a predetermined threshold among the pixel signals included in the comparatively brightly synthesized image signal;
A storage unit for storing the position of the detected pixel signal;
An imaging apparatus comprising: An imaging device having a plurality of pixels, photoelectrically converting incident light, and generating an image signal;
An adder that obtains the image signals from the image sensor at different times and adds the obtained image signals for each pixel;
A detection unit for detecting a pixel signal exceeding a predetermined threshold among the pixel signals included in the added image signal;
A storage unit for storing the position of the detected pixel signal;
An imaging apparatus comprising: An imaging device having a plurality of pixels, photoelectrically converting incident light, and generating an image signal;
A storage unit that stores a position of a pixel signal detected as a blinking defect among the plurality of pixels,
Among the image signals, a correction unit that corrects a pixel signal at a position stored in the storage unit;
An imaging device comprising: The storage unit stores the frequency of the blinking defect together with the position of the pixel signal;
The correction unit selects a pixel to be corrected according to the frequency.
The imaging apparatus according to claim 10.   The imaging device according to claim 11, wherein the correction unit selects a pixel to be corrected according to an ISO sensitivity or a shooting mode in addition to the frequency.   The imaging device according to claim 10, wherein the storage unit stores, as the frequency, the number of times of comparatively bright synthesis performed when the blinking defect is detected or the number of times of addition. In an imaging method in an imaging apparatus having a plurality of pixels, having an imaging element that photoelectrically converts incident light and generates an image signal,
The image signals are acquired from the image sensor at different times, and the acquired image signals are added for each pixel.
The above-mentioned added image signal is relatively brightly synthesized for each pixel,
Among pixel signals included in the comparatively brightly synthesized image signal, a pixel signal exceeding a predetermined threshold is detected,
Storing the position of the detected pixel signal;
An imaging method characterized by the above.

Images