JP2013162223A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid deterioration of an S/N ratio due to flaw correction processing.SOLUTION: An image processing device acquires main image data 421 captured by imaging means in an exposed state and black image data 425 captured by the imaging means while light is shielded. The image processing device does not subtracts, from the main image data 421, black image data 425 corresponding to a pixel having no flaw among plural pixels of the imaging means but subtracts, from the main image data 421, black image data 425 corresponding to a pixel having a flaw among plural pixels.

Description

  The present invention relates to a technique for processing image data captured by an imaging unit in which a plurality of pixels are arranged.

  In recent years, imaging devices such as home video cameras and digital still cameras have become popular. The number of pixels of the image sensor mounted on these image pickup devices ranges from several million to over 10 million, and a certain amount of substrate defects necessarily exist at a certain ratio in a semiconductor wafer. Accordingly, if all of the image pickup devices having one point defective pixel in mass production are made defective, the yield is remarkably lowered and the manufacturing cost is increased. Yield is improved and manufacturing costs are reduced.

  As a conventional technique for correcting such point defect pixels, generally, black image data is photographed after photographing the main image data, and the point defect pixels (scratches) are subtracted by subtracting the black image data from the main image data. There is a blacking shooting process to be corrected. Patent Document 1 stores the position of a defective pixel (scratch) included in an image sensor and the data of the defect component level included in the output signal, and predicts the amount of defect according to the charge accumulation time and the shooting temperature. Thus, a technique for performing correction is disclosed.

JP 2008-187255 A

  However, the technique disclosed in Patent Document 1 has a problem in that the number of defective pixels tends to increase as the number of pixels increases. In addition, since the black image capturing process includes random noise components in the black image data, there is a problem that random noise is added even to pixels having no defects after the black image data is subtracted from the main image data. is there.

  Further, in order to predict the defect component level, the temperature of the image sensor must be accurately measured. However, even if a temperature sensor is arranged outside the image sensor, it is difficult to accurately measure the internal temperature, and it is difficult to accurately predict the defect component level, so that the defect component remains and the image quality is impaired. .

  Therefore, an object of the present invention is to avoid the deterioration of the S / N ratio due to the scratch correction process.

  The image processing apparatus of the present invention is an image processing apparatus that processes image data captured by an imaging unit in which a plurality of pixels are arranged, and is first image data captured by the imaging unit in an exposed state. Acquisition means for acquiring the second image data captured by the imaging means in a light-shielded state, and the second image data corresponding to a pixel having no scratch among the plurality of pixels. And processing means for executing a scratch correction process for subtracting the second image data corresponding to the scratched pixel from the first image data without subtracting from the first image data. Features.

  According to the present invention, it is possible to avoid the deterioration of the S / N ratio due to the scratch correction process.

It is a figure which shows the structure of the image processing apparatus which concerns on embodiment of this invention. 6 is a flowchart illustrating a photographing process of the image processing apparatus according to the embodiment of the present invention. It is a figure which shows the example of the table for managing the imaging | photography process corresponding to the temperature and shutter speed which are imaging | photography conditions. It is a figure which shows the flow of a defective pixel correction process. It is a figure which shows notionally the process which interpolates the pixel shown by defective pixel address data from a surrounding pixel.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. In FIG. 1, reference numeral 100 denotes an image processing apparatus according to the present embodiment. Reference numeral 10 denotes a photographing lens. Reference numeral 12 denotes a mechanical shutter having a diaphragm function for controlling exposure and light shielding with respect to an image sensor to be described later. Reference numeral 14 denotes an imaging device in which a plurality of pixels are arranged and converts an optical image into an electrical signal. Reference numeral 16 denotes an A / D converter that converts an analog signal output from the image sensor 14 into a digital signal. A timing generation circuit 18 supplies a clock signal and a control signal to the image sensor 14, the A / D converter 16, and the D / A converter 26, and is controlled by the memory control circuit 22 and the system control circuit 50. It is possible to control the charge accumulation time by controlling the reset timing of the image sensor 14 and use it as an electronic shutter for moving image shooting or the like.

  An image processing circuit 20 performs predetermined pixel interpolation processing and color conversion processing on the image data input from the A / D converter 16 or the image data input from the memory control circuit 22. Also, the image processing circuit 20 cuts out image data and performs a scaling process to realize an electronic zoom function. In the image processing circuit 20, predetermined calculation processing is performed using the captured image data, and the system control circuit 50 controls the exposure control unit 40 and the distance measurement control unit 42 based on the obtained calculation result. TTL AF processing, AE processing, and EF processing are performed. Further, the image processing circuit 20 performs predetermined arithmetic processing using the captured image data, and also performs TTL AWB (auto white balance) processing based on the obtained arithmetic result.

  A memory control circuit 22 controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the image display memory 24, the D / A converter 26, the memory 30, and the compression / decompression circuit 32. The data of the A / D converter 16 is sent to the image display memory 24 or the memory 30 via the image processing circuit 20 and the memory control circuit 22 or the data of the A / D converter 16 is directly sent to the memory control circuit 22. Written.

  Reference numeral 24 denotes an image display memory. Reference numeral 26 denotes a D / A converter. Reference numeral 28 denotes an image display unit composed of a TFT LCD or the like, and display image data written in the image display memory 24 is displayed by the image display unit 28 via the D / A converter 26. The electronic viewfinder function can be realized by sequentially displaying the image data captured using the image display unit 28. The image display unit 28 can arbitrarily turn on / off the display according to an instruction from the system control circuit 50. When the display is turned off, the power consumption of the image processing apparatus 100 can be greatly reduced. Can do.

  Reference numeral 30 denotes a memory for storing captured still image data and moving image data, and has a sufficient storage capacity for storing a predetermined number of still image data and moving image data for a predetermined time. This makes it possible to write a large amount of image data to the memory 30 even in the case of continuous shooting or panoramic shooting in which a plurality of still image data are continuously shot. The memory 30 can also be used as a work area for the system control circuit 50.

  A compression / decompression circuit 32 compresses / decompresses image data by adaptive discrete cosine transform (ADCT) or the like, reads image data stored in the memory 30, performs compression processing or decompression processing, and stores the processed data in the memory Write to 30. Reference numeral 40 denotes an exposure control unit that controls the shutter 12 having a diaphragm function, and has a flash light control function in cooperation with the flash 48.

  A distance measurement control unit 42 controls focusing of the photographing lens 10. A zoom control unit 44 controls zooming of the photographing lens 10. A barrier control unit 46 controls the operation of the protection unit 102 serving as a barrier.

  A flash 48 has an AF auxiliary light projecting function and a flash light control function. The exposure control unit 40 and the distance measurement control unit 42 are controlled using a TTL method, and the system control circuit 50 is based on the calculation result of the image processing circuit 20 for the captured image data, and the exposure control unit 40 and the distance measurement control. The unit 42 is controlled. Reference numeral 50 denotes a system control circuit that controls the entire image processing apparatus 100. Reference numeral 52 denotes a memory that stores constants, variables, programs, and the like for operation of the system control circuit 50.

  Reference numeral 54 denotes a display unit such as a liquid crystal display device or a speaker that displays an operation state, a message, and the like using characters, images, sounds, and the like according to execution of a program in the system control circuit 50. The display unit 54 is installed at an easily visible position near the operation unit of the image processing apparatus 100, and includes, for example, a combination of an LCD, an LED, a sounding element, and the like.

  The display unit 54 has a part of its function installed in the optical viewfinder 104. Among the display contents of the display unit 54, what is displayed on the LCD or the like includes single shot / continuous shooting display, self-timer display, compression rate display, recording pixel number display, recording number display, remaining image number display, shutter There are speed display, aperture value display, exposure compensation display, flash display, red-eye reduction display, macro shooting display, and the like. Further, a buzzer setting display, a battery remaining amount display, an error display, an information display with a multi-digit number, an attachment / detachment state display of the recording media 200 and 210, a communication I / F operation display, and the like are possible.

  Among the display contents of the display unit 54, what is displayed in the optical viewfinder 104 includes in-focus display, camera shake warning display, flash charge display, shutter speed display, aperture value display, exposure correction display, and the like.

  Reference numeral 56 denotes an electrically erasable and recordable non-volatile memory, such as an EEPROM. Reference numerals 60, 62, 64, 66, 70, 72, and 74 denote operation means for inputting various operation instructions of the system control circuit 50, such as switches, dials, touch panels, pointing by line-of-sight detection, voice recognition devices, and the like. Consists of a single or a plurality of combinations.

  Here, a specific description of these operating means will be given. Reference numeral 60 denotes a mode dial switch that switches between function modes such as power off, automatic shooting mode, shooting mode, panoramic shooting mode, movie shooting mode, playback mode, multi-screen playback / erase mode, PC connection mode, and TV reception mode. Can be set.

  Reference numeral 62 denotes a shutter switch SW1, which is turned ON during the operation of the shutter button, and performs operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing. Instruct the start.

  A shutter switch SW2 64 is turned on when the operation of the shutter button is completed. The shutter switch SW <b> 2 is an exposure process that writes image data to the memory 30 via the A / D converter 16 and the memory control circuit 22, and an operation in the image processing circuit 20 and the memory control circuit 22. Instructs the start of a series of processing operations such as development processing using, image data read from the memory 30, compression performed by the compression / decompression circuit 32, and recording processing for writing image data to the recording medium 200 or 210.

  Reference numeral 66 denotes a display changeover switch, which instructs display changeover of the image display unit 28. With this function, when photographing using the optical viewfinder 104, it is possible to save power by cutting off the current supply to the image display unit including a TFT LCD or the like.

  An operation unit 70 includes various buttons, a touch panel, and the like, and includes a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single shooting / continuous shooting / self-timer switching button, and the like. Also, menu movement + (plus) button, menu movement-(minus) button, playback image movement + (plus) button, playback image-(minus) button, shooting image selection button, exposure compensation button, date / time setting button, etc. There is also.

  Reference numeral 72 denotes a zoom switch unit serving as a zoom operation means for a user to give an instruction to change the magnification of a captured image. The zoom switch 72 includes a tele switch that changes the imaging field angle to the telephoto side and a wide switch that changes the imaging angle of view to the wide angle side. By using the zoom switch 72, the zoom control unit 44 is instructed to change the imaging field angle of the photographing lens 10, and becomes a trigger for performing an optical zoom operation. Further, the use of the zoom switch 72 also serves as a trigger for electronic change of the imaging angle of view by image data extraction by the image processing circuit 20 or pixel interpolation processing. Reference numeral 74 denotes a subject detection unit, which includes an element for detecting the subject. Examples of the subject detection method include a method for detecting the face of the subject.

  A power control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches a block to be energized, and the like, and detects whether or not a battery is installed, the type of battery, and the remaining battery level. Further, the DC-DC converter is controlled based on the detection result and the instruction of the system control circuit 50, and a necessary voltage is supplied to each part including the recording medium for a necessary period.

  82 and 84 are connectors. Reference numeral 86 denotes a power supply means including a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

  Reference numerals 90 and 94 denote interfaces with recording media such as memory cards and hard disks. Reference numerals 92 and 96 denote connectors for connecting to a recording medium such as a memory card or a hard disk.

  In the present embodiment, the description is given assuming that there are two systems of interfaces and connectors for attaching the recording medium. Of course, the interface and the connector for attaching the recording medium may have a single or a plurality of systems, any number of systems. Moreover, it is good also as a structure provided with combining the interface and connector of a different standard. The interface and the connector may be configured using an SD card, a CF card, or the like that conforms to the standard.

  Further, the interfaces 90 and 94, and the connectors 92 and 96 can be configured by using ones conforming to a standard such as a PCMCIA card or a CF (Compact Flash (registered trademark)) card. In other words, by connecting a communication card such as a LAN card, modem card, USB card, IEEE 1394 card, SCSI card or PHS, image data and management attached to peripheral devices such as printers and printers. Information can be transferred to each other.

  Reference numeral 102 denotes protection means that is a barrier that prevents the imaging unit from being soiled or damaged by covering the imaging unit including the lens 10 of the image processing apparatus 100.

  Reference numeral 104 denotes an optical viewfinder, which can take an image using only the optical viewfinder without using the electronic viewfinder function of the image display unit 28. In the optical viewfinder 104, some functions of the display unit 54, for example, a focus display, a camera shake warning display, a flash charge display, a shutter speed display, an aperture value display, an exposure correction display, and the like are installed.

  A communication unit 110 has various communication functions such as USB, IEEE 1394, LAN, and wireless communication. Reference numeral 112 denotes a connector for connecting the image processing apparatus 100 to another device by the communication unit 110 or an antenna in the case of wireless communication.

  Reference numeral 200 denotes a recording medium such as a memory card or a hard disk. The recording medium 200 includes a recording unit 202 composed of a semiconductor memory, a magnetic disk, or the like, an interface 204 with the image processing apparatus 100, and a connector 206 for connecting to the image processing apparatus 100. Reference numeral 210 denotes a recording medium such as a memory card or a hard disk. The recording medium 210 includes a recording unit 212 composed of a semiconductor memory, a magnetic disk, or the like, an interface 214 with the image processing apparatus 100, and a connector 216 that connects to the image processing apparatus 100.

  FIG. 2 is a flowchart showing a photographing process of the image processing apparatus 100 according to the present embodiment. The photographing process shown in FIG. 2 is executed when the shutter switch SW2 (64) is pressed.

  In step S101, the system control circuit 50 acquires the temperature T and the shutter speed Tv as shooting conditions at the time of shooting. In step S102, the system control circuit 50 determines the noise processing type based on the shooting conditions at the time of shooting. That is, in the noise processing determination in step S102, the system control circuit 50 determines a noise processing type from the relationship between the temperature T and the shutter speed Tv with reference to a table as shown in FIG. If it is determined in the noise processing determination in step S <b> 102 that the black-drawing shooting process is performed, the process proceeds to step S <b> 103. In step S <b> 103, the system control circuit 50 performs control so as to execute blacking shooting processing. On the other hand, in the noise processing determination in step S102, when it is determined that the normal shooting process is performed, the process proceeds to step S104. In step S104, the system control circuit 50 performs control so as to execute normal photographing processing. On the other hand, in the noise processing determination in step S102, when it is determined that the adaptive blacking shooting process is performed, the process proceeds to step S105. In step S <b> 105, the system control circuit 50 performs control so as to execute adaptive blackout shooting processing.

  FIG. 3 is a diagram illustrating an example of a table for managing the shooting process corresponding to the temperature T and the shutter speed Tv that are the shooting conditions at the time of shooting the main image data. In the example of FIG. 3, when the temperature T is not less than 30 degrees and less than 50 degrees and the shutter speed is not less than 1/80 and less than 1/8 as the shooting condition, the noise processing type corresponding to the shooting condition is adaptive black. It is shown that this is a pulling photographing process.

  Under imaging conditions in which normal imaging processing is selected as the noise processing type, that is, imaging conditions in which the level of the main image data is saturated, scratches (defective pixels) are relatively inconspicuous. For this reason, only the defective pixels indicated by the defect (defective pixel) address data stored in advance are corrected.

  FIG. 4A shows the flow of a defect (defective pixel) correction process when the noise processing type is a normal photographing process. Hereinafter, with reference to FIG. 4A, the flaw correction processing when the noise processing type is the normal photographing processing will be described. First, the system control circuit 50 stores the main image data 401 photographed as usual in the memory 30. Next, the system control circuit 50 interpolates the image data of the defective pixel indicated by the scratch address data 402 stored in advance among the main image data 401 with the image data of the peripheral pixels. As a result, image data (hereinafter referred to as processed image data) 404 that has been subjected to scratch correction processing is generated.

FIG. 5 is a diagram conceptually illustrating a process of interpolating the image data of the defective pixel indicated by the defective pixel address data with the image data of the peripheral pixels. When there is a defective pixel 501 indicated by defective pixel address data in the center of FIG. 5, four “R” peripheral pixels 502 of the same color arranged around the “R” defective pixel 501 as shown in the following Expression 1. The image data of ˜505 are averaged and replaced with the image data of the “R” defective pixel 501.
“R” defective pixel 501 = (the upper “R” peripheral pixel 502 + the lower “R” peripheral pixel 503 + the right “R” peripheral pixel 504 + the left “R” peripheral pixel 505) / 4.

  Under the shooting condition in which the black-drawing shooting process is selected as the noise processing type, more scratches than the defect address data stored in advance are visible. For this reason, only the defect correction for the defective pixel indicated by the defective pixel address data is insufficient. Therefore, it is necessary to subtract the black image data from the entire main image data so that the scratches are not noticeable.

  FIG. 4B shows the flow of the scratch correction process when the noise processing type is blacking shooting processing. Hereinafter, with reference to FIG. 4B, the flaw correction processing in the case where the noise processing type is the black image capturing processing will be described. First, the system control circuit 50 stores the main image data 411 photographed as usual in the memory 30. Next, the system control circuit 50 causes the black image data 415 to be photographed without changing the shutter speed and ISO sensitivity while the shutter is closed, and stores it in the memory 30. Next, after subtracting the black image data 415 from the main image data 411, the system control circuit 50 converts the image data of the defective pixels indicated by the defective pixel address data 412 stored in advance into the image of the peripheral pixels by the scratch correction processing 413. Processed image data 414 is generated by interpolating from the data.

  FIG. 4C shows the flow of the scratch correction process when the noise processing type is the adaptive blacking shooting process. Hereinafter, with reference to FIG. 4C, the flaw correction processing in the case where the noise processing type is the adaptive blacking shooting processing will be described. First, the system control circuit 50 stores the main image data 421 taken as usual in the memory 30. Next, the system control circuit 50 causes the black image data 425 to be photographed without changing the shutter speed and the ISO sensitivity while the shutter is closed, and stores it in the memory 30. The defective pixel filter 426 receives the black image data 425 and outputs the image data of the defective pixel indicated by the defective pixel address data 422 out of the black image data 425 as it is. On the other hand, the pixels not indicated by the defective pixel address data 422 have no scratch (defect). Accordingly, the defective image filter 426 is indicated by the defective address data 422 in the black image data 425 so that the black image data is not subtracted from the main image data 421 for pixels not indicated by the defective pixel address data 422. The image data of no pixels is output as “0”. As a result, the black image data 425 is subtracted from the main image data 421 only for the defective pixel indicated by the defective pixel address data 422, and deterioration of the S / N ratio due to the defect correction processing can be avoided. The main image data 421 is an example of first image data, and the black image data 425 is an example of second image data.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  10: photographic lens, 12: shutter, 14: image sensor, 16: A / D converter, 18: timing generation circuit, 20: image processing circuit, 22: memory control circuit, 24: image display memory, 26: D / A converter, 28: image display unit, 30: memory, 32: image compression / decompression circuit, 40: exposure control unit, 42: distance measurement control unit, 44: zoom control unit, 46: barrier control unit, 48: flash, 50: System control circuit, 52: Memory, 54: Display unit, 56: Non-volatile memory, 60: Mode dial switch, 62: Shutter switch SW1, 64: Shutter switch SW2, 66: Display changeover switch, 70: Operation unit, 72: Zoom switch, 74: Subject detection unit, 80: Power supply control unit, 82: Connector, 84: Connector, 86: Power supply means, 90: Interface, 92 Connector: 94: Interface, 96: Connector, 100: Image processing apparatus, 102: Protection means, 104: Optical viewfinder, 110: Communication unit, 112: Connector (or antenna), 200: Recording medium, 202: Recording unit, 204: interface, 206: connector, 210: recording medium, 212: recording unit, 214: interface, 216: connector, 100: image processing apparatus

Claims (6)

An image processing apparatus that processes image data captured by an imaging unit in which a plurality of pixels are arranged,
Acquisition means for acquiring first image data photographed by the imaging means in an exposed state and second image data photographed by the imaging means in a light-shielded state;
The second image data corresponding to a pixel having a flaw among the plurality of pixels without subtracting the second image data corresponding to a pixel having no flaw among the plurality of pixels from the first image data. An image processing apparatus comprising: a processing unit that executes a defect correction process for subtracting image data from the first image data.   The image processing apparatus according to claim 1, wherein the processing unit performs the flaw correction processing according to a shooting condition at the time of shooting the first image data.   The processing means does not execute a process of subtracting the second image data from the first image data when the first image data is photographed under a photographing condition in which a level of the first image data is saturated. Item 3. The image processing apparatus according to Item 2.   The image processing apparatus according to claim 1, wherein the position of the pixel with the scratch is indicated in advance. An image processing method for processing image data captured by an imaging means in which a plurality of pixels are arranged,
An acquisition step of acquiring first image data captured by the imaging unit in an exposed state and second image data captured by the imaging unit in a light-shielded state;
The second image data corresponding to a pixel having a flaw among the plurality of pixels without subtracting the second image data corresponding to a pixel having no flaw among the plurality of pixels from the first image data. An image processing method comprising: a processing step of executing a scratch correction process for subtracting from the first image data. A program for causing a computer to execute an image processing method for processing image data captured by an imaging unit in which a plurality of pixels are arranged,
An acquisition step of acquiring first image data captured by the imaging unit in an exposed state and second image data captured by the imaging unit in a light-shielded state;
The second image data corresponding to a pixel having a flaw among the plurality of pixels without subtracting the second image data corresponding to a pixel having no flaw among the plurality of pixels from the first image data. A program for causing a computer to execute a processing step of executing a scratch correction process for subtracting the first image data from the first image data.

Images