Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
  • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
  • H04N25/68—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to defects
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2209/00—Details of colour television systems
  • H04N2209/04—Picture signal generators
  • H04N2209/041—Picture signal generators using solid-state devices
  • H04N2209/042—Picture signal generators using solid-state devices having a single pick-up sensor
  • H04N2209/045—Picture signal generators using solid-state devices having a single pick-up sensor using mosaic colour filter

Definitions

• the present invention relates to an image processing apparatus and a control method therefor, and in particular to an image processing apparatus that processes captured images and a control method therefor.
• Solid-state image sensors may include defective pixels that occur in a manufacturing process. Such defective pixels are one of the causes of the decreased quality of captured images and the decreased fabrication yield of the solid-state image sensors.
• Laid-Open No. 2008-263521 whereby the defective pixel is corrected using the pair of pixels selected from among the pairs of pixels, each pair of pixels being located so as to sandwich the defective pixel along the four directions of 0°, 45°, 90°, and 135°, there are cases where the accuracy of the pixel correction value acquired through the complementing process is not necessarily sufficient.
• an image processing apparatus that corrects a signal level of a defective pixel included in image data captured by an image sensor provided with a color filter where a plurality of certain colors are regularly arranged, the image processing apparatus comprising: determination means for, based on differences in signal levels of a
• each pair of pixels being located so as to sandwich the defective pixel and have the same color as the defective pixel, determining a direction being across the defective pixel and along which pixels used to calculate the signal level of the defective pixel are located; ratio calculation means for calculating a ratio between signal levels of pixels that are adjacent to the defective pixel and that have a different color from the defective pixel, and signal levels of pixels that are adjacent to pixels being located along the direction determined by the
• a control method for an image processing apparatus that corrects a signal level of a defective pixel of an image sensor provided with color filters where a plurality of certain colors are regularly arranged, the defective pixel being included in image data captured by the image sensor, the control method comprising: a determination step of, based on differences in signal levels of a plurality of pairs of pixels, each pair of pixels being located so as to sandwich the defective pixel and have the same color as the defective pixel, determining a direction being across the defective pixel and along which pixels used to calculate the signal level of the defective pixel are located; a ratio calculation step of calculating a ratio between signal levels of pixels that are adjacent to the defective pixel and that have a different color from the defective pixel, and signal levels of pixels that are adjacent to pixels being located along the direction determined in the determination step with respect to the defective pixel and having the same color as the defective pixel and that have a different color from the defective pixel; and a first correction processing step of outputting,
• FIG. 1 is a block diagram schematically showing an example of the functional configuration of an image processing apparatus pertaining to First Embodiment of the present invention.
• FIG. 2 is a flowchart for explaining the operations of the image processing apparatus shown in FIG. 1.
• FIG. 3A is a diagram for explaining
• FIG. 3B is a diagram for explaining the operations of a level ratio calculation unit.
• FIG. 4 is a block diagram schematically showing an example of the functional configuration of an image processing apparatus pertaining to Second Embodiment of the present invention.
• FIG. 5 is a block diagram schematically showing an example of the functional configuration of a pixel correction processing unit shown in FIG. 4.
• FIG. 6 is a flowchart for explaining the operations of the image processing apparatus shown in FIG. 4.
• FIG. 7 is a flowchart for explaining the details of processing for calculating pixel correction values in S1001 of FIG. 6.
• FIG. 8A is a diagram for explaining an example of a weighting coefficient al pertaining to Second Embodiment of the present invention.
• FIG. 8B is a diagram for explaining an example of a weighting coefficient a2 pertaining to Second Embodiment of the present invention.
• FIG. 1 is a block diagram schematically showing an example of the functional configuration of an image processing apparatus pertaining to First Embodiment of the present invention. It should be noted here that the functional blocks shown in FIG. 1 may be realized by a computer executing software, or by hardware logic.
• the image processing apparatus may be embodied by, for example, a central processing unit (CPU) controlling hardware (e.g. a storage device and an interface) of a general-purpose computer through execution of a control program.
• CPU central processing unit
• an image processing apparatus 100 can determine pixel data corresponding to a defective pixel from among a plurality of pixel data constituting the input image data. For instance, the image
• processing apparatus 100 may be provided in advance with information relating to coordinates of the
• the image processing apparatus 100 may obtain position information of the defective pixel that is included in the input image data as supplemental information.
• the input image data has been captured using an image sensor provided with a primary-color Bayer filter.
• the primary-color Bayer filter is a mosaic of red (R) , green (G) and blue
• (B) filters that are arranged such that two same- colored filters neighbor each other with one pixel of a different color therebetween, in both vertical and horizontal directions.
• a signal level ratio calculation unit 102 acquires a ratio between the following:
• a first correction processing unit 103 calculates pixel correction data, which serves as a signal level of the defective pixel, by multiplying an average value of the signal levels of the pixels being located at the periphery of the defective pixel and having the same color as the defective pixel by the ratio calculated by the signal level ratio calculation unit 102.
• FIG. 3A only shows a part of the image data that includes pixels located at the periphery of the defective pixel so as to facilitate the understanding and explanation of the present invention.
• the image processing apparatus 100 receives an input of image data in which red pixels, blue pixels and green pixels are arrayed in accordance with the Bayer arrangement (S501) .
• the image data may be input from a removable recording medium such as a semiconductor memory card, or via a network.
• the input image data may have been captured by the imaging device.
• the direction determination unit 101 obtains, from the input image data, the signal levels of pixels R0, Rl, R2, R3, R5, R6, R7 and R8 that are located at the periphery of the defective pixel 800 and have the same color as the defective pixel 800.
• defective pixel denotes the pixels that are adjacent to the defective pixel along the directions across the defective pixel, such as the horizontal direction (0°), the vertical direction (90°), the direction of 45° and the direction of 135°, and have the same color as the defective pixel, with the defective pixel being the center of these pixels.
• the direction determination unit 101 acquires the absolute value
• the direction determination unit 101 determines the direction along which a pair of adjacent pixels whose difference has the smallest absolute value is located as a determined direction 801.
• R2 — R6 I of the difference between pixels R2 and R6 is the smallest, and therefore 45° is obtained as the determined
• the direction determination unit 101 outputs determined direction information indicating the determined direction 801 (S502) .
• the signal level ratio calculation unit 102 extracts the signal levels of:
• green pixels G2, G5, G6 and G9 which are pixels that are adjacent to pixel R2 and have a
• the pixels in the three areas whose signal levels are extracted have the same color.
• the blue pixels that are located diagonally to the defective pixel may be extracted as the pixels that are adjacent to and have a different color from these red pixels.
• the defective pixel is a blue pixel
• processing can be executed in a similar manner as in the case where the defective pixel is a red pixel.
• the defective pixel is a green pixel
• the pixels that are adjacent to the defective pixel and have a different color from the defective pixel are
• the signal level ratio calculation unit 102 may calculate the signal level ratio from the following expressions by using the two red pixels vertically adjacent to the green pixel and the two blue pixels horizontally adjacent to the green pixel in all of the three areas.
• the signal level ratio calculation unit 102 calculates the signal level ratio in accordance with the following Expressions 1 through 4 (S503) .
• G_AVE1 (G14 + G18 + G17 + G21)/4
• G_AVE2 (G2 + G5 + G6 + G9)/4 [0039] [Expression 4]
• the 103 extracts the signal levels of pixels R2 and R6 from among the pixels that are located at the periphery of the defective pixel and have the same color as the defective pixel, the pixels R2 and R6 being located along the direction (45°) indicated by the determined direction information obtained from the direction determination unit 101. Thereafter, the first
• correction processing unit 103 calculates a pixel correction value that serves as the signal level of the defective pixel in accordance with the following
• the certain direction is selected from among a plurality of directions determined by the arrangement of a color filter. In the case of the Bayer arrangement, the certain direction is one of 0°, 45°, 90°, and 135°.
• the pixel correction value is corrected using the acquired ratio. This correction makes it possible to obtain the pixel
• the included defective pixel may not be used, or alternatively, another pixel that is second-adjacent to the defective pixel targeted for correction may instead be used (provided that the included defective pixel is first-adjacent to the defective pixel targeted for correction) .
• the direction of 45° may not be used as the determined direction, or alternatively, the direction of 45° may be used as the determined direction with the use of the value of the second-adjacent pixel (provided that the pixel R2 is the first-adjacent pixel) , namely a red pixel that is not shown in FIG. 3A and is two pixels away from the pixel R2 along the direction of 45°.
• the pixel correction processing units 200- 1 to 200-4 respectively calculate pixel correction values corresponding to the directions of 0°, 45°, 90°, and 135°.
• a selection unit 701 selects one of the pixel correction values from the pixel correction processing units 200-1 to 200-4 based on the result output from the direction determination unit 101, and outputs the selected value as the final pixel
• the first weighting coefficient calculation unit 202 calculates a weighting coefficient al for the correction value obtained from the second correction processing unit 201 and the correction value obtained from the first correction processing unit 103.
• Embodiment and therefore a description thereof is omitted.
• the following description is provided under the assumption that the pixel correction value for the defective pixel shown in FIG. 3A is calculated, as in First Embodiment.
• the second correction processing unit 201 calculates, as the pixel correction data, an average value of the signal levels of pixels R2 and R6 that are located so as to sandwich the defective pixel 800 along the
• the weighting coefficient al may instead be calculated in accordance with the average value (G AVE0, G AVE 1, G AVE2), the largest value (G_AVE0, G_AVE_1, G_AVE2), and the like. Furthermore, the relationship between G_AVE_MIN and the weighting coefficient al shown in FIG. 8A may be stored in the first weighting coefficient calculation unit 202 as a table or as a function with G_AVE_MIN serving as an argument .
• weighting coefficient a2 may instead be calculated in accordance with the average value (R2, R6, G8, Gil, G12, G15, G2, G5, G6, G9, G14, G17, G18, G19), the smallest value (R2, R6, G8, Gil, G12, G15, G2, G5, G6, G9, G14, G17, G18, G19), and the like. Furthermore, the average value (R2, R6, G8, Gil, G12, G15, G2, G5, G6, G9, G14, G17, G18, G19), and the like. Furthermore, the average value (R2, R6, G8, Gil, G12, G15, G2, G5, G6, G9, G14, G17, G18, G19), and the like. Furthermore, the average value (R2, R6, G8, Gil, G12, G15, G2, G5, G6, G9, G14, G17, G18, G19), and the like. Furthermore, the average value (R2, R6, G8, Gil, G12, G15, G2,
• the second inclination 402 be configured so that the G region is not affected by the saturated region.
• a specific value of the second inclination 402 can be acquired through experimentation and the like.
• the weighting coefficient selection unit 204 compares the weighting coefficient al calculated in S602 with the weighting coefficient a2 calculated in S603, and calculates (selects) the smaller value as the final weighting coefficient a in accordance with the following Expression 9.
• Weighting coefficient a smallest value (weighting coefficient oil, weighting coefficient
• the above selection of the smaller weighting coefficient reduces the weight for the pixel correction value acquired with consideration of the ratio of the color signal levels, and therefore suppresses the effects of the dark region and the saturated region.
• the third correction processing unit 205 The third correction processing unit 205
• the final pixel correction value is acquired by performing weighted addition on the pixel correction value acquired without consideration of the ratio of the color signal levels calculated by the signal level ratio calculation unit 102, and the pixel correction value acquired with consideration of the ratio of the color signal levels calculated by the signal level ratio calculation unit 102.
• the degree of reliability of the ratio of the color signal levels calculated by the signal level ratio calculation unit 102 is considered to be low, the weight for the pixel correction value acquired with consideration of the ratio of the color signal levels is reduced. In this way, when the degree of reliability of the ratio of the color signal levels is high, the present
• the present embodiment achieves the advantageous effects that are similar to those achieved by First Embodiment, and when the degree of reliability of the ratio of the color signal levels is low, the present embodiment can suppress erroneous correction of the pixel correction value .
• Second Embodiment also, instead of green pixels, blue pixels may be extracted as the pixels that are adjacent to and have a different color from the defective pixel, pixel R2 and pixel R6.
• the above- described processing is applicable as-is also in the case where the defective pixel is a blue pixel.
• the green pixels are closer to the red pixels than the blue pixels are to the red pixels, it is thought that the correction accuracy is higher when the green pixels are used as the pixels adjacent to the red pixels than when the blue pixels are used as the pixels adjacent to the red pixels.
• the defective pixel is a green pixel
• the pixels that are adjacent to the defective pixel and have a different color from the defective pixel are constituted by two blue pixels and two red pixels.
• the signal levels of the pixels of both colors may be extracted, or the signal levels of the pixels of one of the two colors may be extracted.
• the principle of the present invention can be applied to image data that has been captured by an image sensor provided with color filters where a plurality of certain colors are regularly arranged .
• Second Embodiment has described a configuration in which the pixel correction values are generated in correspondence with the directions that can be determined by the direction determination unit 101 (in the case of a color filter with the Bayer arrangement, the directions of 0°, 45°, 90°, and 135°), and one of the pixel correction values is selected based on the direction determined by the direction determination unit 101.
• a pixel correction value is calculated in correspondence with the
• the signal level ratio calculation unit 102 the first correction processing unit 103 and the second correction
• processing unit 201 execute similar processing with respect to the determined direction instead of the aforementioned direction of 45°, and the output from the third correction processing unit 205 serves as the pixel correction value for the defective pixel.
• embodiment ( s ) and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described embodiment ( s ) .
• the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium) .

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Transforming Light Signals Into Electric Signals (AREA)
• Color Television Image Signal Generators (AREA)

Priority Applications (3)

Applications Claiming Priority (4)

Publications (1)

Family

ID=46750394

Family Applications (1)

Country Status (5)

Families Citing this family (11)

Citations (6)

Family Cites Families (11)

• 2012
  • 2012-06-21 JP JP2012140037A patent/JP6046927B2/ja not_active Expired - Fee Related
  • 2012-08-07 EP EP12751132.7A patent/EP2742681B1/en not_active Not-in-force
  • 2012-08-07 US US14/131,377 patent/US9030579B2/en not_active Expired - Fee Related
  • 2012-08-07 CN CN201280038880.0A patent/CN103733608B/zh not_active Expired - Fee Related
  • 2012-08-07 WO PCT/JP2012/070586 patent/WO2013022113A1/en not_active Ceased

Patent Citations (6)

Also Published As

Similar Documents

Legal Events