CN1981514A - Dynamic identification and correction of defective pixels - Google Patents
Dynamic identification and correction of defective pixels Download PDFInfo
- Publication number
- CN1981514A CN1981514A CNA2004800230898A CN200480023089A CN1981514A CN 1981514 A CN1981514 A CN 1981514A CN A2004800230898 A CNA2004800230898 A CN A2004800230898A CN 200480023089 A CN200480023089 A CN 200480023089A CN 1981514 A CN1981514 A CN 1981514A
- Authority
- CN
- China
- Prior art keywords
- pixel
- value
- differences
- bad
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
- H04N23/843—Demosaicing, e.g. interpolating colour pixel values
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
- H04N25/68—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to defects
- H04N25/683—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to defects by defect estimation performed on the scene signal, e.g. real time or on the fly detection
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Color Television Image Signal Generators (AREA)
- Image Input (AREA)
Abstract
Rather than storing a table within a digital camera or other optical device of defective pixels within its image sensor, defective pixels are dynamically identified by comparing the output values of individual pixels under test with those of neighboring pixels. If the neighbors are all above or below the pixel under test by a defined threshold amount, the output of the pixel under test is replaced with a value calculated from the outputs of neighboring pixels.
Description
Technical field
[0001] the present invention relates generally to the imageing sensor of two-dimensional photodetector pixel unit array, be specifically related to be used to discern independent bad pixel and produce the technology of replacing these bad pixels r signal.
Background technology
[0002] imageing sensor with continuous growth image detection photodetector unit number is used in the digital camera, and they are designed to be sold to a large amount of consumers.Though this video camera has the transducer of 2 to 4 mega pixels in the most common current utilization, yet it also is very usual finding and adopting the digital camera of 5 to 6 mega-pixel image sensors.And this number also may increase in future.Usually, all operations normally of not every optical detecting unit in these transducers; The some of them unit is a bad element.Bad optical detecting unit is undesired to the response of light, thereby the image of formation is distorted.In some cases, even there is not light, one or more optical detecting units may always produce output.In other cases, even very Ming Liang light incides on the sensor surface, the minority optical detecting unit may not produce the output of light.Consequently respectively significantly bright or dark spot in the image of catching.Under the third situation, the light of generation is exported in response to the light intensity that incides on the sensor surface, still, this output not with incide transducer on light intensity correctly relevant.These outstanding defectives can be that cluster that isolate or integrated occurs.Because the different colours of the information of each pixel capture and photographed interrelates in the common transducer, these bad pixels often are regarded as the pixel of false color, and are not only too bright or too dark pixel.
[0003] a large amount of time and cost consumption have on the imageing sensor of these defectives in letter sorting.In order to utilize the transducer of these part bad pixels as much as possible, people utilize electronic type bad pixel correction technique.This technology adopts the preposition tabulation, it can identification sensor in the position at bad pixel place.Utilization is time-consuming calibration steps on each transducer, obtains the position of these bad pixels.This step adopts usually measures the signal output that the black sensor pixel is arranged, and white image incides on the transducer then.For black image, all light are stopped by transducer, and the pixel that output is higher than on the setting threshold is identified as bad pixel.Similarly,, allow uniform illuminated field to incide on the transducer, in order that identification produces on the defined threshold any bad pixel than low-output signal for white image.Any this pixel is considered to be " exception ", and is included in the bad pixel table of each tested transducer.This table comprises the position in the bad pixel array.
[0004] when this calibration chart image-position sensor is installed in digital camera or other main frames, its bad pixel table is with pack into the memory of digital camera of the form of look-up table.During image processing, the processor of digital camera is searched for this look-up table to determine whether specific pixel is bad pixel and needs to revise.The form of replacement is adopted in this correction usually, utilizes the output valve of the simple average value replacement bad pixel of the good pixel output adjacent with bad pixel.Along with the number of pixel in the transducer becomes increasing, need the memory span of storage look-up table also to increase and consult and calculate then the required time of neighbor mean value also to increase.
Summary of the invention
[0005] after transducer is installed in the device, not during pick up calibration, once to discern bad pixel, but Real time identification bad pixel when in digital camera or other main frames, using transducer.This just can remove from when storage bad pixel table needs very big memory span, thereby reduces the complexity and the cost of this device.Can also remove spent time and expensive calibration process when determining the bad pixel position from.In addition, be not limited to static bad pixel information owing to revise bad pixel, so, can discern automatically and revise and after transducer is installed to main frame, become bad pixel.This just can overcome the transducer aging effect that produces when independent pixel becomes bad pixel in time.When the picture field on projecting transducer is caught by digital camera or other devices, can discern and revise the processing operation of bad pixel, and this operation can be carried out when catching each image.
[0006] according to a concrete feature of the present invention, when exposure sensor is having on intensity and the colored picture field that changes, for example, and when occurring in normal photographs, each pixel of quick testing sensor one by one.Measure each the given test pixel and the output valve of neighbor on every side thereof.Previous treatment step calculates the difference between each output in test pixel output and the neighbor.If these differences comprise positive difference and negative difference, then we think that this pixel is a good pixel, and test operation proceeds to next pixel successively.But, all be positive difference or all be negative difference as if these differences, think that then this given pixel may be a bad pixel, and need to do further test.Suspect that this pixel is because given pixel all becomes clear than neighbor around it or dark is abnormal.The advantage of this method is, determines that independent pixel is can considerably lack the required time of good pixel, and the nearly all normally such result of pixel in the transducer.According to another concrete feature of the present invention, poor by compared pixels output valve and at least one threshold value, test pixel may be that the processing operation of bad pixel is to carry out continuously.If the difference of the output valve of test pixel and its surrounding pixel, determines then that this pixel is a bad pixel greater than predetermined threshold, and utilizes the pixel value that calculates from its surrounding pixel value as view data.Can utilize different threshold values with the neighbor comparison time, it depends on the distance between the neighbor and test pixel on the transducer.We know, when the distance between the pixel increased, the likelihood of transducer epigraph difference increased, when it during with the distance increase of test pixel, the threshold value of application also increases.
[0007], at least once evenly response of light intensity exposure, also can calculate and be used to discern the applied threshold value of bad pixel according to pixel according to another concrete feature of the present invention.The distribution of pixel output valve is derived from this uniform exposure.According to the divergence that output valve distributes, it is included in the percentage of normal good pixel in this transducer, calculates the used threshold value of neighbor output difference.Be used for the compare test pixel and want big away from the used threshold value of pixel value difference.Preferably, utilize same model sensor and imaging optic element, calculated threshold is the data that the sample number according to video camera or other main frames obtains.Perhaps, though complicated more and expensive, can during calibrating installation, calculate the threshold value of each transducer, and/or during using this device calculated threshold automatically, be adapted to any variation of generation when transducer or imaging optic element use in order to make threshold value.
[0008] though the description of each feature of the present invention utilizes the mega pixel transducer to realize, because this transducer generally uses,, the present invention also is applicable to the transducer of less pixel.Other aspects of the present invention, advantage and feature are included in the exemplary embodiments described below, and this description should combine with accompanying drawing.
Description of drawings
[0009] Fig. 1 is a functional-block diagram of representing image processing in the video camera;
[0010] it is green that Fig. 2 A, 2B and 2C represent to can be used on each of particular type of sensor in the video camera shown in Figure 1, the actual arrangement of blue and red sensitive pixels;
[0011] Fig. 3 is a flow chart of handling operation in the video camera shown in Figure 1, is used to discern and revise the interior bad pixel of its optical sensor; With
[0012] Figure 4 and 5 are distribution curves of the video camera inner sensor pixel output valve of exposing under two different numerical value uniform light field, can determine the threshold value of using in the identification bad pixel.
Embodiment
[0013] as shown in Figure 1, video camera comprises imageing sensor 1, and it provides and is proportional to the analog signal that incides light intensity on this transducer.Sensor signal is connected to analog to digital converter (ADC) 2, and it becomes digital signal to the Sensor Analog Relay System conversion of signals.The operation of timing generator 5 control transducers 1 and ADC2.Then, the digital sensor dateout is by the processing level of series connection.Two this, and to handle level be that the bad pixel that is used to discern bad pixel determines 6 and be used to revise the bad pixel correction 8 of identification bad pixel, it be according to and identification bad pixel adjacent pixels output valve.Can comprise or not comprise image processing level 4,7 or 9.The processing level of each series connection can be utilized at different levels in succession of view data after buffer 3 transmission process, can also utilize data that buffer 3 temporary transient storages produce as the part of finishing processing.Video camera also comprises the optical system (not shown), is used for the direct light figure to transducer 1, and also comprises mechanical shutter and controlled diaphragm usually.
[0014] with reference to Fig. 2 A-2C, a kind of very fraction of form pel array is showed separately green (G) respectively in its expression transducer 1, blue (B) and red (R) look sensitive pixels.In the example of describing, the process of identification and correction bad pixel is that every kind of primary colours pixel in these primary colours is carried out respectively.In other words, test green pixel G
TUtilize its adjacent green pixel G1-G8, test blue pixel B
TUtilize its adjacent blue pixel B1-B8, and test red pixel R
TUtilize its adjacent red pixel R1-R8.
[0015] bad pixel described herein determines that method is based on such fact, and imageing sensor and optical element thereof produce the band-limited transducer output signal of telecommunication.Therefore, the existence of bad pixel is pointed out in the position of pixel output valve high frequency variation." just " test pixel is the pixel of its output valve greater than all neighbor output valves, and " bearing " test pixel is the pixel of its output valve less than all neighbor output valves.If its output valve, determines then that it is positive test pixel greater than the one or more positive threshold values of its neighbor; If its output valve, determines then that it is the negative testing pixel less than the one or more negative threshold values of its neighbor.In one embodiment, the neighbor that we consider comprises the pixel of all same colors, the upside of these pixels next-door neighbour test pixel, downside, left side, right side, and the pixel that becomes the diagonal angle vicinity with all four angles of test pixel.
[0016] bad pixel shown in Figure 1 of the flowchart text among Fig. 3 determines 6 and the processing operation finished of bad pixel correction 8, wherein with reference to typical pixel graphics shown in Fig. 2 A-2C.The data normalization step (not shown) that can also comprise standard, and can also change over the sequence of steps among Fig. 3 the order different with it.
[0017] first step 21 is read the digitlization of test pixel and neighbor thereof and/or the output valve after the processing.If green pixel G among Fig. 2 A
TBe test pixel, then read its output valve and the output valve of its contiguous green pixel G1-G4 at least, can also read slightly away from G
TGreen pixel G5-G8 output valve.Similarly, with reference to Fig. 2 B, if Fig. 2 B Smalt pixel B
TBe test pixel, the output valve of then reading it with and the output valve of contiguous blue pixel B1-B4, can also read slightly away from B
TBlue pixel B5-B8 output valve.At last, as if red pixel R among Fig. 2 C
TBe test pixel, the output valve of then reading it with and the output valve of contiguous red pixel R1-R4, can also read adjacent red pixel R5-R8 output valve, pixel R5-R8 and pixel R1-R4 are more slightly away from R
T
[0018] next step in Fig. 3 23, deduct the output valve of neighbor from the output valve of test pixel respectively.For example, if green pixel G
TBe test pixel, then from G
TAt least deduct the output valve of its neighborhood pixels G1-G4 in the output valve, preferably also comprise neighbor G5-G8.This can obtain 4 or 8 differences.For example, utilize the algorithm of all 8 differences, these differences can be expressed as follows:
D1＝G
T-G1；D2＝G
T-G2；D3＝G
T-G3；D4＝G
T-G4；
D5＝G
T-G5；D6＝G
T-G6；D7＝G
T-G7；D8＝G
T-G8。
If blue pixel B
TOr red pixel R
TBe test pixel, then can also form 4 or 8 differences being adjacent pixel.
[0019] then, whether the symbol of 8 difference D1-D8 is all identical to determine them in our research step 25.If they are not identical symbols, then obtained the result.It determines that test pixel is a good pixel, so, in the data file of image, utilize its numerical value, shown in step 27.But if the symbol of 8 difference D1-D8 all is identical, then it points out that test pixel may be a bad pixel.Single test pixel with around it between all pixels difference identical symbol is arranged is unlikely.Inciding light field on the transducer, to have high frequency characteristics on a few directions be possible, for example, some straight lines and turning are arranged in image, but, if on all directions, all show high frequency characteristics, this means that the numerical value of test pixel is greater than or less than the numerical value of all neighbors, be easy to then suspect that this test pixel is a bad pixel.But before announcing that this test pixel is bad pixel, we study the amplitude that it is adjacent difference between the pixel.
[0020] as step 29 among Fig. 3,31,33 and 35 is pointed, difference and one or more threshold value compared, if difference greater than threshold value, announces that then test pixel is a bad pixel.Need to keep positive difference different threshold levels to be arranged with negative difference.So,, then these differences and one or more negative threshold value are compared in step 31 if the difference of determining in step 29 all is a negative value.Similarly, if the difference of determining in step 29 all be on the occasion of, then these differences and one or more positive threshold value are compared in step 33.No matter they are positive difference or negative difference, determine that in step 35 whether the absolute value of each difference is greater than the threshold value of using.If not, announce that then this pixel is a good pixel, so its output valve is as the view data of part, and is pointed as step 27.
[0021] still, if all differences, determine then that this pixel is a bad pixel greater than the threshold value of using, so its output valve should be as the image data of part.The light field that projects object scene on the transducer seldom has this high-resolution situation, and wherein it all neighbor of pixel and its is so different.So bad pixel determines that square frame 6 finishes its function among Fig. 1.Finish the correction of pixel output valve subsequently according to the function of square frame 8.This is the Numerical Implementation that calculates bad pixel according to its neighbor output valve, then, calculated value is replaced actual bad pixel output valve.
[0022] this calculating and replacement operation take place in the step 37 and 39 of Fig. 3.A kind of direct method that is used to calculate the bad pixel value is that at least some adjacent pixel values are got simple arithmetic average.The neighbor of choosing this mean value needn't be the identical neighbor of bad pixel with definite this pixel, but also can be these neighbors.For example, under the situation of green pixel shown in Fig. 2 A, can utilize the most contiguous pixel G1-G4 to average, and save slightly away from neighbor G5-G8, it is in the calculating of bad pixel that these neighbors can be used on this pixel of identification.
[0023] though preferably utilize the adjacent pixel values of same color to be averaged, also can adopt many other math functions that are different from simple average.This function can be linear or nonlinear, not only consider the difference of each pixel and same color neighbor, it and identification other neighbors of same color around the bad pixel are compared, but also adopt other values of multirow in bad pixel and/or the non-bad pixel of different colours or multiple row image pixel.In addition, though abandon the pixel value of bad pixel in this example, and in some cases, it is favourable keeping the output valve of bad pixel and adopt this output valve in the replacement pixel value is calculated.
[0024] from above to the discussion of identification bad pixel as can be seen, if pixel value is greater than or less than its surrounding pixel value predetermined threshold value, then can discerns this pixel and have too high frequecy characteristic.This can regard the pixel spike as in pixel, field quite uniformly.Because the optical element in digital camera or other main frames has low-frequency filter characteristics, should there be this pixel spike, so, determine that causing the pixel of spike is bad pixel.
[0025], preferably utilizes single negative threshold value and single positive threshold value for every kind of color in three kinds of colors.But, if at definite pixel G
TWhether be to utilize a plurality of neighbors far away in the bad pixel, for example, pixel G5-G8, then the threshold value that adopts with these pixel value difference comparisons the time can be different from the threshold value that adopts when neighbour's pixel G1-G4 difference compares.
[0026] bad pixel statistics in a plurality of video cameras by analyze adopting the same type imageing sensor or other devices can find employed threshold value in the bad pixel deterministic process most convenient.In case finish this analysis, the defect property of specific image transducer family can be described, rather than at the video camera of each manufacturing.For every kind of different cameras model that particular type image sensor and optical element are arranged,, just should finish this analysis by test a few representative video camera.This can provide representative statistical data to the distribution of bad image sensor pixel.In case according to determining threshold value, then they are stored in the nonvolatile memory of each video camera or other devices, in the bad pixel identifying, use by this device as the method.
[0027] with reference to Fig. 4, we explain and utilize particular type of sensor and imaging optic element to determine the method for video camera or the positive threshold value of other mounted casts.Get clogged from all light in the transducer of testing apparatus, and collect the data of each pixel output.Preferably carry out repetition for a plurality of identical model equipments.Then, all data are combined into the histogram of number of pixels as the pixel value function, as shown in Figure 4.For each " picture black " histogram that so produces, determine the pixel value Max Value that the maximum number image sensor pixel shows.In addition, choose the imageing sensor output valve and be the pixel value Maiority Value (for example, 99.9%) under Max Value, nearly all pixel value is equal to or greater than Max Value in these pixels.Majority Value deducts the positive threshold value of the numerical value of Max Value as this video camera or other mounted casts.In carrying out process shown in Figure 3, the pixel that pixel value is higher than Majority Value is used as the bad pixel processing.
[0028] percentage of the Majority Value cutoff that when determining positive threshold value, uses shown in being or specific pixel value.Actual Majority Value can be a numerical value quite arbitrarily, outside " positive threshold value " district a plurality of pixels is arranged but preferably be chosen in the histogram shown in Figure 4, and it can make handles the image that operation forms best quality is arranged among Fig. 3.
[0029] utilize similar method to determine negative threshold value, as shown in Figure 5.Transducer in the adequate illumination test video camera, and from pixel output, take out data.This carries out several video cameras, can produce the data of several test same model cameras accumulation and the number of pixels histogram as the pixel value function, as shown in Figure 5.Each almost saturated " white image " histogram for generation like this, determine the pixel value MaxValue that the maximum number image sensor pixel shows, (for example find imageing sensor output valve Majority Value, 99.9%) be pixel on Max Value, nearly all pixel is equal to or less than Max Value in these pixel values.Max Value deducts the negative threshold value of making this model camera numerical value week of Majority Value.
[0030] should be noted that from Fig. 2 A-2C we adopt the transducer of particular type when describing above method.This is a kind of typical type of commercial sensor, and wherein photoelectric detector is along red, and each row pixel green and that blue color filter covers is extended.Comprise red and green sensitive photo-detectors alternately in the delegation, and comprise indigo plant and green sensitive photo-detectors alternately in the next line, photoelectric detector also is placed to like this along each row, and it can provide alternately colored sensitivity along each row.The most of imageing sensors that use in the digital camera utilize such transducer at present, and the number of wherein distributing to green light photodetector position is more than the number of distributing to blue and red light photodetector position.Because human eye is the most responsive to green, revise bad green pixel and do not revise bad blueness and red pixel, picture quality is improved a lot.Except the transducer of type shown here, method of the present invention also is applicable to the transducer that other colour elements are arranged.
[0031] another kind of type commercialization photoelectric detector is at the overlapping a plurality of photoelectric detectors of each optical position.The top detector transmission is red and green, and leaches responsive blueness.Green near this detector transmission below top detector, and leach responsive redness.Bottom sensor is to green sensitive.Green, red is identical with the number of blue sensor.Method described above can be applied to this type sensor similarly.
[0032] though each feature of the present invention is described by exemplary embodiments, should be understood that the scope that the present invention protects is to be limited by appended claims.
Claims (5)
1. method of distinguishing high-quality unit and potential bad element in the photosensitive unit array in the target field illumination that utilize to change light intensity comprises:
Calculate all differences between the output of the output of some independent unit and all adjacent cells,
If all differences of a given independent unit have different symbols, then this given unit of identification be the high-quality unit and
If all differences of described given unit have identical symbol, then this given unit of identification is potential bad element.
2. method that is used for discerning and revising light sensitive pixels array bad pixel comprises:
Guiding changes the light intensity target field to this light sensitive pixels array,
Calculate all differences between the output of the output of some independent pixels and a plurality of neighbors,
If all differences of a given pixel have different symbols, this output that then utilizes this given pixel is as the target field data,
If all differences of given pixel have identical symbol, determine then whether described all differences exceed threshold value,
If described all differences do not exceed threshold value, this output that then utilizes this given pixel as the target field data and
If described all differences exceed threshold value, then from some neighbors, calculate the value of this given pixel at least, and the pixel value of this calculating is used in the target field data.
3. according to the method for claim 2, wherein said threshold value comprises mutually different at least the first value or second value, it depend on described same-sign be on the occasion of or negative value.
4. according to the method for claim 2, wherein said threshold value comprises mutually different at least the first value or second value, and it depends on the distance between given pixel some independent pixels adjacent with it.
5. according to the method for claim 2, wherein said threshold value depend on following both: (a) described same-sign be on the occasion of or negative value, and (b) distance between some independent pixels that given pixel is adjacent with it.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US10/615,277 US7388609B2 (en) | 2003-07-07 | 2003-07-07 | Dynamic identification and correction of defective pixels |
| US10/615,277 | 2003-07-07 | ||
| PCT/US2004/019050 WO2005011261A1 (en) | 2003-07-07 | 2004-06-14 | Dynamic identification and correction of defective pixels |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1981514A true CN1981514A (en) | 2007-06-13 |
| CN1981514B CN1981514B (en) | 2010-04-21 |
Family
ID=33564527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2004800230898A Expired - Fee Related CN1981514B (en) | 2003-07-07 | 2004-06-14 | Method for dynamic identification and correction of defective pixels |
Country Status (5)
| Country | Link |
|---|---|
| US (2) | US7388609B2 (en) |
| EP (1) | EP1645113A1 (en) |
| JP (1) | JP4633717B2 (en) |
| CN (1) | CN1981514B (en) |
| WO (1) | WO2005011261A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101588505B (en) * | 2008-05-21 | 2011-05-11 | 英华达（南京）科技有限公司 | Method for correcting dead pixel on digital camera device and digital camera device with dead pixel automatic-correction function |
| CN104143177A (en) * | 2013-05-07 | 2014-11-12 | 江南大学 | Method for eliminating image interference of line-scan digital camera |
| US9007078B2 (en) | 2012-04-06 | 2015-04-14 | Industrial Technology Research Institute | Pixel array module with self-test function and method thereof |
| CN106358038A (en) * | 2016-09-06 | 2017-01-25 | 上海集成电路研发中心有限公司 | Detecting method of pixel performance |
| CN107395921A (en) * | 2017-08-14 | 2017-11-24 | 深圳怡化电脑股份有限公司 | A kind of difference detecting method and device |
Families Citing this family (92)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3747909B2 (en) * | 2002-12-24 | 2006-02-22 | ソニー株式会社 | Pixel defect detection correction apparatus and pixel defect detection correction method |
| US7388609B2 (en) * | 2003-07-07 | 2008-06-17 | Zoran Corporation | Dynamic identification and correction of defective pixels |
| JP4624054B2 (en) * | 2004-03-29 | 2011-02-02 | 三洋電機株式会社 | Noise reduction device, noise reduction method, and imaging device |
| TWI256847B (en) * | 2004-08-30 | 2006-06-11 | Via Tech Inc | Method and apparatus for dynamically detecting pixel values |
| US7763918B1 (en) | 2004-12-02 | 2010-07-27 | Chen Feng | Image pixel design to enhance the uniformity of intensity distribution on digital image sensors |
| US7450161B1 (en) * | 2004-12-02 | 2008-11-11 | Magnachip Semiconductor Ltd. | System and method to enhance the uniformity of intensity distribution on digital imaging sensors |
| US7564629B1 (en) | 2004-12-02 | 2009-07-21 | Crosstek Capital, LLC | Microlens alignment procedures in CMOS image sensor design |
| KR100652408B1 (en) | 2005-04-27 | 2006-12-01 | 삼성전자주식회사 | Method and apparatus for processing Bayer-pattern color digital image signal |
| WO2007036055A1 (en) * | 2005-09-30 | 2007-04-05 | Simon Fraser University | Methods and apparatus for detecting defects in imaging arrays by image analysis |
| JP4961182B2 (en) * | 2005-10-18 | 2012-06-27 | 株式会社リコー | Noise removal apparatus, noise removal method, noise removal program, and recording medium |
| DE102006050864B4 (en) * | 2005-10-31 | 2015-08-20 | Hitachi Kokusai Electric Inc. | Method and apparatus for correcting defective pixels for a solid state imaging device |
| JP4745816B2 (en) * | 2005-12-20 | 2011-08-10 | 富士通セミコンダクター株式会社 | Image processing circuit and image processing method |
| US8063957B2 (en) * | 2006-03-24 | 2011-11-22 | Qualcomm Incorporated | Method and apparatus for processing bad pixels |
| US9137466B2 (en) | 2006-08-30 | 2015-09-15 | Micron Technology, Inc. | Image sensor defect identification using optical flare |
| US20080055434A1 (en) | 2006-08-31 | 2008-03-06 | Micron Technology, Inc. | Image sensor defect identification using blurring techniques |
| JP4895107B2 (en) * | 2006-09-08 | 2012-03-14 | ソニー株式会社 | Electronic device, information processing method, and program |
| US7800661B2 (en) * | 2006-12-22 | 2010-09-21 | Qualcomm Incorporated | Programmable pattern matching device |
| US8023013B1 (en) | 2007-03-28 | 2011-09-20 | Ambarella, Inc. | Fixed pattern noise correction with compressed gain and offset |
| US8237824B1 (en) * | 2007-03-28 | 2012-08-07 | Ambarella, Inc. | Fixed pattern noise and bad pixel calibration |
| KR101442242B1 (en) * | 2007-12-12 | 2014-09-29 | 삼성전자주식회사 | Defect and noise removal method |
| EP2289235A4 (en) | 2008-05-20 | 2011-12-28 | Pelican Imaging Corp | Capturing and processing of images using monolithic camera array with hetergeneous imagers |
| US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
| US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
| TWI505702B (en) * | 2008-10-03 | 2015-10-21 | Silicon Motion Inc | Camera and compensating method for defective pixels of an image sensor |
| US9544516B2 (en) * | 2008-12-26 | 2017-01-10 | Digimarc Corporation | Method and apparatus for sensor characterization |
| WO2010101945A2 (en) * | 2009-03-02 | 2010-09-10 | Flextronics Ap, Llc | Calibration techniques for camera modules |
| EP2449525A4 (en) * | 2009-07-02 | 2015-06-24 | Scint X Ab | Reduction of noise caused by absorption of x-ray photons in pixelated image sensors |
| US8514491B2 (en) | 2009-11-20 | 2013-08-20 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
| US8660335B2 (en) * | 2010-03-24 | 2014-02-25 | Varian Medical Systems, Inc. | Transient pixel defect detection and correction |
| KR101824672B1 (en) | 2010-05-12 | 2018-02-05 | 포토네이션 케이맨 리미티드 | Architectures for imager arrays and array cameras |
| JP5656561B2 (en) * | 2010-10-28 | 2015-01-21 | キヤノン株式会社 | Image processing apparatus, control method thereof, and control program |
| US8878950B2 (en) | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
| EP2708019B1 (en) | 2011-05-11 | 2019-10-16 | FotoNation Limited | Systems and methods for transmitting and receiving array camera image data |
| US20130265459A1 (en) | 2011-06-28 | 2013-10-10 | Pelican Imaging Corporation | Optical arrangements for use with an array camera |
| EP2726930A4 (en) | 2011-06-28 | 2015-03-04 | Pelican Imaging Corp | Optical arrangements for use with an array camera |
| US20130070060A1 (en) | 2011-09-19 | 2013-03-21 | Pelican Imaging Corporation | Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion |
| WO2013049699A1 (en) | 2011-09-28 | 2013-04-04 | Pelican Imaging Corporation | Systems and methods for encoding and decoding light field image files |
| US9412206B2 (en) | 2012-02-21 | 2016-08-09 | Pelican Imaging Corporation | Systems and methods for the manipulation of captured light field image data |
| US9210392B2 (en) | 2012-05-01 | 2015-12-08 | Pelican Imaging Coporation | Camera modules patterned with pi filter groups |
| CN104508681B (en) | 2012-06-28 | 2018-10-30 | Fotonation开曼有限公司 | For detecting defective camera array, optical device array and the system and method for sensor |
| US20140002674A1 (en) | 2012-06-30 | 2014-01-02 | Pelican Imaging Corporation | Systems and Methods for Manufacturing Camera Modules Using Active Alignment of Lens Stack Arrays and Sensors |
| US8619082B1 (en) | 2012-08-21 | 2013-12-31 | Pelican Imaging Corporation | Systems and methods for parallax detection and correction in images captured using array cameras that contain occlusions using subsets of images to perform depth estimation |
| US20140055632A1 (en) | 2012-08-23 | 2014-02-27 | Pelican Imaging Corporation | Feature based high resolution motion estimation from low resolution images captured using an array source |
| WO2014043641A1 (en) | 2012-09-14 | 2014-03-20 | Pelican Imaging Corporation | Systems and methods for correcting user identified artifacts in light field images |
| US20140092281A1 (en) | 2012-09-28 | 2014-04-03 | Pelican Imaging Corporation | Generating Images from Light Fields Utilizing Virtual Viewpoints |
| TWI495862B (en) * | 2012-10-04 | 2015-08-11 | Pixart Imaging Inc | Method of testing image sensor and realted apparatus thereof |
| JP6071419B2 (en) * | 2012-10-25 | 2017-02-01 | キヤノン株式会社 | Image processing apparatus and image processing method |
| US9143711B2 (en) | 2012-11-13 | 2015-09-22 | Pelican Imaging Corporation | Systems and methods for array camera focal plane control |
| EP2765765A1 (en) | 2013-02-08 | 2014-08-13 | Harvest Imaging bvba | Digital camera and method of capturing an image |
| WO2014130849A1 (en) | 2013-02-21 | 2014-08-28 | Pelican Imaging Corporation | Generating compressed light field representation data |
| US9253380B2 (en) | 2013-02-24 | 2016-02-02 | Pelican Imaging Corporation | Thin form factor computational array cameras and modular array cameras |
| US9638883B1 (en) | 2013-03-04 | 2017-05-02 | Fotonation Cayman Limited | Passive alignment of array camera modules constructed from lens stack arrays and sensors based upon alignment information obtained during manufacture of array camera modules using an active alignment process |
| WO2014138695A1 (en) | 2013-03-08 | 2014-09-12 | Pelican Imaging Corporation | Systems and methods for measuring scene information while capturing images using array cameras |
| US8866912B2 (en) | 2013-03-10 | 2014-10-21 | Pelican Imaging Corporation | System and methods for calibration of an array camera using a single captured image |
| US9519972B2 (en) | 2013-03-13 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies |
| US9124831B2 (en) | 2013-03-13 | 2015-09-01 | Pelican Imaging Corporation | System and methods for calibration of an array camera |
| WO2014164909A1 (en) | 2013-03-13 | 2014-10-09 | Pelican Imaging Corporation | Array camera architecture implementing quantum film sensors |
| US9106784B2 (en) | 2013-03-13 | 2015-08-11 | Pelican Imaging Corporation | Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing |
| WO2014159779A1 (en) | 2013-03-14 | 2014-10-02 | Pelican Imaging Corporation | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
| WO2014153098A1 (en) | 2013-03-14 | 2014-09-25 | Pelican Imaging Corporation | Photmetric normalization in array cameras |
| US9445003B1 (en) | 2013-03-15 | 2016-09-13 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
| US9633442B2 (en) | 2013-03-15 | 2017-04-25 | Fotonation Cayman Limited | Array cameras including an array camera module augmented with a separate camera |
| US20140304110A1 (en) * | 2013-03-15 | 2014-10-09 | W.W. Grainger, Inc. | Procurement process utilizing a light sensor |
| WO2014145856A1 (en) | 2013-03-15 | 2014-09-18 | Pelican Imaging Corporation | Systems and methods for stereo imaging with camera arrays |
| US10122993B2 (en) | 2013-03-15 | 2018-11-06 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
| US9497429B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Extended color processing on pelican array cameras |
| WO2014150856A1 (en) | 2013-03-15 | 2014-09-25 | Pelican Imaging Corporation | Array camera implementing quantum dot color filters |
| US9898856B2 (en) | 2013-09-27 | 2018-02-20 | Fotonation Cayman Limited | Systems and methods for depth-assisted perspective distortion correction |
| US9264592B2 (en) | 2013-11-07 | 2016-02-16 | Pelican Imaging Corporation | Array camera modules incorporating independently aligned lens stacks |
| WO2015074078A1 (en) | 2013-11-18 | 2015-05-21 | Pelican Imaging Corporation | Estimating depth from projected texture using camera arrays |
| US9456134B2 (en) | 2013-11-26 | 2016-09-27 | Pelican Imaging Corporation | Array camera configurations incorporating constituent array cameras and constituent cameras |
| WO2015134996A1 (en) | 2014-03-07 | 2015-09-11 | Pelican Imaging Corporation | System and methods for depth regularization and semiautomatic interactive matting using rgb-d images |
| JP6291940B2 (en) * | 2014-03-20 | 2018-03-14 | 株式会社ソシオネクスト | Defective pixel correction device, imaging device, and defective pixel correction method |
| US9247117B2 (en) | 2014-04-07 | 2016-01-26 | Pelican Imaging Corporation | Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array |
| US9521319B2 (en) | 2014-06-18 | 2016-12-13 | Pelican Imaging Corporation | Array cameras and array camera modules including spectral filters disposed outside of a constituent image sensor |
| WO2016054089A1 (en) | 2014-09-29 | 2016-04-07 | Pelican Imaging Corporation | Systems and methods for dynamic calibration of array cameras |
| US9885784B2 (en) * | 2014-12-08 | 2018-02-06 | The United States Of America As Represented By The Secretary Of The Army | Method and system for detecting man-made objects using polarimetric, synthetic aperture radar imagery |
| US9942474B2 (en) | 2015-04-17 | 2018-04-10 | Fotonation Cayman Limited | Systems and methods for performing high speed video capture and depth estimation using array cameras |
| US10482618B2 (en) | 2017-08-21 | 2019-11-19 | Fotonation Limited | Systems and methods for hybrid depth regularization |
| CN114600165A (en) | 2019-09-17 | 2022-06-07 | 波士顿偏振测定公司 | System and method for surface modeling using polarization cues |
| EP4042101A4 (en) | 2019-10-07 | 2023-11-22 | Boston Polarimetrics, Inc. | Systems and methods for surface normals sensing with polarization |
| KR102558903B1 (en) | 2019-11-30 | 2023-07-24 | 보스턴 폴라리메트릭스, 인크. | System and Method for Segmenting Transparent Objects Using Polarized Signals |
| JP7462769B2 (en) | 2020-01-29 | 2024-04-05 | イントリンジック イノベーション エルエルシー | System and method for characterizing an object pose detection and measurement system - Patents.com |
| CN115428028A (en) | 2020-01-30 | 2022-12-02 | 因思创新有限责任公司 | System and method for synthesizing data for training statistical models in different imaging modalities including polarized images |
| WO2021243088A1 (en) | 2020-05-27 | 2021-12-02 | Boston Polarimetrics, Inc. | Multi-aperture polarization optical systems using beam splitters |
| KR20220043611A (en) * | 2020-09-29 | 2022-04-05 | 엘지전자 주식회사 | Dishwasher and method for determining a camera malfunction thereof |
| US12069227B2 (en) | 2021-03-10 | 2024-08-20 | Intrinsic Innovation Llc | Multi-modal and multi-spectral stereo camera arrays |
| US12020455B2 (en) | 2021-03-10 | 2024-06-25 | Intrinsic Innovation Llc | Systems and methods for high dynamic range image reconstruction |
| US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
| US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
| US12067746B2 (en) | 2021-05-07 | 2024-08-20 | Intrinsic Innovation Llc | Systems and methods for using computer vision to pick up small objects |
| US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
Family Cites Families (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS63245199A (en) * | 1987-03-31 | 1988-10-12 | Toshiba Corp | Method and device for image sensor inspection |
| EP0496573B1 (en) | 1991-01-24 | 1995-12-20 | Matsushita Electric Industrial Co., Ltd. | Pixel defect removing circuit for solid-state image pickup device |
| JP3042159B2 (en) | 1992-04-10 | 2000-05-15 | ソニー株式会社 | CCD element defective pixel correction circuit |
| JP3227815B2 (en) * | 1992-07-06 | 2001-11-12 | ソニー株式会社 | Solid-state imaging device |
| JP3125124B2 (en) * | 1994-06-06 | 2001-01-15 | 松下電器産業株式会社 | Defective pixel damage correction circuit |
| JP3830574B2 (en) | 1996-04-10 | 2006-10-04 | 松下電器産業株式会社 | Defect correction device and solid-state imaging device using the same |
| US5982946A (en) | 1996-09-20 | 1999-11-09 | Dainippon Screen Mfg. Co., Ltd. | Method of identifying defective pixels in digital images, and method of correcting the defective pixels, and apparatus and recording media therefor |
| JP3351704B2 (en) | 1997-04-09 | 2002-12-03 | ペンタックス株式会社 | Image signal correction device |
| JP3544304B2 (en) * | 1998-08-04 | 2004-07-21 | 日本ビクター株式会社 | Pixel defect correction device and pixel defect correction method |
| GB9825086D0 (en) | 1998-11-17 | 1999-01-13 | Vision Group Plc | Defect correction in electronic imaging systems |
| JP2000285229A (en) * | 1999-03-15 | 2000-10-13 | Texas Instr Inc <Ti> | Defective pixel filtering for digital imager |
| ATE247885T1 (en) | 1999-03-15 | 2003-09-15 | Texas Instruments Inc | FILTERING DEFECTIVE IMAGE ELEMENTS IN DIGITAL IMAGERS |
| JP2001086517A (en) * | 1999-09-13 | 2001-03-30 | Toshiba Corp | Pixel defect detector |
| KR100362178B1 (en) | 1999-12-30 | 2002-11-23 | 주식회사 하이닉스반도체 | Apparatus for real-time detecting and correcting defect pixels in image sensor |
| US6724945B1 (en) | 2000-05-24 | 2004-04-20 | Hewlett-Packard Development Company, L.P. | Correcting defect pixels in a digital image |
| JP2002135801A (en) * | 2000-10-25 | 2002-05-10 | Sony Corp | Image processor |
| JP4108278B2 (en) * | 2001-01-24 | 2008-06-25 | シャープ株式会社 | Automatic defect detection and correction apparatus for solid-state image sensor and imaging apparatus using the same |
| US20030030738A1 (en) | 2001-05-22 | 2003-02-13 | Clynes Steven Derrick | On-chip 2D adjustable defective pixel filtering for CMOS imagers |
| JP4583680B2 (en) * | 2001-09-28 | 2010-11-17 | パナソニック株式会社 | Solid-state imaging device |
| JP4166974B2 (en) * | 2001-11-22 | 2008-10-15 | 松下電器産業株式会社 | Pixel scratch detection / correction device |
| US7202894B2 (en) | 2002-06-04 | 2007-04-10 | Micron Technology, Inc. | Method and apparatus for real time identification and correction of pixel defects for image sensor arrays |
| US7283164B2 (en) * | 2002-09-18 | 2007-10-16 | Micron Technology, Inc. | Method for detecting and correcting defective pixels in a digital image sensor |
| US7034874B1 (en) * | 2003-03-17 | 2006-04-25 | Biomorphic Vlsi, Inc | Automatic bad pixel correction in image sensors |
| US7388609B2 (en) | 2003-07-07 | 2008-06-17 | Zoran Corporation | Dynamic identification and correction of defective pixels |
-
2003
- 2003-07-07 US US10/615,277 patent/US7388609B2/en not_active Expired - Fee Related
-
2004
- 2004-06-14 EP EP20040755309 patent/EP1645113A1/en not_active Withdrawn
- 2004-06-14 CN CN2004800230898A patent/CN1981514B/en not_active Expired - Fee Related
- 2004-06-14 WO PCT/US2004/019050 patent/WO2005011261A1/en active Application Filing
- 2004-06-14 JP JP2006518648A patent/JP4633717B2/en not_active Expired - Fee Related
-
2008
- 2008-02-26 US US12/037,699 patent/US8098304B2/en not_active Expired - Lifetime
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101588505B (en) * | 2008-05-21 | 2011-05-11 | 英华达（南京）科技有限公司 | Method for correcting dead pixel on digital camera device and digital camera device with dead pixel automatic-correction function |
| US9007078B2 (en) | 2012-04-06 | 2015-04-14 | Industrial Technology Research Institute | Pixel array module with self-test function and method thereof |
| CN104143177A (en) * | 2013-05-07 | 2014-11-12 | 江南大学 | Method for eliminating image interference of line-scan digital camera |
| CN106358038A (en) * | 2016-09-06 | 2017-01-25 | 上海集成电路研发中心有限公司 | Detecting method of pixel performance |
| CN106358038B (en) * | 2016-09-06 | 2018-05-29 | 上海集成电路研发中心有限公司 | A kind of detection method of pixel performance |
| CN107395921A (en) * | 2017-08-14 | 2017-11-24 | 深圳怡化电脑股份有限公司 | A kind of difference detecting method and device |
| CN107395921B (en) * | 2017-08-14 | 2020-03-27 | 深圳怡化电脑股份有限公司 | Difference detection method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| JP2007527646A (en) | 2007-09-27 |
| US20080143856A1 (en) | 2008-06-19 |
| US8098304B2 (en) | 2012-01-17 |
| EP1645113A1 (en) | 2006-04-12 |
| US7388609B2 (en) | 2008-06-17 |
| WO2005011261A1 (en) | 2005-02-03 |
| US20050010621A1 (en) | 2005-01-13 |
| CN1981514B (en) | 2010-04-21 |
| JP4633717B2 (en) | 2011-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1981514B (en) | Method for dynamic identification and correction of defective pixels | |
| US6965395B1 (en) | Methods and systems for detecting defective imaging pixels and pixel values | |
| US7283164B2 (en) | Method for detecting and correcting defective pixels in a digital image sensor | |
| EP0997845B1 (en) | A test efficient method of classifying image quality of an optical sensor using three categories of pixels | |
| US8154632B2 (en) | Detection of defective pixels in an image sensor | |
| US20030179418A1 (en) | Producing a defective pixel map from defective cluster pixels in an area array image sensor | |
| US20150271423A1 (en) | Image-pickup apparatus and method of detecting defective pixel thereof | |
| US7286171B2 (en) | Apparatus and method for concealing defective pixels in image sensors having test mode | |
| JPH11252464A (en) | System and method for correcting pixel for cmos imager | |
| CN107613229A (en) | A kind of dead pixels of image sensor surveys means for correcting and method | |
| CN110740314B (en) | Method and system for correcting defective pixel of color line array camera | |
| JP4579570B2 (en) | Shading method | |
| JP3884952B2 (en) | Imaging device | |
| JPH06284346A (en) | Automatic defect detector for solid-state image pickup device | |
| CN102625055B (en) | Digital imaging device and image processing method thereof | |
| US20100231763A1 (en) | Defective pixel detector for a digital video camera and associated methods | |
| US20100201850A1 (en) | Noise reduction methods and systems for imaging devices | |
| US7821539B2 (en) | Method and apparatus for inspecting solid-state image pickup device | |
| CN101335827B (en) | Image defect correcting system using oriented detection method | |
| TWI528817B (en) | Image capture devices and image processing method thereof | |
| US20240196109A1 (en) | Global shutter sensor with parasitic light sensitivity compoensation | |
| Zhang et al. | An Approach to detect defective CCD in digital cameras | |
| JP2672735B2 (en) | Inspection method for solid-state image sensor | |
| CN115760776A (en) | Dead pixel detection method and device, computer equipment and storage medium | |
| CN114339210A (en) | Camera module multithreading detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| ASS | Succession or assignment of patent right |
Owner name: GAOTONG TECHNOLOGY INC. Free format text: FORMER OWNER: ZORAN CORP. Effective date: 20150505 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20150505 Address after: American California Patentee after: QUALCOMM TECHNOLOGIES, INC. Address before: American California Patentee before: Zoran Corp. |
|
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20100421 Termination date: 20160614 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |