JP2012124795A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system in which information is converted in an information form that can properly be corrected in a subsequent stage correction unit.SOLUTION: In the image processing system having information and means instructing correction of a pixel in a specified part, information (defective pixel information) instructing the correction of the pixel has information (positional information) on a position in a photographed image and information (attribute information) on a type of the correction in the pixel (defective pixel) requiring correction processing when it is processed as photographed image information. The system includes: non-volatile storage means storing the defective pixel information; temporary storage means which temporarily stores the defective pixel information; a control unit of the temporary storage means; and a decoding unit receiving the defective pixel information from the temporary storage means and analyzing content of information. The system also includes means overlapping a decoding result of the decoding unit on image information being a processing object as a status at every attribute information and/or storing the decoding result in the temporary storage means as information at every attribute information such as {0, 1} information.

Description

  The present invention relates to an image processing apparatus that inputs a captured image / video by an imaging means and processes it.

  Conventionally, as a specific example of an image processing apparatus, there is an imaging apparatus such as a digital camera. The image pickup apparatus such as a digital camera is provided with a predetermined image pickup means, and the electric signal photoelectrically converted therein is digitized, subjected to image processing and stored.

  In a digital camera, photoelectric conversion units such as a CCD and a CMOS sensor are arranged at high density in a matrix to form a pixel array, and an object is imaged. Among these pixels, there is a pixel that does not operate normally with a certain probability (hereinafter referred to as a defective pixel), which becomes point-like noise and causes image quality degradation.

  The existence of a defective pixel is already known when the sensor is incorporated into an imaging device such as a digital camera, and the defective pixel always requires correction processing of the defective pixel (hereinafter referred to as a process flaw), or the sensitivity of the sensor ( Due to (abnormal sensitivity), if it is less than (or more than) a certain threshold value, it will not be scratched, and there are those that use input values for image information (sensitivity scratch). Some input ranges may be used as image information by performing offset value addition / subtraction processing. (Process flaws can also be said to be low in sensitivity flaws (normally flaws)).

  In recent years, means for performing focus detection processing by providing a distance measuring sensor in a pixel array of an image sensor has been proposed. The focus detection sensor pixel realizes a desired pupil characteristic with an aperture shape different from that of the normal pixel, for example, as shown in S1 and S2 in FIG. 7, and aims to realize focus detection by phase difference detection. Yes.

  Patent Document 1 describes a solid-state imaging device, a control method thereof, an imaging device, a basic arrangement of photoelectric conversion cells, and a storage medium. FIG. 8 shows an arrangement of photoelectric conversion cells in the document 1. In a solid-state imaging device in which photoelectric conversion cells for converting an optical image formed by an optical system into an electrical signal are two-dimensionally arranged, at least some of the photoelectric conversion cell groups S1 and S2 receive an image signal. It is configured to output a signal other than for forming.

  If the pixel data of these distance measuring sensors is subjected to image processing as it is for normal pixels, the quality of the resulting image is degraded. Even for such pixel data that does not contribute to the photographing information of the subject, it is necessary to perform correction processing equivalent to the defective pixel at the time of image processing.

  Pixels that need correction processing in the pixel array of the sensor, such as pixels that require correction processing for defective pixels such as process scratches and sensitivity scratches, pixels that are used for focus detection and the like and require correction processing equivalent to defective pixels, etc. In the case where the absolute position information is stored in the apparatus as it is, a large amount of storage means is required.

Therefore, as a method for specifying the positions of these correction-needed pixels, a method of storing in the storage means as relative position information having a difference between the previous correction position and the next correction position has been proposed.
For example, Patent Document 2 proposes that the relative position information and information indicating the necessity of correction be associated with each other and stored in a storage unit as defective pixel information.

  Conventionally, the detection position of the defective pixel correction is detected while referring to the position information in the defective pixel information stored in the storage unit as described above and the position count value of the imaging data input from the imaging unit. Is going. Patent Document 3 proposes a method for controlling a storage unit for defective pixel information and a temporary buffer for use in defective pixel compensation. This is means for supplying the defective pixel information to be consumed for the imaging data to be subjected to flaw correction so as not to lower the throughput.

  FIG. 4 is an example of a main block diagram of defective pixel correction using a temporary buffer for use in the defective pixel. The defective pixel information 402 is read from the volatile storage means 401 such as a DRAM and stored in the temporary buffer 404. FIG. 3 is a block diagram of a main part of a flaw correction circuit having a distance counter 408 and a grade determination means 407 for storing and determining position information in the defective pixel information 402 and whether or not correction is performed.

  As described above, when the position information of the defective pixel is included as relative distance information with respect to the previous defect, the distance to the next wound is counted in the distance counter 408 while decrementing effective image data each time it is received. The received data when the decrement value becomes 0 is a defect correction target. When the decrement value reaches 0, the distance counter 408 updates the output buffer instruction to the read / write controller 405 (of the buffer 404) in FIG. In response, new data is sent.

  In addition, the controller 405 requests the volatile storage unit control unit 403 to send new data in the defective pixel information 402. The distance counter 408 sends a status indicating that the image data to be processed is defective (1 value in the figure is defective) to the logical product 411. The defect information analysis means 406 extracts the position information (distance information), defect grade information, and incidental information from the information for each pixel of the defect pixel information 402, if any.

  In FIG. 4, reference numeral 407 denotes grade determination means for analyzing the grade information. Each defective pixel has a different defective state. Although the above-mentioned process scratches are always subject to correction, depending on the defective state, the process scratches are of a level that can be used as normal pixels without being scratched depending on the shooting conditions such as ISO sensitivity, second setting, temperature state, etc. Is also present. For such defective pixels, several grades are provided as information, and the grade at the time of this processing is set in the grade judgment means as a correction condition on the circuit side to select whether or not to make a scratch It will be set.

  If the result of the determination by the grade determination means is that the processed pixel is defective, the grade determination means outputs a value of 1. The defect information analysis unit 406 also extracts incidental information for the defective pixel. For example, in FIG. 4, it is assumed that pixel information having threshold information of defective pixels exists as the incidental information. In this case, when the defective pixel analyzing unit 406 extracts the threshold information, the threshold value is output to the comparing unit 409 in FIG. The comparator 409 in FIG. 4 compares the value with the current image data (or / and coincidence comparison). If the condition is satisfied, the pixel is regarded as a defective pixel with respect to the logical sum 410. 1 value is output.

  When the output of the logical sum 410 is 1 value, a defective pixel is detected. The value is output as it is to the logical product 411. In the logical product 411, when the values of the image data valid status, the distance counter 408 result, and the logical sum 410 result are all one, the flaw correction process is executed. 1 value is output as the status for obtaining. As a result, the selector 412 selects the previous image data value as the defective pixel correction value.

  In FIG. 4, reference numeral 413 denotes a register that holds the status of the image data valid status. In this figure, it is assumed that the image data and the image data valid status are transferred in synchronization with the same clock. The register 413 is realized by a flip-flop circuit that holds the operation by the rising edge (or falling edge) of the same clock. The clock signal for synchronous transfer and the reset signal of the register are not shown in the figure because they are complicated.

  414 in FIG. 4 is a register that temporarily holds image data, and is realized by a flip-flop circuit configuration like the register 413. The register 414 holds the output result of the selector 412. The selector 412 outputs the previous image data value when the logical product 411 result is 1 and the current pixel of interest is a defective pixel, and outputs the pixel value of interest as it is otherwise. In FIG. 4, as a defective pixel in general, pre-interpolation for replacing the defective pixel with the previous pixel value is illustrated.

  Further, when an image data transmission stop request is received from a subsequent circuit of the pre-interpolation circuit unit, the state of the registers 413 and 414 is temporarily held, and further, image data transmission is stopped with respect to the previous circuit. A circuit that immediately outputs the status of the request is provided.

  Japanese Patent Application Laid-Open No. 2004-228688 provides a means for generating a status by decoding the presence / absence of defective pixel correction processing in advance and sending the status in synchronization with input of imaging data to the correction processing unit. As a result, a high rate is realized with a small circuit configuration.

JP 2000-156823 A JP 2008-187402 A JP 2000-023051 A JP 2003-259220 A

  Since the number of defective pixels to be corrected is not small, the defective pixel information should be retained as the relative position information. In addition, it is more efficient to express the relative position if the information of each of the process scratch, the sensitivity scratch, and the focus detection sensor is held together.

  However, as described above, in the case where the defective pixel information is stored as the relative position information, and information on each of the process scratches, sensitivity scratches, and focus detection sensors is held together, each processing unit is individually provided. Holding the decoding unit for the defective pixel information hinders simplification of the apparatus.

  The process scratch is always subject to correction, and it is only necessary to know its position information. Therefore, it is possible to develop a scratch processing image in advance for each line of photographed image data or for each frame. For example, it can be developed into 0/1 1-bit data such as {0: normal processing, 1: flaw correction processing}. If it can be processed in this way, it is advantageous for increasing the rate of input data from the imaging means.

  However, for the sensitivity flaw, it is necessary to determine whether the input data from the imaging means is a flaw, and pre-processing like the process flaw cannot be performed. Further, the focus detection sensor pixel or the like as the correction target is always a correction target in the normal pixel as in the above-described process scratch, but when the focusing operation is controlled, the pixel data is separately extracted and the focusing processing calculation means Need to be sent out. In this way, there are cases where the processing time is different even if the pixel is a defect correction target pixel in the same apparatus.

  The sensor pixel for focus detection for the phase difference method is not a defective pixel. However, in order to create left and right pupils, the aperture shape, microlens position, etc. are devised, or the color filter may be different from normal pixels to obtain light, so as normal image processing, A case where correction processing similar to that for defective pixels needs to be performed can also be assumed. In this proposal, the pixel information including the sensor pixel information for focus detection is called defective pixel information.

  It is a problem to cope with different processing times by a decoder for defective pixel information at one location under the above-mentioned retention condition of defective pixel information.

  The present invention provides a means for adding status indicating whether or not defect correction is necessary to the imaged data, and means for appropriately performing a correction process in response to the status as necessary in each subsequent image processing section. By adding the above, for each of the imaging data, it is possible to add a correction necessity status as a result of decoding the defective image information having the same information as described above, and to execute independent correction processing in each circuit. .

  Further, according to the present invention, in the process of decoding the defective pixel information having the same information, the defective pixel position information that is always corrected is separated from the defective pixel position information that is corrected in view of the input data state. Thus, an apparatus configuration is provided that can be appropriately sent to each subsequent image processing unit.

  According to the present invention, even if defective pixel information having relative position information with the information as one is used for each diversified correction target, the independent defective pixel position is indicated to each image processing. Since the defective pixel position detection (decoding) process is completed once, the circuit can be simplified.

  In addition, since it is possible to separate the timing of the defective pixel position detection (decoding) and the correction processing, it is possible to suppress a decrease in the throughput of the input image processing related to the call of the ancestor defective position information.

The present invention, main part block diagram Embodiment 1 of the present invention, main part block diagram Embodiment 2 of the present invention, main part block diagram Conventional device configuration, defective pixel information processing unit Conventional device configuration, principal block diagram Embodiment 1 of the present invention, main part of defective pixel information processing unit shown Sensor pixel for focus detection Sensor pixel (photoelectric conversion cell) array for focus detection

[Example 1]
In the embodiment of the present invention, an example in which a status indicating whether defective pixel correction is necessary is added to each of the imaging data will be presented. Before illustrating the effect of the present invention, the situation when the present invention is not implemented will be illustrated with reference to FIG.

  FIG. 5 is an illustration of a conventional configuration to which the embodiment of the present invention is not applied. In FIG. 5, reference numeral 500 denotes the defective pixel information. Here, it is assumed that each attribute requiring processing 1, processing 2, and processing 3 and defective pixel position information are held. In addition, as described above, the position information holds relative position information as distance information between the defective pixel of interest and the previous defective pixel. The distance may be defined by counting pixels in the sensor scanning order (imaging data arrangement order).

  In FIG. 5, the defective pixel information 500 is stored in a non-volatile storage means (not shown) in the apparatus, but in use, the non-volatile storage means (501 in FIG. ) For temporary storage. The nonvolatile storage means 501 is controlled by a controller 502 in FIG.

  When photographing a subject, the image pickup unit 503 in FIG. 5 outputs optical information by converting it into electrical information by photoelectric conversion. A CDS (correlated double sampling) / AGC (Auto Gain Control) 504 in FIG. 5 is a processing unit for improving the S / N of an analog electric signal. The imaged electrical signal is converted into digital imaged data by the A / D conversion means 505 in FIG. The FIFO 506 in FIG. 5 is a buffer for timing delay buffering in the subsequent data processing.

  In FIG. 5, reference numerals 507, 508, and 509 are block diagrams of modules indicating the processing 1, the processing 2, and the processing 3, respectively. However, the compensation processing of the sensor, the image processing such as synchronization / noise suppression, and the combination are performed. This is related to digital camera data processing, such as calculation processing for focus detection, and the processing order is arbitrary in the selection range by using a mutually connected selector between modules and internal submodules. I can do it. The illustration is merely an example.

  For example, the module 507 is related to the sensor compensation process, and processes only the process scratch. In FIG. 5, reference numeral 5071 denotes a decoding unit for extracting the defective pixel position, which extracts the position information of the process scratch from the defect information stored in the buffer 50711 in FIG. Pre-interpolation that replaces the previous data as information may be performed.

  In module 507 in FIG. 5, there are several submodules. For example, in FIG. 5, 5072 is a part for performing offset subtraction processing, 5073 is a part for performing shading correction, 5074 is a digital gain calculation unit, and the like.

  For example, the module 508 in FIG. 5 processes a sensor pixel for focus detection. In this case, it is necessary to extract data necessary for focusing and to pass the other pixels to subsequent processing without changing them as normal pixels for an image. In FIG. 5, reference numeral 5081 denotes a decoding unit that extracts the position of the focus detection sensor, and the buffer 50811 stores defective pixel information similar to that of the buffer 50711.

  For example, in the focusing process, the compensation calculation unit 5082 for the special sensor pixel, the calculation unit 5083 for reproducing the pupil by compensating for the effect of information deterioration due to the image height, and the phase difference by mapping the pixel value as an image. There are a correlation calculation unit 5084 and the like. In addition, there are filter processes for increasing accuracy, and the description thereof is omitted here.

  As the focusing process, the result of the module 508 is returned to the volatile storage unit 501 so that the CPU (not shown) can perform subsequent processes (lens driving, etc.). When executing image processing (development processing), the module 508 sends the result of correcting the focus detection sensor position data for image processing to the module 509 in FIG.

  For example, the module 509 in FIG. 5 may exist as an image processing unit, take white balance at 5092, perform synchronization processing at 5093, and perform false color suppression processing at 5094. Sensor compensation is captured as a single element for each pixel, but image processing requires horizontal and vertical (or every other frame in the time axis direction) adjacent reference pixels, All defective pixels need to be corrected first.

  The module 509 in FIG. 5 receives the captured image data in a state where the process flaw correction / focus detection sensor pixel correction has been completed, so the decoding unit 5091 in FIG. Then, the threshold information obtained in the same manner is compared with the received imaging data value, and if it is judged as a scratch, a correction process is performed. Interpolated pixel data may be generated from horizontal and vertical pixel data as long as the line buffer for the reference pixel of the image processing unit can be used.

  In FIG. 5, reference numeral 510 denotes a data management unit related to data exchange between the volatile storage unit 501 and the processing modules 507, 508, 509,. Here, the connection destination will be described as a configuration targeting the processing modules 507, 508, and 509.

  5101 in the data management unit 510 is a buffer for sending data read from the volatile storage unit 501 to the buffer 50711, and has a FIFO configuration. When the decoding unit 5071 issues a request instruction to the status controller 5102 in the data management unit 510 to fill the buffer 50711, the status controller 5102 sends the arbiter 5100 in FIG. Submit a data request.

  If there is no request from another status controller, the arbiter 5100 issues a data request to the controller 502 of the volatile storage means. Upon receiving the request, the controller 502 sends an instruction to read next data from the volatile storage unit 501 to the decoding unit 5071 of the defective pixel information 500. The read data is stored in the FIFO 5101 in the data management unit 510. Although there is a difference in direction between reading from the volatile storage unit 501 and writing of processing results to the volatile storage unit 501, the same applies to other FIFOs and status controllers in the data management unit 510. Perform the process.

  FIG. 2 is a block diagram showing the main part of the first embodiment of the present invention. In FIG. 2, reference numeral 200 denotes defective pixel information including a defective pixel position at a relative position, and is developed in the volatile storage unit 201 as described above. In FIG. 2, reference numeral 202 denotes a controller of the volatile storage means. The operations 203 to 206 in FIG. 2 are the same as the operations 503 to 506 in FIG.

  Here, the modules 207, 208, and 209 in FIG. 2 are defined similarly to the modules 507, 508, and 509 in FIG. The decoding unit 2071 in FIG. 2 decodes all the contents of the defective pixel information 200. The defective pixel information in this embodiment includes defective pixel information of the process scratch, the focus detection sensor, and the sensitivity scratch. Of course, when the present invention is actually applied, other types of exceptional pixel information may be superimposed.

  In FIG. 2, the output signal of the decoding unit 2071 is (m + n) bits. When the imaging data input to the module 207 is m bits, n bits are merged. For example, if the input of imaging data is 12-bit data and the merge amount is 3 bits, the output data of the decoding unit 2071 is 15 bits.

  The 3 bits for the merge are assigned sensor positions for process scratches, sensitivity scratches, and focus detection, respectively. For example, you may assign from a low-order bit in this description order. The 3 bits may be the least significant bit side in 15 bits, or may be the most significant bit side. In carrying out the present invention, the attachment position is not limited.

  Each of the 3 bits plays a role as a status signal indicating a defective pixel position. For example, in the least significant bit, a normal pixel state may be indicated when the value is 0, and a process scratch state may be indicated when the value is 1 value. For example, all the sub-modules (2072, 2073, 2074) in the module 207 react only to the process scratch of the least significant bit. For example, if the least significant bit is 1 during integration processing, pre-interpolation is performed, and so on.

  For example, if 208 in FIG. 2 is a processing unit of the focus detection sensor, the module refers to the state of the second bit when viewed from the lowest order. When the bit is 1, it is the position of the focus detection sensor.

  For example, reference numeral 209 in FIG. 2 denotes an image processing unit. As described above, it is necessary to correct all defective pixels here. Since image processing includes digital filter processing, defective pixel correction must be performed before image processing. For example, it may be executed at the position 2092 in FIG. In that case, the sub-module confirms the one-value state in all three bits of the status, and appropriately performs defective pixel correction for each. The process scratches and focus detection sensor pixels are all corrected as appropriate, and the sensitivity scratches are corrected when they are determined to be scratches in view of the value of input data.

  In this manner, the decoding unit 2071 is configured to add the status signal indicating the defect correction position to the imaging data, so that the decoding units in the processing units 208 and 209 (5081 and 5091 in FIG. 5). Equivalent). Also, the configuration of the FIFO, status controller, etc. in the data management unit between the volatile storage unit 201 and the image processing units 207, 208, and 209 located at 210 in FIG. 2 can be reduced.

  FIG. 6 is a diagram illustrating the main part of the defective pixel information processing unit in the image processing unit 207 presented earlier. For simplification and clarification of the description, reference numerals for elements equivalent to those in FIG. 2 have the same numbers as in FIG. In the image processing unit 207 in FIG. 2, the function of the decoding unit 2071 for decoding the defective pixel information is as follows.

  In FIG. 6, if the buffer 20711 for storing defective pixel information is in a storable state, the decoding unit 2071 sends a data transfer request to the data management unit 210 via the read / write controller of the buffer 20712. In response to the transfer request, the status controller 2102 in the data management unit 210 issues a transfer request for the defective pixel information 200 data to the controller 202 of the volatile storage unit 201. In FIG. 6, 2100 is an arbiter for arbitrating data transmission / reception requests from other controllers of the status controller 2102 (not shown in FIG. 6).

  When the defective pixel information 200 is stored in the buffer 20711 via the FIFO 2101 in the data management unit 210 in accordance with the request, the decoding unit 2017 causes the defective pixel information extraction unit 20713 in FIG. Is extracted and sent to the counter 20714 in FIG. The counter 20714 is loaded, for example, as an initial value of a counter that counts input image data (actually, the status of the image data valid status), and the counter 20717 decrements by 1 every time the input pixel data is counted. A down counter configuration can be adopted. When the counter 20714 value becomes 0 (relative distance is zero), a status is output so that the input data is treated as a defective pixel.

  The defective pixel status is input to the AND processing units 207151, 207152, and 207153 in FIG. 6, and is also input to the read / write controller 20712 in FIG. The read / write controller 20712 recognizes the completion of defective processing for one pixel, switches the reading destination of the buffer 20711 to the next data, reduces the number of unprocessed data in the stored data, and stores it in the storage area. If there is room, a request for next data is issued to the status controller 2102 of the data management unit 210.

  Further, when the defective pixel information does not reach the buffer 20711 in spite of the next data transfer request, and there is no unprocessed data in the buffer 20711, the read / write controller 20712 displays the image data sending unit. In this case, an image data transmission stop request status is issued to (FIFO 206 in FIG. 2). The FIFO 206 requires a capacity for absorbing the stop request for imaging data input.

  In addition to position information, the defective pixel information 200 in the embodiment of the present invention is accompanied by defect attribute information, information for specifying whether or not to perform defect correction processing in the image processing pass, and the like. In this embodiment, the position information (regardless of the attribute of the defect) is distance information (relative position) with the previous defect, the defect attribute is the process defect, the sensitivity defect, and the focus detection sensor. It is set to have.

  In addition, the defective pixel information 200 in the embodiment of the present invention also includes grade information indicating a defective state. This is information for processing as a normal pixel without considering it to be defective under certain photographing conditions depending on the degree of the defect. For example, the process scratch is defined as a constant correction under a certain condition, but there is a degree that does not require correction depending on the certain condition (ISO sensitivity, shooting time, temperature, etc.). Therefore, by attaching a grade of that degree to the attribute of the process scratch, it is possible to determine whether or not it is a scratch in accordance with the grade threshold set in the circuit. The grade information refers to the information.

  In FIG. 6, reference numerals 20715, 20716, and 20717 denote grade determination means, and the value of the grade information existing together with the position information in the defective pixel information 200, and the value (setting unit not shown) set in the determination means. If the grade of the attention information is larger than the grade setting value (may be smaller or equal), a status is issued so that the defect information is effectively handled.

  In the embodiment of the present invention, the focus detection sensor pixel position is also included in the defect information. However, these are only selections of the normal sensor or the special sensor. If defective attribute information having no concept of grade is handled, the grade determining means corresponding thereto may be omitted or may be passed. In FIG. 6, this means is provided for all defect attributes as an apparatus specification, and an embodiment in which, for example, the status of the process flaw can be assigned to any path of these 20715 to 20717 is possible. However, when considering the hardware configuration, it is not easy to add to or delete from the device later, so it is described because it is more practical to provide a uniform explanation. .

  The defective pixel information extracting unit 20713 extracts the information of the defective attribute and the grade information from the defective pixel information to be processed from the buffer 20711, and appropriately transmits the information to the grade determining units 20715 to 20717. In the embodiment of the present invention, for example, the grade determination means 20715 is used as the process scratch status path, 20716 is used as the focus detection sensor pixel status path, and 20717 is used as the sensitivity scratch status path. Assign each one.

  The grade determination means 20715 determines the above-mentioned scratch grade. Judgment criteria can be obtained by comparing with a reference grade separately set in a register (not shown). If it is applied as a process scratch, 1 value is output. Further, for example, if grade judgment is unnecessary on the grade judging means 20716, 20717 side in FIG. 6, only the attribute result of the defective pixel information extracting means is referred to, and if applied, 1 value is set. Output.

  The logical product of 207151 in FIG. 6 is the logical product of the output of the grade determination means 20715, the output of the counter 20714, and the image data valid status (valid with 1 value). If it is determined that the current target image data is the process flaw, the logical product 207151 outputs a 1 value. The same applies to the logical product of 207161 and 207171 in FIG.

  The output results of the logical products 207151, 207161, and 207171 are merged with the image data and held in the register 20719. The embodiment of the present invention presents an implementation as a digital circuit configuration, and the image processing unit 207 operates in synchronization with the same clock. The register 20719 is a flip-flop circuit, and performs data transfer in synchronization with the clock, but the clock line is not shown for simplification of the drawing. The same applies to the registers 20718, 20721, 20722, and 20723 in FIG. Similarly, the reset condition is not shown.

  In the embodiment of the present invention, the image data is set to 12 bits, and the outputs of the logical products 207151, 207161, and 207171 are merged as the defect correction status, so the output of the register 20719 is 15 bits. The bit arrangement relationship between the image data and the defect correction status is not limited, but in FIG. 6, the upper 12 bits are used as image data and the lower 3 bits are used as the status. The bit arrangement of the status is the correction status of each of the process flaw, the focus detection sensor pixel, and the sensitivity flaw from the lower bit.

  Connection between 2071 and 2072 in FIG. 6 is performed via selectors 601 and 602. The selectors 601 (data valid status side) and 602 (image data side) select the input side connection destination of the submodule 2072. Here, the connection with the decoding unit 2071 is illustrated. The selector can be set by setting a connection register (not shown).

  The submodule 2072 can be connected to each submodule in FIG. 2 (for example, 2071, 2073, 2074, etc.), and the relationship is the same between the other submodules. This is made possible by the interface having (image) data, (image) data valid status, and (image) data transmission stop request status shown in FIG. The data valid status indicates that the image data is valid when the value is 1, and the stop request status requests that the previous image data and the valid status be immediately stopped when the value is 1. In the embodiment of the present invention, these signals guarantee in-phase transfer synchronized with the rising edge of the clock.

  The logical sum of 603 in FIG. 6 is for superimposing the stop request status coming from the submodule that follows the decoding unit 2071. As an actual logic, it is necessary to take a logical product (not shown) with the stop request status from each submodule in the subsequent stage according to the value of a connection register (not shown) set in the selectors 601 and 602. That is, it is because the stop request from the submodule not connected is not included in the logical sum 603.

  Reference numeral 2072 in FIG. 6 provides a simple pre-interpolation function. The adoption of pre-interpolation is for the sake of simplicity, and does not limit the adoption of other defect corrections in the apparatus according to the present invention. In the embodiment of the present invention, the replacement with the immediately preceding image data is presented as a measure when the pre-interpolation unit 2072 is defective. This is also for simplifying the explanation.

  In FIG. 6, reference numeral 20721 denotes a register that temporarily holds image data to be transferred to the subsequent stage. Similarly, reference numeral 20722 in FIG. 6 is a register that temporarily holds the state of the image data valid status. In FIG. 6, reference numeral 20723 denotes a register that temporarily holds the state of the defect correction status portion. In the embodiment of the present invention, these are flip-flop circuit configurations as described above, and have a synchronous clock input (not shown).

  The registers 20721 to 20723 hold the current value when the stop request status from the subsequent stage is 1 at the rising edge of the synchronous clock. In the embodiment of the present invention, as described above, the upper 12 bits are the image signal and the lower 3 bits are the defect correction status for each defect attribute.

  The product-sum logic unit 20725 in FIG. 6 has an attribute selection value (register unit not shown. In this embodiment, the attribute 1 selection in FIG. 6 is the process flaw selection, the attribute 2 selection is the focus detection sensor pixel selection, The attribute AND selection with the corresponding defect correction status is used as the selection condition of the selector 20724 in FIG. If the condition is 1 value, the image data at that time is replaced with the value of the register 20721 (the previous image data) as a defective pixel.

  The setting value for the attribute selection can be set to enable / disable independently for each of the process defect, the sensitivity defect, and the focus detection sensor pixel. If the pre-interpolation unit 2072 wishes to perform pre-interpolation for the sensitivity flaw, the attribute 3 selection may be set to enable as in the process flaw.

  The product-sum logic 20726 in FIG. 6 is for canceling the defect correction status. For example, when it is desired to cancel the correction status for the pixel whose defect has been corrected in the pre-interpolation unit 2072, the selection is canceled so that the output value of the product-sum logic unit 20726 is selected by the selector 20727 in FIG. The register value may be set (the register unit is not shown).

  Such updating of the correction status is effective. For example, sensitivity flaws are divided into detection means and correction means (for example, detection at 2092 in FIG. 2 and correction at 2093 in FIG. 2). It is also possible to easily superimpose the result detected by the detecting means on the correction status that has been determined (attribute detection by the decoding unit 2071).

  For example, when 208 in FIG. 2 is the focus detection sensor value processing module, it is possible to extract only the focus detection sensor pixel position in the sub module 2082 in FIG. For example, if 209 in FIG. 2 is an image development processing unit and 2092 in FIG. 2 is a defective pixel correction processing unit, the defect correction status is unnecessary in the subsequent submodules. In such a case, the correction processing unit 2092 only needs to output 12 bits of image data.

[Example 2]
In this embodiment, the means for independently extracting the flaw position information for each attribute from the defective pixel information, including the attribute and position information of the pixel that requires correction of different defective pixels of the process flaw and a plurality of other interpretations, Present.

  Also by constructing a decoding unit for the defective pixel information that embodies the above-described method, and appropriately referring to the defective processing target pixel position information that is the decoding result in each image processing unit, as in the first embodiment, The correction of the defective pixel information can be performed independently while the reading of the defective pixel information is made uniform.

  FIG. 1 is a principal block diagram of the present embodiment, showing decoding means for reading the defective pixel information including a plurality of different correction target attributes and extracting the defective processing target pixel position information for each target. It is a thing.

  In FIG. 1, reference numeral 101 denotes defect information analysis means for interpreting defect attributes (representing process scratches, sensitivity scratches, and types) and their pixel positions from the defective pixel information. In FIG. 1, reference numeral 102 denotes defect information conversion means for converting the position information of defective pixels that are always subject to correction into {0, 1} information in units of lines or frames. In FIG. 1, reference numeral 103 denotes volatile storage means, which is temporary storage means for temporarily developing and storing the defective pixel information stored in the apparatus.

  In FIG. 1, reference numeral 1031 denotes the defective pixel information. In this embodiment, each of the process scratches and the sensitivity scratches has respective attributes together with pixel position information. As described above, the position information is held as relative position information. It shall be. The defective pixel information 1031 is expanded in the volatile storage means 103 and is read into the defect information analysis means 101 (in FIG. 1, the control unit related to data transmission / reception of the volatile storage means 103 is It is not shown because the figure becomes complicated.)

  In the defective pixel information 1031 read by the defect information analyzing unit 101, only the data relating to the defective position is extracted by the position information extracting unit 1011 in FIG. 1, and the defect position counter 1012 in FIG. Send it out. The defect position counter 1012 accumulates distance data that arrives as a relative position. The integration result is converted into two-dimensional coordinate information by the absolute position conversion means 1016 in FIG. By assigning the higher order of the integration results to the number of lines and the lower order to the pixel positions in the line, it can be easily converted into two-dimensional coordinate information.

  As described above, since a plurality of attributes are merged as one piece of defective pixel information, the interpretation of the relative position must be a continuous operation regardless of the attributes. Therefore, the process up to the absolute position converting means is performed for all input data.

  An absolute position adjusting unit 1017 in FIG. 1 is a unit for adjusting the correction position extracted with respect to image frame information for actually executing correction. For example, when the defective pixel information 1031 includes information for the entire screen of the image sensor, the actually captured frame information may be part of the sensor information or pixel information at certain intervals. In this case, the pixel position cannot be matched and the correction is incorrect, so that means for matching the pixel position is necessary. If no adjustment is required, the absolute position adjusting means 1017 does nothing and outputs the input absolute position information as it is.

  Although not included in the present embodiment, there is a case where the result of adding several pieces of pixel information of the image sensor is used as shooting data as pixel addition. Even in such a case, it is necessary to match the position information of the pixel including the defective pixel (for example, the position including the defective pixel in the addition pixel is determined as the defective pixel).

  In FIG. 1, reference numeral 1013 denotes failure attribute analysis means. Here, the attributes of the process scratch / sensitivity scratch are extracted. If the information on the sensitivity flaw is recognized here, the defect attribute analyzing means 1013 instructs the extraction buffer 1018 in FIG. 1 to hold the current absolute coordinate information. The extraction buffer 1018 sends the accumulated data to the volatile storage unit 103 (a data transmission / reception control unit of the volatile storage unit 103 is not shown). In FIG. 1, 1033 is sensitivity flaw position information stored in the volatile storage means 103. If the threshold value for determining the flaw related to the sensitive flaw is set to be one in the image frame, the sensitive flaw position information may include only the position information.

  In FIG. 1, 1014 is a flaw data execution analysis means. The process scratch is defined as a constant correction under a certain condition, but depending on the certain condition (ISO sensitivity, shooting time, temperature), there is a defect that does not require correction. Therefore, by attaching a grade of that degree to the attribute of the process scratch, it is possible to determine whether or not it is a scratch in accordance with the grade threshold set in the circuit. The part to be executed is the flaw data execution analysis means 1014.

  In response to the result of the process flaw correction request from the flaw data execution analysis means 1014, the H buffer information of the absolute position conversion means result 1016 is stored in the execution buffer 1015 in FIG.

  The defect information conversion means 102 in FIG. 1 is a part that converts whether or not the process flaws need to be corrected into a {0, 1} image. The line buffer 1021 in FIG. 1 has a storage capacity for 1H of the processing target image frame. In the line buffer 1021, every time the result coordinate of the absolute position converting means 1016 advances by 1H, all data is initialized to 0 value by the initializing means 1022 in FIG. The initialization unit 1022 generates a reset pulse when the position designation of the defective pixel information crosses 1H of the captured image data.

  In FIG. 1, reference numeral 1023 denotes a write controller that indicates a rewrite position within the storage range of the line buffer 1021. Each time the H position information of the execution buffer 1015 is updated, the H position is sent as a write address to the line buffer 1021 to instruct one-value writing. The correction required information is 1 bit for one pixel. If the line buffer 1021 is composed of, for example, a 32-bit SRAM, a write instruction is given for every 32 pixels, for example. In this case, adjustment of the bit position for single value writing is also performed in the write controller 1023.

  In FIG. 1, reference numeral 1024 denotes a read controller that gives an instruction to send the data in the line buffer 1021 to the volatile storage means 103. As described above, for example, even if the defective pixel information 1031 stores information for all pixels of the image sensor, there are cases where the defect correction is actually applied in some cases. Is performed by the effective area determination means 1025 in FIG.

  For example, if the captured image data to be actually developed is smaller than the sensor pixel, the effective area determination unit 1025 instructs the read controller 1024 not to read unnecessary portions.

  In response to the instruction from the read controller 1024, the line buffer 1021 sends data to the arrangement adjusting means 1026 in FIG. The arrangement adjusting unit 1026 includes a buffer, adjusts the data arrangement to the received data width of the volatile storage unit 103, and sends data to the unit 103 (data transmission / reception to the volatile storage unit 103). (The control unit is not shown).

  For example, for the captured image having valid data for every two pixels of the image sensor, the arrangement adjusting means 1026 also performs the work of repacking the data for every 2 bits in the line buffer 1021. In FIG. 1, 1032 is {0, 1} status information of necessity / unnecessity of process scratches developed on the volatile storage unit 103 in this way.

  FIG. 3 is a diagram showing a processing configuration for the defective pixel information including information on each of the process defect, the sensitivity defect, and the focus detection sensor pixel. Since the components are the same as those in FIG. 1, only the differences will be described here. Two means corresponding to the flaw data execution analysis means 1014 are mounted, the process flaw 3013 and the focus detection sensor pixel 3015.

  The flaw data execution analysis means 3015 for the focus detection sensor pixel is not necessary because it is subject to correction at all times during image processing depending on its structure. For example, when only the focus detection sensor pixel in an arbitrary region is used (for focus detection), the discrimination is necessary, and the means 3015 is provided here. If the focus detection sensor pixel outside the region is image-processed, it is necessary to send a status to correct it, so the output of the block 3015 is input to the execution buffer 1 at 3014 in FIG. Is also connected.

  The line buffer corresponding to 1021 in FIG. 1 is also mounted with two for process scratch status 3020 and focus detection pixel status 3021. Moreover, two light controllers 3023 and 3024 corresponding to each are mounted. Since there is only one progress management by the defect position counter 3011, each of the line buffer initialization means 3022 and the effective area determination means 3026 for actually reading out as valid data is provided as one.

  The arrangement adjustment means 1026 is equivalent to two independent processing defect statuses 3027 and focus detection sensor pixel statuses 3028, respectively. As a result, the defect information of the defective pixel information 3031 to be processed can be separated and extracted into the process defect correction status 3032, the focus detection sensor position status 3033, and the sensitivity defect correction position information 3034, respectively.

  Since the correction for the process scratch correction status and the focus detection sensor pixel correction status that are {0, 1} information is performed, the 0 value and the 1 value are in one-to-one correspondence with the presence or absence of defects for each pixel. For example, the correction can be made by handling equivalent to the output 411 in FIG. Also, sensitivity flaw correction position information. In the present embodiment, the details of the method for correcting the defective pixel itself are not included in the features of the invention and will not be described here.

406 Failure information analysis means 102 Failure information conversion means 207, 208, 209 Image processing means 2101, 2103, 2105 FIFO
2102, 2104, 2106 Status controller

Claims (23)

Located in an image processing device that processes image information,
Correction pixel information analysis means for receiving correction request information (hereinafter referred to as defective pixel information) for an arbitrary pixel in the image information and analyzing the information;
The defective pixel information includes position information for specifying a pixel to be corrected (hereinafter referred to as a defective pixel), and attribute information regarding the cause of the defect or (and) the type of correction,
From the position information and the attribute information, generate a status indicating whether it is a normal pixel or a defective pixel for each attribute,
Defective pixel information analysis means, wherein the statuses are added together to the image information.   The status indicating whether the pixel is a normal pixel or a defective pixel according to an arbitrary attribute is indicated by 1 bit in which a 0 value or a 1 value is assigned to either of them. Defective pixel information analysis means.   The addition of the respective statuses to the image information is performed so as to be extended with respect to the least significant bit or the most significant bit of the image information. Defective pixel information analysis means. The defective pixel information analyzed by the defective pixel information analyzing means according to any one of claims 1 to 3,
The defective pixel which is a target for the defective pixel information includes information on a pixel (focus detection sensor pixel) used for an application different from the image formation existing in the image information. , Defective pixel information. An image processing apparatus that includes an imaging unit, and includes information (hereinafter, defective pixel information) that instructs correction of an arbitrary pixel in the imaging unit, and the defective pixel information analysis unit.
The defective pixel information includes position information for specifying a pixel to be corrected (hereinafter referred to as a defective pixel), and attribute information regarding the cause of the defect or (and) the type of correction,
Nonvolatile storage means for storing the defective pixel information;
Temporary storage means for temporarily storing it,
A controller of the temporary storage means;
A decoding unit that receives the defective pixel information from the temporary storage unit and analyzes the content of the information;
The decoding unit comprises defective pixel information extraction means and a counter for counting pixels, respectively.
The decoding unit receives the defective pixel information and the image information obtained from the imaging unit,
The defective pixel information extracting means is means for extracting the position information and the attribute information, respectively.
The counter is a counter that counts and updates each time it receives the valid image information,
From the counting / update result of the counter value in consideration of the position information, it is determined whether the target pixel is defective, and a defect status indicating whether the target pixel is defective is generated,
The decoding unit outputs the generated current failure status by superimposing (merging) the generated current defect status with the image information as one or more status information based on the attribute information when outputting the received image information. An image processing apparatus. Receiving image information on which the defect status information is superimposed (merged), and having image information processing means for processing the image information;
6. The image processing apparatus according to claim 5, wherein the image information processing means includes means (register) for holding the superimposed (merged) defect status during image information processing. Receiving image information on which the defect status information is superimposed (merged), and having image information processing means for processing the image information;
The image processing apparatus according to claim 5, wherein the image information processing unit is configured to separate the superimposed (merged) defect statuses during image information processing. Receiving image information on which the defect status information is superimposed (merged), and having image information processing means for processing the image information;
6. The image processing apparatus according to claim 5, further comprising a processing configuration that updates the state of the superimposed (merged) defect status information based on an image information processing result of the image information processing means. . The image information processing means includes a process of performing a defective pixel correction process on an arbitrary pixel based on the defect status information,
The update of the defect status information indicates that, when the target pixel is corrected by the above-described defective pixel correction process, the defect status is changed from a defective state to a normal pixel handling state. Image information processing apparatus. The defective pixel information analyzed by the image information processing apparatus according to any one of claims 5 to 9,
The defective pixel which is a target for the defective pixel information includes information on a pixel (focus detection sensor pixel) used for an application different from the image formation existing in the image information. , Defective pixel information. Located in an image processing device that processes image information,
Correction pixel information analysis means for receiving correction request information (hereinafter referred to as defective pixel information) for an arbitrary pixel in the image information and analyzing the information;
The defective pixel information includes position information for specifying a pixel to be corrected (hereinafter referred to as a defective pixel), and attribute information regarding the cause of the defect or (and) the type of correction,
The defective pixel information analyzing means comprises means for extracting the defective pixel position from the position information, and means for extracting the cause of the defect or (and) the type of correction from the attribute information.
Depending on the cause of the extracted failure and / or the type of correction,
Control for converting the information of the extracted defective pixel position into an arbitrary format;
The defective pixel information analyzing means characterized in that it can selectively execute control of switching to a different status at the extracted defective pixel position and other positions. The position information in the defective pixel information includes relative position information indicating a distance to the defective pixel immediately before the defective pixel of interest,
12. The defective pixel information analyzing unit according to claim 11, wherein the arbitrary format for converting the information on the defective pixel position includes absolute position information indicating coordinates in a frame. The statuses that are different between the defective pixel position and other positions are:
12. The defective pixel information analyzing means according to claim 11, wherein whether the pixel is a normal pixel or a defective pixel is indicated by 1 bit in which 0 value and 1 value are assigned to either. The defective pixel information analyzed by the defective pixel information analyzing unit according to any one of claims 11 to 13,
The defective pixel which is a target for the defective pixel information includes information on a pixel (focus detection sensor pixel) used for an application different from the image formation existing in the image information. , Defective pixel information. An image processing apparatus including the defective pixel information analysis unit according to claim 11 and a storage unit,
An image processing apparatus, wherein the defective pixel information analyzing unit includes an interface for storing, in the storage unit, information on a defective pixel position converted into the arbitrary format. An image processing apparatus including the defective pixel information analysis unit according to claim 11 and a storage unit,
An image processing apparatus comprising an interface for storing in the storage means information on statuses that differ between the generated defective pixel position and other positions in the storage means. An image processing apparatus that includes an imaging unit, and includes information (hereinafter, defective pixel information) that instructs correction of an arbitrary pixel in the imaging unit, and the defective pixel information analysis unit.
The defective pixel information includes position information for specifying a pixel to be corrected (hereinafter referred to as a defective pixel), and attribute information regarding the cause of the defect or (and) the type of correction,
Nonvolatile storage means for storing the defective pixel information;
Temporary storage means for temporarily storing it,
A controller of the temporary storage means;
Receiving the defective pixel information from the temporary storage means, and analyzing the content of the information;
Receiving the output information of the defect information analysis means, and having defect information conversion means for converting the content of the information,
The defect information analysis means includes position information extraction means for extracting the position information from the defective pixel information, a defect position counter for counting and updating a value corresponding to the position information, and the attribute information from the defective pixel information. A defect attribute analyzing means for extracting,
The defect information conversion means includes a temporary buffer for holding a status signal status indicating whether or not defect correction is performed for pixels in a continuous positional relationship, an initialization means for the temporary buffer, and a defect attribute analysis of the defect information analysis means. A write controller for the temporary buffer controlled by information in view of the output of the means, a read controller for the temporary buffer, and an arrangement adjusting means,
The state of the status signal temporarily stored in the temporary buffer is a {0, 1} signal, and the signal is handled as data in a unit suitable for storage in the temporary storage unit via the array adjustment unit. And an image processing apparatus characterized in that the image processing apparatus stores the temporary storage means.   The temporary buffer in the defect information converting means is mounted as a line buffer having a capacity sufficient to store the status signal status for one horizontal line of the processed image. The image processing apparatus according to 17. The write controller of the temporary buffer makes the status signal state {0, 1}, which can be expressed as 1-bit information, a plurality of bits collectively in 1-word units,
19. The image processing apparatus according to claim 17, wherein the temporary buffer has an SRAM memory configuration in which a data storage / read unit is the word unit. The read controller of the temporary buffer issues a timing for reading the read word unit of the temporary buffer, and
The image processing according to any one of claims 17 to 19, wherein the arrangement adjusting unit temporarily receives the data in the temporary buffer in units of words according to a read timing of the read controller. apparatus.   The arrangement adjusting means receives at least one word unit data from the temporary buffer once received according to the read timing of the read controller, adjusts it to the handling data arrangement of the temporary storage means, and then stores the data in the temporary storage means. The image processing apparatus according to claim 17, wherein the image processing apparatus outputs the output of the image processing apparatus.   The temporary storage unit is a volatile memory (DRAM) that can be read / written a plurality of times, or a nonvolatile memory (FlashROM), according to any one of claims 17 to 21. Image processing device. The defective pixel information analyzed by the image information processing apparatus according to any one of claims 17 to 22.
The defective pixel which is a target for the defective pixel information includes information on a pixel (focus detection sensor pixel) used for an application different from the image formation existing in the image information. , Defective pixel information.