JP4560353B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a method for correcting a defective pixel included in an image obtained by an imaging means.

  Conventionally, defective pixel correction of an image sensor has been performed on image data captured by the image sensor. As a specific method, an image obtained from an image sensor is temporarily stored in a memory, and a defective pixel is interpolated using other pixel information based on a defect position stored in advance in a ROM (for example, JP-A-6-105242).

  Japanese Patent Application Laid-Open No. 10-42244 describes a method of storing 10-bit pixel data with 16-bit memory bits packed.

  Address calculation in writing processing of pixel data read from the image sensor to the memory will be described.

  FIG. 2 is a diagram when 10-bit pixel data is stored in a 16-bit width memory. Although there are variations such as whether the divided pixels are packed in the upper 16 bits or in the lower 16 bits, it is understood that the repetition is 5 words (10 bytes). A value obtained by dividing 80 bits, which is the least common multiple of 10 and 16, by 16 bits is a repetition size of 5 words. FIG. 3 shows a case where 9-bit pixel data is stored in a 16-bit width memory. It is repeated in units of 9 words (18 bytes). On the other hand, the position of the defective pixel is generally expressed by coordinates on the image of the pixel, for example, as shown in FIG. 29 from the left on the X line from the left to the Y pixel.

With respect to defective pixels, the pixel data values are generally not used as they are, but the pixel values are generally generated by correcting the defective pixel data from the peripheral pixel values and corrected.
JP-A-6-105242 JP-A-10-42244

  Considering the case of performing the defective pixel correction processing as described above, for example, when calculating a value obtained by averaging four pixels at the top, bottom, left, and right in order to correct a defective pixel, bits are clogged from the coordinates of the pixel on the image. The location in memory must be calculated at least 5 times. Further, assuming that bit filling is performed in order from the lower bit, the following calculation is required to obtain the memory address and the stored bit position from the coordinates of the pixel on the image.

(1) Number in pixel order = X line from top × horizontal width of image + Y pixel from left (2) Address on memory = ((number in pixel order + 1) × bit width of pixel) / bit width of memory (3) Stored bit position = ((pixel order number + 1) × pixel bit width)% bit width of memory and even if it is optimized to avoid recalculation, at least two times multiplication and one time Since it includes division, if the average of the pixel values of the upper, lower, left, and right sides is calculated to correct one defective pixel, the address calculation requires at least 10 multiplications and 5 divisions. The required calculation is also required.

  This means that in the case of the defective pixel correction program as described above, the address calculation becomes a dominant element of the processing time, and the time required for the address calculation directly affects the delay of the processing time.

  There are other reasons for increasing the processing time. Considering the case of using CPUs with different endians, the processing becomes further complicated. FIGS. 2 and 3 are examples of a 16-bit memory, but a 32-bit memory may be used. Since such bit padding can be configured by relatively small scale hardware, the configuration in which the bit converted by the AD from the image pickup device is optimal by hardware is also optimal in terms of efficiency. Japanese Patent Laid-Open No. 10-42244 has such a configuration. However, depending on the CPU to be used, there may be a case where the endianness, that is, the number of bits as a basic information unit does not match between the CPU and the bit stuffing hardware. If a CPU with a different endian is adopted for each product lineup due to cost or the like, the endian cannot be matched between the bit-packed hardware and the CPU. This means that it is necessary to know a method for extracting pixels from stored bit positions, which requires more complicated calculation and increases the processing time.

Also, the cause of increasing the processing time is the access operation to the DRAM. Since the number of pixels of a digital camera is increasing and the image size becomes large, there is a cost merit by using a large capacity and inexpensive DRAM. On the other hand, data storage in the DRAM is managed based on the concept of pages, and access to the same page is bursty (burst transfer), so access to different locations on the page is time consuming. It takes. In other words, it is expected that efficiency will be improved by accessing as locally as possible and avoiding access across pages. However, considering the use of pixels of the same color on the top, bottom, left and right, in general, the same color of the color filter of the image sensor is often not continuous. For example, every other pixel of the same color as in the arrangement shown in FIG. Therefore, the upper and lower pixels of the same color mean pixels above and below two lines. In order to interpolate a defective pixel surrounded by a circle in FIG. 30, the value of the pixel surrounded by a dotted circle is used. That is, in order to correct one defective pixel, access to an area extending over a distance of 4 lines is required. For example, if 1 pixel is 12 bits and 1 line is 1440 pixels, the distance of 4 lines is
(4) 12 bits x 1440 pixels x 4 lines / 16 bits = 4320 words (8640 bytes)
It will exceed 8K bytes. Although it depends on the configuration of the DRAM, etc., one page uses a size of 16 Kbytes or 32 Kbytes, so if you simply access the upper and lower two lines of pixels sequentially, accesses that cross the page occur frequently. It becomes a factor which lengthens processing time.

  Also, from the viewpoint of synchronization with image transfer, as the number of pixels of digital cameras increases and the image size increases, the transfer time for transferring AD-converted bits from the image sensor to the memory also increases. Therefore, it is difficult to perform defective pixel correction processing efficiently. Especially when a defective pixel is dynamically corrected by a digital camera or the like, the processing time has an important influence on the continuous shooting performance, etc., so that the defective pixel correction processing time is shortened and high-speed defective pixel correction processing can be performed. Is very important.

  Accordingly, an object of the present invention is to speed up the storage processing and access operation of image data in a memory such as the above-described digital camera, to shorten the time required for various processing as much as possible, and to correct image data such as defective pixel correction processing. It is intended to increase the speed of various processes.

In order to solve the above-described problem, according to the first aspect of the present invention, in an image processing apparatus that corrects a defective pixel included in image data obtained by an imaging unit, the image processing unit obtains the image from the imaging unit. Means for converting the received image data into digital image data, bit-packing it in an image memory, and transferring a plurality of peripheral digital image data based on position data of defective pixels to be corrected from the image memory Correction processing means for performing defective pixel correction by performing an interpolation operation and writing the corrected image data into the image memory, the image memory is logically handled in units of pages, and the correction processing means When reading peripheral digital image data used for pixel interpolation, the peripheral digital image data straddling the page boundary of the image memory If present, and wherein the image processing apparatus number of times of switching the pages is to control the reading order so as to minimize.

  According to the present invention, the process for extracting an arbitrary pixel from the image memory can be executed by an intermediate language program, eliminating the calculation time of the address and bit position, which has been dominant in the conventional data processing, Processing speed can be increased.

(First embodiment)
FIG. 1 is a block diagram of the first embodiment. A CPU 1 executes a computer language interpreter according to the present invention in addition to processing of the entire camera. For example, a lens and a shutter are connected to perform processing of the entire camera, but are omitted because they are not necessary for the description of the present invention. Reference numeral 2 denotes a chip built-in memory configured in the same chip as the CPU 1. Since the CPU 1 is connected by a dedicated BUS in the chip, it is a high-speed memory that can be accessed at high speed with almost no waiting time. A ROM 3 stores a program that is read and operated by the CPU 1. In addition to the native code that the CPU 1 can understand, this ROM also stores a computer language code according to the present invention.

  Reference numeral 8 denotes an image sensor that photoelectrically converts an optical image incident on the imaging surface into an image signal. Reference numeral 7 denotes an AD converter that converts an analog image signal output from the image sensor 8 into a digital signal. Reference numeral 4 denotes an image processor that performs processing to pack the output from the AD converter 7 into bits and store it in the image memory 6. The image processor 4 and the image memory 6 are connected by an image transfer BUS, and the image processor 4 functions as a bus master of the image transfer BUS.

  The image processor 4 has functions such as reading out an image that has been bit-packed in addition to transfer while bit-packing the image, performing color signal processing, and writing it again to an arbitrary address in the image memory 6. Is activated and controlled by the control from the CPU 1. Reference numeral 5 denotes a memory window for enabling reading / writing to an arbitrary address space of the image transfer BUS through an arbitrary address space of the CPU external BUS. The image memory 6 can be read and written from the CPU 1 via the memory window 5. The memory window 5 can change the base address of the image memory 6 by switching the page number. As described above, the image transfer BUS is provided exclusively for image transfer, and by separating the CPU external BUS, control of various camera mechanisms and shooting operations, which are the responsibility of the CPU, the image processor 4 and the image memory It is possible to efficiently transfer image data between the six in parallel, and as a result, it is possible to reduce power consumption by stopping the CPU with a HALT command during the remaining CPU time.

  FIG. 4 is a diagram showing an internal representation of the bit-stuffed image of the first embodiment. In this embodiment, one pixel is 12 bits, and is repeated for 3 words and 4 pixels. Here, when the base address of the repetitive pattern is M, the bit position names of Pixels 1 to 4 are defined for the offset and bit position of each pixel with respect to M, respectively.

  The size of the memory window 5 in FIG. 1 is 64 Kbytes, and the bit length necessary for expressing the offset address in the memory window 5 is 16 bits. If it is expressed by the offset address in one image instead of the window offset, an image of about 8 million pixels requires 24 bits. By expressing the offset of the memory window 5, the operand size can be reduced to two-thirds capacity.

  On the other hand, since there are four types of bit positions, there are four instructions that extract each pixel and load it on the stack, and four instructions that calculate the interpolated pixel value from the four pixels packed in the stack and store it in the designated pixel. Since the total number of instructions and other page switching instructions is 128 or less, it is possible to design an instruction set based on a 3-byte instruction of 1 byte of opcode and 2 bytes of operand.

  FIG. 5 is a computer language instruction table of this embodiment. For example, an instruction “PushPixel1” is an instruction that performs an operation of extracting pixel information of a bit position in the Pixel1 format from the address as an argument and an argument in a 2-byte (16-bit) window as an operand, and stacking it on the stack. Similarly, there are instructions of the formats from Pixel 1 to Pixel 4. Also, the command ReduceAndSavePixel1 compares the pixel level difference between the left and right pixels and the pixel level difference between the upper and lower pixels from the information for the four pixels stacked on the stack, and calculates the average value of the pair on the side with the lower level difference. This is an instruction for storing the value of the interpolated pixel at the pixel 1 bit position of the operand address. Similarly, ReduceAndSave also has instructions in the format of Pixels 1 to 4.

  ChangeNextPage is an instruction to increase the page number of the memory window 5 in FIG. 1 by 1 and switch to the next page. ChangePrevPage is an instruction to decrease the page number by 1 and switch to the previous page. The ChangePage instruction is an instruction for directly switching to a page number designated by a 2-byte operand.

  The Swap instruction has no operand, and is an instruction to swap the top and second data on the stack. By using such commands together, page switching commands can be reduced, but details will be described later.

  The Exit command is a command indicating that one image has been completed, and the Return command is a command for once terminating the interpreter and returning.

  Next, synchronization with image transfer will be described. It is possible to shorten the total processing time by starting defective pixel correction for image data that has been transferred even during image transfer, rather than performing defective pixel correction after the transfer of one image has been completed. is there. However, if the defective pixel correction is simply started simultaneously with the image transfer, the defective pixel correction ends before the image transfer, and correct correction cannot be performed. Therefore, in this embodiment, the image is divided into four blocks, and the image processor 4 in FIG. 1 is interrupted to the CPU 1 in FIG. 1 every time a quarter of the image is transferred. In the program, a Return instruction is inserted in a unit of a quarter of the image. When an interrupt is input from the image processor 4 in FIG. 1, the interpreter is started, and the interpreter is temporarily exited by executing a Return instruction. The image transfer and the defective pixel correction are synchronized by causing the interpreter to restart again from the next instruction when the next interrupt is input. Thus, the image data transfer and the defective pixel correction process can be efficiently executed in parallel and in synchronization without waiting for the defective pixel correction process until all the image data for one image is transferred. it can.

  Next, the implementation of the interpreter will be described. FIG. 7 shows the implementation of the heart portion of the interpreter in this embodiment. The CPU of this embodiment is, for example, an Intel 8086 compatible instruction set. During execution of the interpreter, registers are used as shown in FIG. 6. A jump table to the interpretation part of each instruction is registered in the label InstTable in FIG. 7, and the binary code of the instruction is offset from the label InstTable. Defined by value. The Intel 8086 can express a jump in a segment with a 16-bit offset address.

  Therefore, since all binary values of the instruction code are even numbers, 128 instructions can be expressed instead of 256 as all instructions. Four instructions from the label Fetch are the heart of this interpreter. This shows that it can be interpreted very quickly. It is also possible to implement an interpreter with a very small code size. In this embodiment, this interpreter is stored in the ROM 3 in FIG. 1, but is transferred to the memory 2 in the main storage area that can be accessed at high speed by the built-in BUS in FIG. It is possible to operate.

  Further, the stack architecture will be described. List 1 in FIG. 9 is an assembler notation list of programs written in the computer language of this embodiment. Although the actual program is expressed in binary code, it will be explained in a list in assembler notation for easy understanding. The interpreter executes data processing through the stack. A program that collects data of upper, lower, left, and right pixels is as shown in Listing 1. In the program of the list, by executing the program from address 0000 to address 0009, the pixel information of the upper, lower, left and right is loaded on the stack. The stack after execution up to address 0009 is as shown in FIG. If the 000C instruction is executed, four stacks are consumed, and interpolation data is calculated and stored in the defective pixel address. This is the basic configuration of a program for interpolating defective pixels. Since the stack uses the memory 2 which is a main memory area built in the chip, which is configured in the same chip as the CPU 1 of FIG. 1, it can be accessed at a very high speed.

  Next, the defective pixel correction process across the memory pages will be described. Since one pixel is corrected using pixels extending over the upper and lower four lines, in many cases, the corrected pixel value cannot be obtained unless the pixels are actually collected across the page boundary. FIG. 10 categorizes the arrangements that can occur across the page boundaries. Arrangement that crosses the border of the page applies to any of patterns A to D. Also, since there are many patterns D and A before and after one boundary line, it is expected that the efficiency will be greatly reduced if page switching occurs frequently. It is necessary to perform correction while minimizing the number of page switching.

  Here, considering the optimization of the number of page switching, a processing method when defective pixels are concentrated before and after the boundary line will be described. List 2 in FIG. 11 is a program when there are three pixels of pattern A near the boundary of the page. When the execution from the address 0000 to the address 0006 in the list 2 is finished, the same color pixels on the two lines are stacked on the stack for the three defective pixels. FIG. 12 shows the state of the stack at that time. If the three defects are called defect 1, defect 2, and defect 3, the pixel values above defect 1, defect 2, and defect 3 are packed in the stack.

  By executing the instruction at address 0009, the window page is switched to the next page. The pixel under the defect 1 is placed on the stack by the instruction at address 000C, the pixel at the right side of defect 1 is placed at the instruction at address 000F, and the pixel left at the defect 1 is placed on the stack by the instruction at address 0012.

  FIG. 13 shows the state of the stack when list 2 is executed up to address 0012. Looking at the four from the top of the stack, it can be seen that pixels for correcting defect 1 are stacked in exactly the same order as in FIG.

  If address 0015 is executed in this state, a value for correcting the defective pixel by calculating four stacks is calculated, and then written to the defective pixel 1 designated by the operand. FIG. 14 shows the state of the stack immediately after execution of address 0015. The pixel above the defect 2 and the pixel above the defect 3 existing in the previous page remain in the stack. Originally, the page must be returned again and the pixels above defect 2 must be stacked on the stack, but this program proves that page switching can be greatly reduced by optimizing the order of instructions.

  If the processing up to the address 001D is continued, the stack becomes as shown in FIG. Looking at the four from the top of the stack, it can be seen that pixels for correcting defect 2 are stacked in exactly the same order as in FIG.

  Similarly, if list 2 is executed to the end, only one page switching command can be used to correct three defective pixels that require pixels for interpolation beyond the page boundary.

  Here, the limit of the stack will be described. If there are only three defective pixels that need interpolation pixels beyond the page boundary, the page switching may be performed once. However, if there are many defective pixels, the amount of stacks prepared by the interpreter may be exceeded. It is done. Therefore, the compiler performs code generation by switching pages in units that do not exceed the stack range prepared by the interpreter, restoring the stack, returning the page, and performing the remaining processing.

  Next, the stack SWAP will be described. In the case of pattern B in FIG. 10, since the upper and left data are the same page, the upper and left data are jammed in the stack before the page switching. The upper and left pixels on the defective pixel are adjacent to each other on the stack, and it is necessary to devise in order to make the same order as in FIG. The top and bottom and left and right must be paired.

  Listing 3 is a program that solves it. As a result of executing the addresses 0003 and 0006, the state of the stack is as shown in FIG. When the address 0009 in list 3 is executed, the page is switched. Then, 000C in list 3 is executed to stack the pixels under the defect, and the SWAP instruction at address 000F is executed to swap the stack top and the second. The left side of FIG. 18 shows the state of the stack before the execution of the SWAP instruction, and the right side of FIG. 18 shows the state of the stack after the execution of the SWAP instruction.

  Here, a process for pixels across pages will be described. Since it is a 6-byte cycle, when it is handled with one page 64k, pixels that cross the page appear. FIG. 19 is a diagram showing pixels straddling a page boundary. Pixel 3 in FIG. 19 spans two pages when the page boundary reaches the line page boundary A, and Pixel 2 in FIG. 19 spans two pages when the page boundary B reaches. FIG. 20 shows instructions for handling pixels that cross a page boundary.

  Next, the correction program compiler will be described. The above program is executed inside the camera. On the other hand, the program as described above is generated in the production process of the camera and stored in the ROM inside the camera. FIG. 21 shows a process in which a camera defect correction program is manufactured. A defective pixel on the image sensor is detected by a defective pixel detection program inside the camera. Then, it is transferred to factory equipment and a defective pixel position information file is created. Next, a defective pixel correction program is generated by a correction program compiler inside the factory equipment. The generated defective pixel correction program is transferred again into the camera. Thereafter, a defective pixel correction program is executed by an interpreter inside the camera.

  FIG. 22 is a block diagram of the correction program compiler. In general, a compiler is constituted by a plurality of blocks so that the role of each block is clarified and the overall complexity is reduced. Each block input is the output of the preceding block, and the output is the input of the next block.

In FIG. 22, reference numeral 221 denotes a file phrase analysis unit which analyzes the format of an input defective pixel position information file and converts it into an internal representation of the compiler. Reference numeral 222 denotes a part for sorting in the order of positions. Since the block subsequent to this block is designed on the assumption that the defective pixel position information flows in order from the top to the bottom of the image, sorting is performed in this block. Reference numeral 223 denotes a portion for extracting correction pixels. In order to calculate an interpolation value using information of four pixels on the top, bottom, left, and right for one defective pixel, the coordinates are extracted. The output of this block can be handled as a quaternary operator with a prefix notation with four operands if the coordinates of one defective pixel and four pixels are one set and one defective pixel is an operator. . The equation for determining the defective pixel is (5) Defect position correction value = interpolation value calculation (upper pixel value, upper pixel value, right pixel value, left pixel value)
An intermediate representation of this stage of the program that does it is, for example:
(6) Defective pixel position The defective pixel position can be handled as a series of expressions of upper pixel position, lower pixel value position, right pixel value position, and left pixel value position. Each position information is still expressed by coordinates on the image.

Reference numeral 224 in FIG. 22 converts the coordinates on the image into bit number-packed memory storage format, that is, page number + offset address + storage bit position. If the expression of page number + offset address + storage bit position is defined as the pixel address format, the output of this block is (7) a list of defective pixel address upper pixel address, lower pixel value address, right pixel value address, left pixel value address It becomes. This internal representation is defined as a “defective pixel address expression”.

In block 225 in FIG. 22, it is determined to which pattern in FIG. 10 the defective pixel position applies, and set as an attribute for the defective pixel address. Further, a pattern that does not apply to any of the patterns A to D is defined as a pattern N and set as an attribute. “Defective pixel address expression” which is an intermediate expression with attributes is: (8) defective pixel address N upper pixel value address, lower pixel value address, right pixel value address, left pixel value address (9) defective pixel address D upper pixel Value address, lower pixel value address, right pixel value address, left pixel value address (10) defective pixel address D upper pixel value address, lower pixel value address, right pixel value address, left pixel value address (11) defective pixel address A The upper pixel value address, the lower pixel value address, the right pixel value address, the left pixel value address, and so on.

  Here, the conversion from the prefix notation to the postfix notation (reverse Polish method) will be described.

22 of FIG. 22 converts the intermediate representation expressed by the prefix notation into the postfix notation (reverse Polish method). If the “defective pixel address” corresponding to the operator is brought to the end, it is converted into postfix notation. That means
(12) Defective pixel address Internal representation of upper pixel value address, lower pixel value address, right pixel value address, left pixel value address (13) upper pixel value address, lower pixel value address, right pixel value address, left pixel value Address The main function of this block is to change the internal representation of the defective image address.

  This is because the postfix notation (reverse Polish notation) is the processing order of the program language of the present embodiment because the program language of the present embodiment proceeds with processing using the stack.

  Here, the analysis to the optimal order will be described. The input of the analysis unit 226 in FIG. 22 is a matrix of a defective pixel address formula of the prefix notation, and a format (pattern A to pattern D + pattern N) exceeding the page boundary is added as an attribute. When this is expressed in BNF, it becomes as shown in FIG. In the vicinity of the page boundary, in many cases, there is a formula pattern B or a formula pattern C after a continuous formula pattern D, and then the sequential formula pattern A is in this order. This is because defective pixels are sorted at 222 in FIG. 22, and it can be seen from the pattern in FIG. 10 that they appear only in that order in the structure. Therefore, the processing is advanced by understanding the continuous expression pattern D and the continuous expression pattern A as a syntax.

  The BNF expression of the expression pattern D is defined and handled as shown in FIG. The portion surrounded by <> means the end and corresponds to that of the defective pixel address type. The formula pattern D is a defective pixel address formula consisting of upper, lower, left and right pixels following the defective pixel address D. In addition, the structure recursively includes the expression pattern D.

  In this block, conversion to postfix notation (reverse Polish notation) is performed while creating an analysis tree of the expression. The continuous equation pattern D is developed as an analysis tree as shown in FIG. It can be seen that the recursively defined expression pattern D is converted into a postscript notation and simultaneously expanded into a deep parse tree. Finally, the expression pattern C is expanded so that the end of the analysis tree is reached, and then code generation is performed in order from the left side of the analysis tree. FIG. 26 is a BNF expression of expression patterns A, B, C, and N. The expression pattern A is also recursively defined like the expression pattern D. FIG. 27 is an analysis tree of a continuous equation pattern A and equation pattern B.

  List 4 in FIG. 28 is source code written in C language for more clearly explaining the principle of the analysis unit 226 in FIG. In the function program, the expression is processed until the expression ends, and in the function Formula, the “defective pixel address expression” is processed. The “defective pixel address type” is any one of PatternN, PatternA, PatternB, PatternC, and PatternD. The function PatternN performs processing corresponding to the expression pattern N. If the next “defective pixel address expression” is an expression pattern N, a push command for extracting the values of the pixels in the order of the upper pixel address, the lower pixel address, the left pixel address, and the right pixel address and loading them on the stack Next, the stack is consumed, the interpolation value is calculated, and the instruction to write to the defective pixel address is generated. A function GeneratePush and a function GenerateReduceAndSave are functions of the code generation unit 227 in FIG. Also, if the function GetNextIfPatternN is a pattern specified by the function belonging to the page boundary pattern extraction unit 225 in FIG. 22, the “defective pixel address expression” returns the address of the pixel specified by the read parameter. A constant NOTHING is returned as a return value.

  The function PatternA does not perform any processing unless the next “defective pixel address expression” is PatternA, and if the next “defective pixel address expression” is PatternA, first, the value of the upper pixel address is extracted and stacked on the stack. Generate instructions. The next line is a Pattern A recursive call again.

  By this recursive call, an analysis tree for continuous Pattern A is constructed on the stack of this program, and as a result, the same analysis tree as in FIG. 27 is processed. Such a structure is known as a recursive descent parser, and it can be seen that a program for correcting defective pixels is generated by a structural analysis unit having a very low complexity.

  A defective pixel correction program is generated by the code generation unit 227 in FIG. Here, a command to be commanded is derived from the damaged pixel address. The code generation request is received by the function GeneratePush and the function GenerateReduceAndSave from the analysis unit 226 in FIG. It is also the role of the code generation unit to generate a page switching instruction at an appropriate place. The code generation unit 227 in FIG. 22 knows the current page by the instruction from the analysis unit 226 in FIG. 22, detects that the page is switched, and inserts a page switching instruction. The code generation unit 227 in FIG. 22 knows the binary code assigned to each instruction and directly converts it into a code for output.

  Although omitted in the explanation of the structure of the compiler because it is not important in terms of structure, instructions are generated so as not to exceed the upper limit of the stack prepared by the interpreter, and a Return instruction for synchronizing with image transfer is also inserted. . Both are instructions generated by the analysis unit.

(Other examples)
In the first embodiment, the hardware page unit to be accessed through the window is switched. However, the overhead reduction effect is great even if it is performed in the DRAM page unit.

  In the first embodiment, the computer language is interpreted and executed by the computer program of the interpreter. However, the computer language may be executed by a processor that can directly interpret and execute the computer language.

  In the first embodiment, a recursive descent parser is used to explain the simplicity of the structure, but structural analysis may be performed using a shift ready use parser or other parsers.

  In the first embodiment, 12-bit pixels are stored in a 16-bit memory, but the same effect can be obtained with other bit widths.

  In the first embodiment, an area sensor is assumed as the image sensor, but an image scanner or a film scanner using a line sensor has the same effect.

  As described above, according to the above-described embodiment, the process of obtaining the memory address and the stored bit position from the address on the coordinate of the image is performed in advance, and the result is obtained from the bit-padded image memory. By installing it in the camera as a computer language composed of an instruction system with instructions for extracting pixels, it is not necessary to calculate addresses in the camera, and a high-speed defect correction program can be created. . As a result, the calculation time of the address and bit position, which has been dominant in the past, can be eliminated. For example, 10 multiplications and 5 divisions can be reduced for one defective pixel.

  Also, it has an instruction to extract an arbitrary pixel from the bit-padded image memory, put it on the stack, consume the data on the stack, perform an operation, and write the operation result to an arbitrary pixel position in the bit-padded image memory By using a computer language having instructions, it is possible to separate the collection of pixel information used for correction and the algorithm for calculating the corrected pixel value, so that the program can be flexibly changed. For example, change the command to write the average value of the upper, lower, left, and right pixels to the defective pixel position to the command that collects the upper, lower, left, and right pixels when the algorithm is changed to use only the average value of the set that has a small difference between the top, bottom, left, and right. There is an effect that the influence of.

  In addition, the image memory is logically handled in units of pages, such as bank switching in the bank switching method and DRAM pages, and a computer language having an instruction to switch pages makes the pixel address a relative address from the top of the page. Therefore, it is possible to shorten the tone of the instruction while minimizing the penalty at the time of access beyond the memory page. For example, if 64K bytes are handled as one page, the page can be accessed with 16 bits, so that the operand can be expressed with 2 bytes.

  In addition, the position of the defective pixel on the image coordinates is measured in advance, and a compiler that converts the data from the source program described in the first language system including the coordinates on the image into the computer language is used as the source program. Thus, such a program creation process is performed. Then, by using the compiler that generates the computer language, the instruction order is optimized so that the number of page switching times of the image memory is minimized. Processing such as switching can be reduced, and memory access can be speeded up.

  It also includes computer language execution code for correcting defective pixels on the bit-clogged image memory generated by the compiler, and computer language execution means such as an interpreter or hardware for interpreting and executing the computer language. Imaging means for converting, digitizing means for converting the output of the imaging means into a digital signal, bit-padded memory storing means for bit-packing the digitized signal and transferring it to an image memory, and the imaging means An image obtained by digitizing the output from the digitizing means and bit-packing the image memory by the bit-packing memory storage means is executed on the image memory bit-padded by executing the computer language execution code by the computer language execution means. Missing images obtained from imaging means By the structure of correcting the pixel, to enhance storage efficiency of the memory is possible, it is possible to configure the imaging device that can correct the 且 defective pixels at high speed.

  In other words, it is possible to automatically generate a program that corrects defective pixels by minimizing page switching of the image memory in the computer language of the instruction set, and it is possible to increase the storage efficiency by storing the image by bit-packing it, and at higher speed An imaging apparatus capable of correcting defective pixels can be configured.

  A computer comprising a plurality of different program codes written in a computer language, comprising means for selecting one of the plurality of different program codes according to conditions such as temperature and sensitivity, and executed according to conditions such as temperature and sensitivity By configuring the language program to be determined and executed, it is possible to perform defective pixel correction that can be adapted to the number and quality of defective pixels that change depending on conditions such as ISO sensitivity and temperature.

  There is a division processing unit transfer end notification means that divides one image into a plurality of division processing units and enables transfer end notification from the image sensor for each division processing unit. The computer language program according to claim 1 is configured to wait for notification from the division processing unit transfer end notification means, so that correction of defective pixels is started before transfer of all one image is ended. This makes it possible to significantly reduce the time from the start of image transfer from the image sensor to the end of image defect correction.

  That is, it is possible to start correction of defective pixels before the transfer of the image from the image sensor is completed, and it is possible to improve the performance of the imaging device such as continuous shooting performance and response.

Block diagram showing the configuration of the first embodiment The figure which shows the state which packed the bit of 10 bits into the memory of 16 bits width A diagram showing a state in which 9-bit pixels are packed into a 16-bit width memory The figure which shows the bit padding state of 1st Example The figure which shows the computer language instruction table of a 1st Example Figure showing the register role table Diagram showing an implementation example of the first interpreter heart Diagram explaining the state of the stack after executing the program in Listing 1 Diagram showing List 1 of basic program assembler notation Diagram explaining classification of defective pixel correction across page boundaries The figure which shows list 2 of the assembler notation example of the program with few page switching The figure which shows the state of the stack after execution of address 0006 of list 2 The figure which shows the state of the stack after execution of address 0012 of list 2 The figure which shows the state of the stack after execution of address 0015 of list 2 The figure which shows the state of the stack after execution of address 001D of list 2 The figure which shows the state of the stack after execution of address 0006 of list 3 The figure which shows list 3 of the assembler notation example of the program which includes the swap instruction The figure which shows the state of the stack before and after execution of the SWAP instruction of list 3 Diagram showing the state of pixels across pages Figure showing page boundary access instruction list Diagram explaining the operation of the compiler Compiler block diagram The figure which shows a defective pixel address type top level BNF type The figure which shows the BNF type | formula of Formula pattern D The figure which shows the analysis tree of the continuous pattern D The figure which shows the BNF type | formula of formula pattern A, B, C, N The figure which shows the analysis tree of continuous pattern A The figure which shows the list 4 of the program for demonstrating the principle of the analysis part of FIG. The figure explaining how to express a defective pixel position coordinate The figure explaining the interpolation method of a defective pixel

Explanation of symbols

1 CPU
2 Main memory 3 ROM
4 Image processor 6 Image memory

Claims (4)

In an image processing apparatus that corrects defective pixels included in image data obtained by an imaging unit,
Means for converting the image data obtained from the imaging means into digital image data and bit-packing it into an image memory;
Based on the position data of the defective pixel to be corrected from the image memory, a plurality of digital image data around the defective pixel is read out, and the defective pixel is corrected by performing an interpolation operation, and the corrected image data is written to the image memory. Processing means,
The image memory is logically handled in units of pages, and when the correction processing unit reads out the peripheral digital image data used for the interpolation operation of the defective pixel, the peripheral digital image data crosses the page boundary of the image memory. If present , the image processing apparatus controls the reading order so that the number of page switching is minimized . 2. A division processing unit transfer end notification unit that divides one image into a plurality of division processing units and enables transfer end notification from the imaging unit to the image memory for each division processing unit. An image processing apparatus configured to start a defective pixel correction operation by the correction processing unit in response to a notification from the divided processing unit transfer end notification unit in accordance with the divided single processing unit . An image processing method for correcting defective pixels included in image data obtained by an imaging means,
A transfer step of converting the image data obtained from the imaging means into digital image data and filling the image memory with bits;
Based on the position data of the defective pixel to be corrected from the image memory, a plurality of digital image data around the defective pixel is read out, and the defective pixel is corrected by performing an interpolation operation, and the corrected image data is written to the image memory. Processing steps,
The image memory is logically handled in units of pages, and when the peripheral digital image data used for the interpolation calculation of the defective pixel is read out, the peripheral digital image data straddling the page boundary of the image memory is read. If present , the image processing method is characterized in that the reading order is controlled so that the number of page switching times is minimized . 4. The division processing unit transfer end notification step according to claim 3, wherein one image is divided into a plurality of division processing units, and transfer completion notification to the image memory is possible for each division processing unit from the imaging unit. A defective pixel correction operation in the correction processing step is started in response to a notification from the divided processing unit transfer end notification step in accordance with the divided single processing unit .

Images