JP6415086B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  As an example of processing in which a raw image before image processing has a different color scheme for each pixel, there is image processing of a digital camera. For example, in image processing of a digital camera, an electrical signal is obtained by photoelectrically converting optical information by an image sensor (image sensor), and image processing is performed as digital data obtained by A / D conversion (analog / digital conversion). Then, it is recorded as a captured image.

  In image processing of digital data by a digital camera, the attribute of input pixel data depends on the specifications of the image sensor. For example, a Bayer array, which is an example of a color array for each pixel, obtains each color information through optical information into an array of color filters of R / Gr / Gb / B pixels arrayed in a grid pattern on the light receiving portion of the image sensor. is there. Hereinafter, the R pixel is a pixel having red information, the B pixel is a pixel having blue information, and the G pixel is a pixel having green information. Note that G pixels existing on a horizontal line having R pixels in one screen of the Bayer array are indicated as Gr pixels, and G pixels existing on a horizontal line having B pixels are indicated as Gb pixels.

  Recently, an image sensor in which color arrangement information different from the Bayer arrangement is set for each pixel has been put into practical use. For example, in Patent Document 1, in addition to a filter that transmits only light of a specific wavelength such as RGB as shown in FIG. 9A, light in the visible light range is widely transmitted as a white (W) pixel. An image sensor having a pixel array incorporating filters in the array is described.

  The Bayer array is a periodic array in units of 2 × 2 pixels, and moire or false colors may be generated during processing of a striped pattern or the like. As a countermeasure, a method of suppressing moire and false colors by inserting an optical low-pass filter in front of the image sensor is taken, but the resolution as an image is deteriorated. In order to solve the Bayer array problem, Non-Patent Document 1 discloses an image sensor having a pixel array as shown in FIG. In the 6 × 6 color array as shown in FIG. 9B, the array structure has a non-periodicity with respect to the Bayer array, and moire can be reduced. Further, since there are always R pixels, G pixels, and B pixels in the vertical direction, there is an effect of suppressing generation of false colors and making an optical low-pass filter unnecessary.

JP 2011-55038 A

"APS-C size 16.3 million pixels" X-Trans CMOS ", [online], FUJIFILM Corporation, [Search May 16, 2012], Internet <URL: http: //fujifilm.jp/personal/digitalcamera /x/fujifilm_x_pro1/features/index.html>

  In the image processing for an image in which the color arrangement information for each pixel is a two-dimensional arrangement repetitive pattern, the target pixel and the reference pixel in the processing before the image data synchronization (synonymous with demosaicing, interpolation processing, etc.) It is necessary to grasp the color information. That is, it is necessary to grasp the color of the target pixel. Since the image data obtained by the Bayer array image sensor has a 2 × 2 color array repetition pattern, if the position information (or color information) of the target pixel can be grasped, the color of the surrounding reference pixel region Sequence relationships can be easily determined.

  However, in the 6 × 6 color information array as shown in FIG. 9B, there are many combinations of the color information array of the target pixel and the reference pixel for each progress of scanning of the image processing, and it is difficult to determine. is there. When pixel defect correction is performed before image data is synchronized, if the reference pixel area for generating the correction value of the target pixel (defective pixel) is large, the difficulty of processing increases and the circuit scale increases. This leads to an increase (high cost), a long design period, and a decrease in processing throughput.

  An object of the present invention is to provide an image processing apparatus capable of obtaining a color arrangement in a reference range according to the progress of scanning of image processing in image processing for an image having color arrangement information as a repeated pattern.

An image processing apparatus according to the present invention includes a first register that holds a minimum pattern of color arrangement information that is repeated in a pixel arrangement of an image sensor that photoelectrically converts optical information to obtain an electrical signal, and 2 in a row direction and a column direction. Color array information of a reference range having a dimensional array structure and including a target pixel and a reference pixel to be referred to in image processing for the target pixel, based on the color array information of the minimum pattern held by the first register A second register for holding the pixel, a control unit for updating the color arrangement information held by the second register in accordance with the target pixel, and a pixel based on the color arrangement information held by the second register selected, have a processing means for performing image processing on the pixel of interest, the control means, the row direction the color sequence information the second register is stored for each horizontal scanning progress of the image processing Control is performed so that the color arrangement information held in the second register is shifted in the column direction for each vertical scanning progress of the image processing, and the shift control of the color arrangement information is performed using a barrel shifter. The color arrangement information is circulated .

  According to the present invention, in the image processing for an image in which the color arrangement information becomes a repetitive pattern, it is possible to easily obtain the color arrangement information in the reference range corresponding to the progress of the image processing scan.

It is a figure which shows the structural example of the imaging device which has an image processing apparatus by embodiment of this invention. It is a figure which shows the structural example of the damage correction process part by this embodiment. It is a timing chart of data transfer I / F. It is a figure which shows the circuit mounting example of data transfer I / F. It is a figure for demonstrating the data storage image in a pattern register. It is a figure for demonstrating the data storage image in an operation register. It is a figure which shows the state transition of the data stored in the operation register in this embodiment. It is a timing chart which shows the example of a drive of the signal concerning control of an operation register. It is a figure showing an example of color arrangement information in an image sensor.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus (digital camera) having an image processing apparatus according to an embodiment of the present invention. In FIG. 1, a functional unit related to still image processing is shown in the configuration of the imaging apparatus (digital camera).

  A CPU (Central Processing Unit) 100 controls the entire imaging apparatus by reading and executing a program stored in a ROM (Read Only Memory) or the like (not shown). For example, the CPU 100 makes settings regarding the cycle of the processing frame and instructs the synchronization signal generation unit 101 to start execution. The synchronization signal generator 101 supplies digital data capture timing to a FIFO (First In First Out) 106 and supplies timing to the timing pulse generator 102. These timings include horizontal switching (horizontal synchronization signal) and vertical switching (vertical synchronization signal) of frame data.

  The imaging processing unit 103 includes an imaging element in which color filters are arranged in a color array that is a two-dimensional array repeating pattern and photoelectrically converts optical information into an electrical signal. The imaging processing unit 103 performs charge accumulation and charge (or potential) transfer, and the timing is received from the timing pulse generation unit 102.

  The analog processing unit 104 performs processing (correlated double sampling, gain adjustment, etc.) after converting the electric charge obtained by photoelectric conversion into a potential. The A / D conversion (analog / digital conversion) unit 105 digitizes the clamped analog signal according to the reference potential. The timing pulse generation unit 102 also supplies timing to the analog processing unit 104 and the A / D conversion unit 105. Recently, a column A / D type sensor has been put into practical use, and in such an image pickup means, the output of the image pickup element is subjected to A / D conversion processing (103 to 105 shown in FIG. 1) on-chip. The integration of sensor devices is progressing.

  The FIFO 106 is a buffer for buffering data transfer timing. Timing adjustment relies on a blanking period during frame processing. The shading correction unit 107 executes shading correction in the frame data in order to suppress the influence of the characteristics of the amplifier in the image pickup signal transfer path or reduce the influence of various characteristics (color shading, etc.) of the image pickup section. To do.

  The flaw correction processing unit 108 is an example of the image processing apparatus according to the present embodiment, and executes correction processing for pixel defects (defective pixels, flawed pixels). In addition to fixed pattern scratches that become white spots and black spots, pixel defects include scratches with sensitivity-dependent scratches and fluctuations. Pixel defects targeted by the scratch correction processing unit 108 in this embodiment (Scratches) are optional.

  A white balance (WB) processing unit 109 performs gain adjustment (white balance adjustment) between each color pixel when a color image is received. The pixel interpolation unit 110 performs a synchronization process (demosaic, interpolation, etc.) of each color data. For example, the pixel interpolation unit 110 performs synchronization processing and generates R / G / B planes.

  The color difference signal processing unit 111 generates color difference signals (Cr and Cb signals) from the R / G / B signals. Here, the color difference signal Cr (Cb) is defined by R−Y (B−Y, Y indicates luminance), but the human visibility curve (roughly Y = 0.6G + 0.3R + 0. 1B) to G may be used as RG (BG) as simple luminance. The color γ processing unit 112 performs γ processing on the obtained color signal. The chroma / knee processing unit 113 adjusts the saturation gain after the γ processing by the color γ processing unit 112.

  The luminance signal processing unit 114 generates a luminance signal. The luminance γ processing unit 115 performs γ processing on the obtained luminance signal. Each of the luminance signal and the color difference signal (color signal) generated so far is stored in the temporary storage unit 116. As the temporary storage unit 116, for example, an SDRAM capable of handling a large amount of data at high speed may be used. The storage controller 117 performs write control and read control on the temporary storage unit 116.

  FIG. 2 is a diagram illustrating a configuration example of a flaw correction processing unit as the image processing apparatus according to the present embodiment. In FIG. 2, in addition to the flaw correction processing unit 200, a line buffer (221 to 226) for providing reference line data for obtaining reference pixels to the flaw correction processing unit 200 in addition to the data of the target pixel. It shows. The pixel defect (scratched pixel) correction by the scratch correction processing unit 200 is performed, for example, as part of image data correction processing in the imaging apparatus.

  The flaw correction processing unit 200 includes a correction processing unit 201, a frame control unit 202, a pattern register 203, an operation register 204, a slice controller 205, barrel shifters 206 and 207, and a reference image register 209. The flaw correction processing unit 200 includes AND gates (logical product operation circuits) 210, 211, and 212, and an OR gate (logical sum operation circuit) 213.

  The signal SIG100 is input data, the signal SIG107 is a valid signal indicating that the input data SIG100 is valid, and the signal SIG108 is a stop signal that requests an immediate stop of the input data SIG100 with respect to the previous stage. is there. The signal SIG120 is output data, the signal SIG121 is a valid signal indicating that the output data SIG120 is valid, and the signal SIG122 is a stop requesting an immediate stop of the output data SIG120 from the subsequent processing. Signal.

  Here, the data transfer interface (I / F) of the flaw correction processing unit 200 will be described with reference to FIGS. FIG. 3 defines the data transfer I / F of the flaw correction processing unit 200. FIG. 3 is a timing chart of the data transfer I / F. In FIG. 3, a signal SIG301 is a clock <CLK> (not shown in FIG. 2) of the synchronous circuit. The signal SIG302 is a valid signal <Valid> indicating data validity. The signal SIG107 corresponds to the data input side and the signal SIG121 corresponds to the data output side in the flaw correction processing unit 200. The signal SIG303 is transfer data <Data>, and the signal SIG100 corresponds to the data input side in the flaw correction processing unit 200, and the SIG120 corresponds to the data output side. The signal SIG304 is a stop signal <Stop> that requests an immediate stop of data transfer, and corresponds to the signal SIG108 on the data input side and the SIG122 on the data output side in the flaw correction processing unit 200.

  In the present embodiment, it is assumed that the valid signal SIG302 is a value “1” when the data is in a valid state and a value “0” when the data is in an invalid state. The stop signal SIG304 is a value “1” when an immediate stop of the transfer data SIG303 is requested (in a valid state), and a value “0” when an immediate stop is not requested (in an invalid state). ”.

  FIG. 4 is a circuit diagram showing an example in which the I / F of FIG. 3 is incorporated in the data transfer I / F. FIG. 4 shows an example of connection and mounting of data, valid signals, and stop signals between the front module 401 and the rear module 402. The internal logic of the pre-stage module 401 is 403, and the internal logic of the post-stage module 402 is 404.

  When outputting a valid signal indicating that the output data of the internal logic 403 is valid, the AND gate 405 is valid when there is a request for immediate stop from the subsequent module 402 (the stop signal SIG304 is “1”). The signal SIG302 is set to the value “0”. When the valid signal SIG302 is transmitted to other modules in parallel, it can indicate that the current data is in a stopped (invalid) state. The AND gate 406 is for efficiently transmitting the immediate stop request from the rear module 402 to the previous stage. When the preceding module 401 does not transfer valid data, there is no need to react to a stop request from the succeeding stage, so that the immediate stop request from the succeeding module 402 is not transmitted at that time. .

  Here, when a plurality of subsequent modules are connected to the I / F of the output data of the upstream module 401, the data and the valid signal may be distributed as they are. Further, the stop signal may be input to the AND gate 406 by taking OR (logical sum) of all the stop signals before the AND gate 406. In the configuration shown in FIG. 2, the AND gate 210 realizes the same function as the AND gate 405 shown in FIG. 4, and the AND gate 211 realizes the same function as the AND gate 406 shown in FIG.

  Returning to FIG. 2, the flaw correction processing unit 200 will be described. In the present embodiment, for example, the flaw correction processing unit 200 uses a result value obtained by performing median filter processing (processing for selecting a median value), for example, several peripheral reference pixels having the same color as the target pixel (defective pixel). Then, correction processing is performed so as to replace the pixel value of the target pixel.

  The reference pixel register 209 holds each pixel value within the range of M × N pixels centering on the target pixel (defective pixel) to be corrected. Therefore, it is necessary to prepare data of several lines above and below the line where the target pixel exists, which is the center in the frame processing, while maintaining a synchronous relationship. The reference pixel register 209 can be realized, for example, by adopting a shift register configuration with a flip-flop circuit that holds data of each line.

  In the present embodiment, an M × N reference range is described as 7 × 7. Therefore, as shown in FIG. 2, the data input of each line to the reference pixel register 209 is each of the signals SIG100 to SIG106 (for 7 lines). A line having a target pixel (defective pixel) to be corrected is a line of the signal SIG103. Each of the signal SIG100 input from the previous stage and the signals SIG101 to SIG106 input from the horizontal line buffers 201 to 206 holding data for one horizontal line have a bit width of data for one pixel.

  The reference pixel register 209 according to the present embodiment sends data for 7 × 7 pixels in one processing cycle (corresponding to one clock in the case of a synchronous circuit) to the correction processing unit 201 that performs flaw correction processing. Each of the signals SIG110 to SIG116 shown in FIG. 2 indicates data of each line. Therefore, for example, the data width of the signal SIG110, which is one of the output data signals of the reference pixel register 209, has a bit width of data for seven pixels. The same applies to the signals SIG111 to SIG116.

  The reference pixel data supplied by the signals SIG100 to SIG106 is taken into the reference pixel register 209 when the valid signal SIG107 is in the value “1”. The fetch instruction status in the reference pixel register 209 is composed of the output of the AND gate 212, that is, the relationship between the valid signal SIG107 from the previous stage and the stop signal SIG108 to the previous stage.

  The frame control unit 202 monitors the processing state of the frame by counting the number of valid data, and updates the state of the operation register 204 for the reference pixel. The frame control unit 202 also generates a valid signal for the subsequent stage in consideration of the entire latency of the flaw correction processing unit 200. The generated valid signal SIG135 is sent as a valid signal SIG121 to the subsequent stage via the AND gate 210. The sending of the valid signal SIG121 to the subsequent stage via the AND gate 210 is performed in view of the immediate stop request from the subsequent stage, that is, the state of the stop signal SIG122.

  The operation register 204 has a two-dimensional array structure in the row direction and the column direction, and loads the contents held in the pattern register 203 at the start of image processing for each frame. Here, the number of stored data in the pattern register 203 as the first register and the operation register 204 as the second register is not equivalent. For example, as in the present embodiment, the operation register 204 has more array elements than the pattern register 203, and when the range on the operation register 204 side is wide, the contents of the pattern register 203 are made to circulate. It is expanded in the operation register 204 for such a relationship. The pattern register 203 holds a minimum pattern of color arrangement information (a pattern of color arrangement information that is repeated as a basic unit) in the imaging device, which is a two-dimensional arrangement repeating pattern.

  The slice controller 205 controls updating of data in the operation register 204 in batches in units of rows or columns. The operation register 204 updates the data storage position in units of rows or columns in at least one of the ROW direction and the COLUMN direction in accordance with the data load instruction of the slice controller 205. Details of the behavior of the input / output signals of the slice controller 205 will be described later with reference to FIG. The column shifter 206 in the column (COLUMN) direction and the barrel shifter 207 in the row (ROW) direction replenish data that protrudes in the shift direction as a result of data shift control in the operation register 204 to the insufficient data on the shift start side.

  FIGS. 5A to 5C are diagrams for explaining data storage images in the pattern register 203. FIG. In the present embodiment, a case of a 6 × 6 color arrangement configuration is shown as an example. FIG. 5A shows a register arrangement of the pattern register 203. The data stored in the pattern register 203 is illustrated in a two-dimensional array, and the positional relationship is represented by numbers in a frame with the upper left as a base point. In both the horizontal direction and the vertical direction, the base point is 0, the right side of the number is represented as a transition in the horizontal direction, and the left side is represented as a transition in the vertical direction. That is, the notation “00” is the upper left end, the notation “05” is the upper right end, the notation “50” is the lower left end, and the notation “55” is the lower right end. FIG. 5B shows a color arrangement of pixels in which the color arrangement pattern of the image sensor is indicated at the corresponding storage position of the pattern register 203. The repetition of the illustrated color arrangement forms a color arrangement on the entire surface of the image sensor.

  FIG. 5C is a diagram showing the color number storage result in the pattern register 203. In order to store {R, G, B} information, which is color information, as numerical values in a hardened pattern register, here, as color numbers, {0 (= G), 1 (= R), 2 (= B)} Is defined. Similar numerical values are stored in the pattern register 103 shown in FIG. In the example shown in FIG. 5C, the value “0” corresponding to the G pixel corresponds to the upper left end array “00” of the pattern register 203, and the value “0” corresponds to the R pixel in the pixel array “01” on the right side of the pattern register 203. 1 ″ is stored as a register value. Similarly, the color numbers corresponding to the respective color arrangement patterns are stored in the other arrangements.

  6A to 6D are diagrams for explaining a data storage image in the operation register 204. FIG. In the present embodiment, an operation register 204 corresponding to a 7 × 7 reference pixel range is shown as an example. FIG. 6A shows the register arrangement of the operation register 204. The data stored in the operation register 204 is illustrated in a two-dimensional array, and the positional relationship is represented by numbers in a frame with the upper left as a base point. In both the horizontal direction and the vertical direction, the base point is 0, the right side of the number is represented as a transition in the horizontal direction, and the left side is represented as a transition in the vertical direction. That is, the notation “00” is the upper left end, the notation “06” is the upper right end, the notation “60” is the lower left end, and the notation “66” is the lower right end.

  FIG. 6B shows a reflection image of the contents of the pattern register 203 to the operation register 204. Reflecting the contents of the pattern register 203 to the operation register 204 is performed so that all elements (arrays) in the operation register 204 are filled. This process may be performed once at the start of frame scanning. The status of the operation register 204 during the frame processing can be updated by updating the data with the output values of the barrel shifters 206 and 207 without requiring the pattern register 203 to be referenced.

  An instruction to start frame scanning is transmitted from the frame control unit 202 to the operation register 204 via the signal SIG 131 and the signal SIG 136. If this process is defined as initialization of the operation register 204, the initialization of the operation register 204 may be performed at the end of one frame process. However, at the beginning of the frame processing, it is necessary to generate an event by receiving an initialization instruction signal (signal SIG109 shown in FIG. 2) from the CPU 100 shown in FIG.

  In the present embodiment, the pattern register 203 is a 6 × 6 two-dimensional array, and the operation register 204 is a 7 × 7 two-dimensional array. Therefore, when the contents of the pattern register 203 are reflected in the operation register 204, the shortage from the pattern register 203 is stored in the operation register 204 by circulating the contents of the pattern register 203. For example, the same content as the column of hpos = 0 is expanded in the column of hpos = 6 shown in FIG. Further, the content of the pattern register value developed in the operation register 204 is determined by the position of the pattern register 203 from the upper left end of the image sensor. The example shown in FIG. 6 (B) is the contents expanded into a 7 × 7 array with the information of the position “01” in FIG. 6 (A) as the upper left end.

  FIG. 6C shows a reflection image of the contents of the pixel color array, which is a reflection result of the pattern register 203 shown in FIG. In FIG. 6D, the color number definition {0 (= G), 1 (= R), 2 (= B)} is applied to the result of reflecting the contents of the color array shown in FIG. An example of color number storage in the operation register 204 in a state where numerical values are stored is shown.

  In FIG. 6D, 900 to 906 are images in the case of handling registers grouped in columns. For example, 900 in FIG. 6D is a register array {“00”, “10”, “20”, “30”, “40”, “50”, “60” in FIG. "} Indicates each stored data. Similarly, in FIG. 6D, reference numerals 910 to 916 denote images in the case of handling registers grouped in units of rows. For example, 910 in FIG. 6D is a row_0 row unit, and the register array {“00”, “01”, “02”, “03”, “04”, “05”, “06” in FIG. } Indicates each stored data.

  In the present embodiment, the update contents of the operation register 204 shown in FIG. 6D are transmitted to the correction processing unit 201 via the signal SIG 134 shown in FIG. The correction processing unit 201 obtains the current color arrangement information of the target pixel and the reference pixel based on the contents of the operation register 204, performs correction processing with reference to data of the same color pixel as the target pixel, and obtains a correction result. . As information on whether or not the current pixel of interest is a defective pixel, for example, positional information of the defective pixel may be input to the frame control unit 202 (I / F is not shown. In this case, the information may be transmitted from the CPU 500. Possible). Further, for example, a specific value (maximum value, minimum value, etc.) of pixel data may be given a meaning indicating a pixel defect (in this case, the data value needs to be decoded; not shown).

  In this embodiment, in accordance with an instruction from the slice controller 205, the data storage state in the operation register 204 is shifted (updated) in the units shown in FIGS. 7A to 7D during the progress of image processing in the frame. ) FIG. 7A to FIG. 7D are diagrams illustrating state transitions of data stored in the operation register 204 having a 7 × 7 arrangement according to this embodiment. 7A and 7C show state transitions in the horizontal scanning direction, and FIGS. 7B and 7D show state transitions in the vertical scanning direction.

  The frame control unit 202 outputs a status signal SIG132 indicating completion of horizontal one pixel processing to the slice controller 105 every time one horizontal pixel is scanned (every horizontal scanning progress). The slice controller 105 outputs a load signal SIG139 so as to hold the output result of the barrel shifter 206 to the operation register 204. Thereby, the contents of the column_0 group 900 shown in FIG. 6D are updated with the contents of the immediately preceding column_1 group 901, and the contents of the column_1 group 901 are updated with the contents of the immediately preceding column_2 group 902. Similarly, the contents of the column_2 group 902 to the column_5 group 905 are updated with the contents of the immediately preceding column_3 group 903 to the column_6 group 906, and the contents of the column_6 group 906 are updated with the contents of the immediately preceding column_1 group 901. FIG. 7C illustrates this state.

  When scanning for one horizontal line is completed during frame scanning (every vertical scanning progress), the frame control unit 202 outputs a horizontal scanning completion status signal SIG 133 to the slice controller 105. The slice controller 105 holds the output results of the barrel shifters 206 and 207 in the operation register 204 a plurality of times, so that each of the load signals SIG139 and SIG140 over a plurality of cycles is initialized to the processing state at the head of the next line. Output. Accordingly, for example, the contents of the row_0 group 910 illustrated in FIG. 6D are updated with the contents of the immediately preceding row_1 group 911, and the contents of the row_1 group 911 are updated with the contents of the immediately preceding row_2 group 912. Similarly, the contents of the row_2 group 912 to the row_5 group 915 are updated with the contents of the immediately preceding row_3 group 913 to the row_6 group 916, and the contents of the row_6 group 916 are updated with the contents of the immediately preceding row_1 group 911. FIG. 7D illustrates this state.

  FIG. 8 is a timing chart showing an example of driving signals related to the control of the operation register 204. In FIG. 8, each of the signals SIG131 to SIG140 conveys binary digital information, the value “0” is in the idle state, and the value “1” conveys the request event and the load status.

  A signal SIG 109 is a frame processing start request from the CPU 100. Based on this request, initialization of the frame control unit 202 and initialization of the operation register 204 are started. The frame control unit 202 sends an event of the signal SIG 131 to the slice controller 205 in order to initialize the operation register 204. A period A shown in FIG. 8 is an initialization phase of the operation register 204. The operation during this period A will be described.

  When the slice controller 205 receives the event of the signal SIG 131 (high pulse in the illustrated example), the slice controller 205 sends the event of the signal SIG 136 to the operation register 204. The event (high pulse in the illustrated example) of the signal SIG 136 is a request event for copying (loading) the contents of the pattern register 203 to the operation register 204. When the operation register 204 receives the event of the signal SIG 136, the range of the 6 × 6 array (“55” is the lower right end) with the array element “00” in the operation register 204 as the upper left end is set to the value of the pattern register 203. Update with.

  The slice controller 205 generates the event of the signal SIG137 after sending the event of the signal SIG136. The operation register 104 receives the event of the signal SIG137 and updates only the array portion corresponding to column_6 to the output value of the barrel shifter 206. The column direction transition of the barrel shifter 206 at this time is as shown in FIG. After sending the event of the signal SIG137, the slice controller 205 generates the event of the signal SIG138. The operation register 104 receives the event of the signal SIG138, and updates only the array portion corresponding to row_6 to the output value of the barrel shifter 207. The transition in the row direction of the barrel shifter 207 at this time is as shown in FIG.

  At this time, the entire array of operation registers 204 is stored in a state where the pattern register 203 is expanded to a 7 × 7 array. Thereafter, the initialization is completed by circulating the pattern of the operation register 204 at the actual arrangement position of the head of the frame data. The signal SIG139 is a column direction shift request status of the operation register 204. The column shift by the signal SIG139 takes a transition as shown in FIG. 7C (column_6 is updated with the state of column_1 at that time). The signal SIG 140 is a row direction shift request status of the operation register 204. The row shift by the signal SIG140 takes a transition as shown in FIG. 7D (row_6 is updated with the state of row_1 at that time).

  A period B and a period D illustrated in FIG. 8 indicate execution periods of one line processing during frame processing. When the flaw correction processing unit 200 according to the present embodiment receives the data SIG 100 and the valid signal “valid” of the value “1” from the previous stage in a state where there is no immediate stop request for the previous stage, the frame control unit 202 sends the data to the slice controller 205. Signal SIG132 is sent out. In the state where the signal SIG132 is “1”, this indicates that data processing is proceeding in the horizontal direction (in units of pixels). At this time, the slice controller 205 sends a load signal SIG139 to the operation register 204 (loads with the value “1”).

  The operation register 204 is a synchronization circuit based on a clock (not shown), and loads the output of the barrel shifter 206 every clock cycle when the load signal SIG139 is received. The barrel shifter 206 shifts the array data in the register in units of columns (900, 901, 902, 903, 904, 905, 906 shown in FIG. 6D) as described above.

  The period C and the period E shown in FIG. 8 indicate the execution period of the update process of the operation register 204 when the one-line process is completed during the frame process. When the horizontal scanning for one line is completed during the frame processing, the frame control unit 202 sends a line completion event by a signal SIG133. When the slice controller 205 receives the event of the signal SIG133, the slice controller 205 generates a load event of the signal SIG139 a plurality of times so as to return the data arrangement in units of columns to the processing position at the head of the line. This multiple-time event is a result value of {the number of horizontal pixels mod the number of operation register arrays} (the remainder when the number of horizontal pixels is divided by the number of operation register arrays), and can be handled as a fixed value. In this embodiment, since the arrangement of the operation registers 204 is 7 × 7, if the number of horizontal pixels is 3000 (3000 mod 7 = 4), the shift is performed three times in the column direction (7-4 = 3) Thus, the column state at the beginning of the line is obtained.

  The slice controller 205 subsequently generates an event of the signal SIG140. When the line processing is completed, the operation register 204 is shifted in the row direction as shown in FIG. If the image sensor is not skipped in the vertical direction, the row shift event may be one time (one cycle as a synchronization circuit).

  When the frame processing is completed, the event of the signal SIG 131 is generated by the frame control unit 202, so that the processing of the period A can be performed. Then, even if there is no initialization instruction from the CPU 100, a standby state for the next frame process can be created. Further, the signal SIG125 shown in FIG. 2 is a stop signal for requesting an immediate stop to the previous stage. In each of period A, period C, and period E shown in FIG. 8, as in the state of the value “1” of the signal SIG304 shown in FIG. A request to cancel the transfer of. This immediate stop request (signal SIG125) is ORed (ORed) with the stop signal (output of the AND gate 211) from the subsequent stage and transmitted to the previous stage. Since the signal is transmitted to the frame control unit 202 via the AND gate 212, the output of the AND gate 212 is clock-synchronized by a flip-flop or the like in the frame control unit 202 so as not to loop as a combination logic.

  In the example illustrated in FIG. 8, the description is simplified on the assumption that there is no immediate stop request from the subsequent stage (the signal SIG 122 is always in the value “0” state). For example, when there is a request for an immediate stop in the period B and the period D shown in FIG. 8, the state of the valid signal SIG107 from the previous stage is masked by the AND gate 212, so that the states of the signals SIG132 and SIG139 are withdrawn. .

  In this way, in image processing that requires a reference pixel, the pattern register 203 that holds the color arrangement information of the image sensor and the color arrangement relationship of the reference range based on the pattern register 203 according to the change in the position of the pixel of interest The operation register 204 to be incorporated is incorporated into the system configuration. This makes it possible to easily obtain a color array of reference pixels according to the progress of processing even in a color array having a repetitive pattern in a two-dimensional array different from the Bayer array. Processing of imaging data having (color arrangement) can be performed. In the above description, the pattern register 203 is a 6 × 6 two-dimensional array and the operation register 204 is a 7 × 7 two-dimensional array. However, this is an example, and the present invention is limited to this. It is not a thing.

(Other embodiments of the present invention)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

100: CPU 103: Imaging processing unit 108, 200: Scratch correction processing unit 201: Correction processing unit 202: Frame control unit 203: Pattern register 204: Operation register 205: Slice controller 206, 207: Barrel shifter 209: Reference pixel register

Claims (10)

A first register that holds a minimum pattern of color arrangement information that is repeated in a pixel arrangement of an image sensor that photoelectrically converts optical information to obtain an electrical signal;
The pixel of interest has a two-dimensional arrangement structure in the row direction and the column direction, and includes a pixel of interest and a reference pixel to be referred to in image processing for the pixel of interest based on the color arrangement information of the minimum pattern held by the first register A second register holding color arrangement information of the reference range
Control means for updating color arrangement information held by the second register in accordance with the pixel of interest;
The second register is selected pixels based on the color arrangement information held, have a processing means for performing image processing on the pixel of interest,
The control means shifts the color arrangement information held in the second register in the row direction for each horizontal scanning progress of the image processing, and performs the second processing for each vertical scanning progress of the image processing. An image processing apparatus that controls to shift the color arrangement information held in a register in a column direction, and in the color arrangement information shift control, the color arrangement information is circulated using a barrel shifter .   The control means performs control so as to shift the storage position of the color arrangement information held in the second register every time the position of the target pixel is changed by scanning of image processing. The image processing apparatus according to 1. A first register holding color arrangement information of the M × N pixels of an image having a color arrangement repeated with M × N pixels;
Based on the color arrangement information of the M × N pixels held by the first register, the second holding color arrangement information of a reference range including the target pixel and a reference pixel to be referred to in image processing for the target pixel. And the register
Control means for updating color arrangement information held by the second register in accordance with the pixel of interest;
The second register is selected pixels based on the color arrangement information held, have a processing means for performing image processing on the pixel of interest,
The first register and the second register are registers that hold the color array information in a two-dimensional array;
The second register has more array elements than the first register;
An image processing apparatus comprising: circulating the color arrangement information of the M × N pixels held in the first register; and holding the color arrangement information of the reference range in the second register . The second register has a two-dimensional array structure in a row direction and a column direction,
The control means shifts the color arrangement information held in the second register in the row direction for each horizontal scanning progress of the image processing, and performs the second processing for each vertical scanning progress of the image processing. The image processing apparatus according to claim 3, wherein the color arrangement information held by the register is controlled to shift in a column direction. 5. The image processing apparatus according to claim 4 , wherein the control means uses a barrel shifter to rotate the color arrangement information in shift control of the color arrangement information held by the second register. The image processing is correction of pixel defects;
The pixel of interest is a defective pixel;
The image processing apparatus according to any one of claim 1 to 5, wherein said reference pixels, which is a pixel array range including the pixels to be used in the correction of at least the defective pixel. A first register that holds a minimum pattern of color arrangement information that is repeated in a pixel arrangement of an image sensor that photoelectrically converts optical information to obtain an electrical signal; and a two-dimensional arrangement structure in a row direction and a column direction, Based on the color arrangement information of the minimum pattern held by one register, a second register holding color arrangement information of a reference range including the target pixel and a reference pixel to be referred to in image processing for the target pixel; An image processing method for an image processing apparatus comprising:
After the second register holds the color arrangement information of the reference range and the second register holds the color arrangement information based on the color arrangement information of the minimum pattern held by the first register A process of updating the color arrangement information held by the second register in accordance with the pixel of interest, and selecting a pixel based on the color arrangement information held by the second register, and performing image processing on the pixel of interest have a and the applying process,
In the control step, the color arrangement information held by the second register is shifted in the row direction for each horizontal scanning progress of the image processing, and the second scanning is performed for each vertical scanning progress of the image processing. An image processing method comprising: controlling the color arrangement information held in the register in a column direction to shift the color arrangement information using a barrel shifter in the shift control of the color arrangement information . A first register that holds a minimum pattern of color arrangement information that is repeated in a pixel arrangement of an image sensor that photoelectrically converts optical information to obtain an electrical signal; and a two-dimensional arrangement structure in a row direction and a column direction, Based on the color arrangement information of the minimum pattern held by one register, a second register holding color arrangement information of a reference range including the target pixel and a reference pixel to be referred to in image processing for the target pixel; A program for causing a computer to execute an image processing method of an image processing apparatus,
A first step of causing the second register to hold the color arrangement information of the reference range based on the color arrangement information of the minimum pattern held by the first register;
A second step of updating the color arrangement information held by the second register in accordance with the target pixel after the color arrangement information is held in the second register; and the color held by the second register Selecting a pixel based on the array information, and causing the computer to execute a third step of performing image processing on the pixel of interest ;
In the second step, the color arrangement information held in the second register is shifted in the row direction for each horizontal scanning progress of the image processing, and for each vertical scanning progress of the image processing, the color processing is performed. A program for controlling the color arrangement information held in the second register to shift in a column direction, and causing the computer to execute processing for circulating the color arrangement information using a barrel shifter in the shift control of the color arrangement information . A first register for holding color arrangement information in a two-dimensional array, the first register for holding color arrangement information of the M × N pixels of an image having a color arrangement repeated by M × N pixels, and the first register And the reference pixel to be referred to in the image processing for the target pixel is included based on the color array information of the M × N pixels held in the first register. An image processing method of an image processing apparatus having a second register that holds color arrangement information of a reference range,
Based on the color arrangement information of the M × N pixels held by the first register, the second register holds the color arrangement information of the reference range, and the second register holds the color arrangement information. A control step of updating the color arrangement information held by the second register in accordance with the target pixel;
A processing step of selecting a pixel based on the color arrangement information held by the second register and performing image processing on the target pixel;
In the control step, the color arrangement information of the M × N pixels held by the first register is circulated, and the color arrangement information of the reference range is held by the second register. . A first register for holding color arrangement information in a two-dimensional array, the first register for holding color arrangement information of the M × N pixels of an image having a color arrangement repeated by M × N pixels, and the first register And the reference pixel to be referred to in the image processing for the target pixel is included based on the color array information of the M × N pixels held in the first register. A program that causes a computer to execute an image processing method of an image processing apparatus having a second register that holds color arrangement information of a reference range,
A first step of causing the second register to hold the color arrangement information of the reference range based on the color arrangement information of the M × N pixels held by the first register;
A second step of updating the color arrangement information held by the second register in accordance with the target pixel after holding the color arrangement information in the second register;
Selecting a pixel based on the color arrangement information held by the second register, and causing the computer to execute a third step of performing image processing on the pixel of interest;
In the first step, the computer executes a process of circulating the color arrangement information of the M × N pixels held in the first register and causing the second register to hold the color arrangement information of the reference range. Program for.

Images