JP6395398B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and is particularly suitable for performing processing of a pixel of interest with reference to surrounding pixels.

Generally, CMOS and CCD image sensors mounted on digital cameras and the like are composed of color filters of one R (red) color, two G (green) colors, and one B (blue) color. A Bayer array in units of 2 × 2 pixels is often used.
However, in recent years, an image pickup device having a color filter array other than the conventional Bayer array has been commercialized for the purpose of preventing the occurrence of moire. Non-Patent Document 1 discloses an image sensor having a highly non-periodic color filter array in units of 6 × 6 pixels. Due to the improvement of the non-periodicity of the color filter array, the occurrence of moire can be reduced in this color filter array compared to the conventional Bayer array.

In the image processing apparatus, the pixel data output from the image sensor is subjected to edge enhancement processing, correction processing using a low-pass filter, and the like using a one-dimensional or two-dimensional filter. In the process using the filter, when the pixel of interest of the filter is located at the end of the image data when performing the process on the entire image data, a part of the reference pixel does not exist. Here, the target pixel is a pixel at the center position of the filter coefficient, and the reference pixel is a pixel located around other than the center position of the filter coefficient.
In Patent Document 1, when a part of reference pixels for a target pixel does not exist in a process using a filter, the effective pixel data at the end of the image data is copied and extrapolated to the position of the reference pixel that does not exist. It is disclosed.

JP 2006-094160 A

"APS-C size 16.3 million pixels" X-Trans CMOS "", [online], FUJIFILM Corporation, [searched on January 29, 2014], Internet <URL; http://fujifilm.jp/personal/ digitalcamera / x / fujifilm_x_pro1 / features / index.html>

However, when the technique described in Patent Document 1 is used for image data captured by an image sensor having a special color filter array as disclosed in Non-Patent Document 1, pixel data that becomes a copied reference pixel And the pixel data of the copy source may be long.
In general, when the interval between two pixels of image data is long, the subject captured by each pixel is different and the color is also different, so the difference in pixel value of each pixel often increases. Therefore, it is desirable that the reference pixel copied to the image end portion is a pixel located in the vicinity of the image end portion and an effective pixel having the same color as the target pixel.
The present invention has been made in view of such problems, and when copying effective pixel data as reference pixel data for a target pixel, the position of a copy source effective pixel and a copy destination position are determined. The purpose is to shorten the interval.

An image processing apparatus according to the present invention is a color filter having a color arrangement of M × N (M and N are positive integers), and is output from an image sensor having a color filter having a color arrangement other than a Bayer arrangement. And an array of the colors in the color filter in each of the reference pixels when the pixel data is processed, an acquisition means for acquiring the pixel data and an area outside the edge of the image determined by the pixel data And a combination of the same color as the combination of colors arranged by the arranging unit with respect to the pixel block in the reference pixel area, which is an area composed of the reference pixels, so that the repetitive pattern is maintained. The search is performed from the effective pixel area, which is the area of the image based on the pixel data, with the horizontal or vertical direction as the search direction. Yes and searching means, the data of pixels included in the pixel block in the effective pixel region which is searched by the searching means, and copying means for copying the data of pixels included in the pixel block in the reference pixel region, the The search means is the pixel block having the same color combination as the color combination arranged by the arrangement means for the pixel block in the reference pixel area, and the pixel block in the reference pixel area The pixel block closest to is searched from the effective pixel region .

  According to the present invention, when valid pixel data is copied as reference pixel data with respect to a target pixel, the interval between the copy source valid pixel position and the copy destination position can be shortened.

It is a figure which shows the structure of an image processing apparatus. It is a figure which shows the structure of an image processing circuit. It is a figure which shows the structure of a pixel copy process part. It is a figure which shows a color ID address value. It is a figure which shows the 1st example of the result of a pixel copy process. It is a timing chart of each signal in a pixel copy process part. It is a figure which shows the content of the pixel data stored in a buffer memory. It is a figure which shows the 2nd example of the result of a pixel copy process. It is a figure which shows the 3rd example of the result of a pixel copy process. It is a figure which shows a color filter arrangement | sequence. It is a figure which shows a mode that the reference pixel was extended to the outer side of the image edge part. It is a figure which shows a Bayer arrangement | sequence. It is a figure which shows a mode that the pixel copy process was implemented.

Prior to describing the embodiments of the present invention, the technology that is the premise of the embodiments of the present invention will be described.
Here, when one-dimensional filter processing is performed on pixel data of R, G, and B colors output from an image sensor having a color filter array of Bayer array, the pixel data of the same color has a tap number of 5 pixels. The reference pixel expansion process at the image edge will be described as an example. In the one-dimensional filter processing of this example, data for four pixels on the left and right are required centering on the target pixel to be processed.
FIG. 11 shows that pixel data output from an image sensor having a color filter array of Bayer array is copied from the effective pixel data at the image edge by four pixels on the left and right sides, and the reference pixel is extended outside the image edge. It is a figure which shows the concept at the time of processing. Hereinafter, the process of copying the effective pixel data at the image end and expanding the reference pixel to the outside of the image end is referred to as a pixel copy process. FIG. 12 is a diagram showing a Bayer array.
FIG. 11A is a diagram illustrating pixel data for one line that is input to an image processing apparatus (pixel copy processing unit) from an image sensor having a Bayer color filter array. Pixel data from the image sensor is pixel data R0 (1101), G0 (1102), R1 (1103), G1 (1104),..., R98 (1109), G98 (1110), R99 (1111), G99. Input to the pixel copy processing unit in the order of (1112). Here, R * (*: number) represents R pixel data, and G * represents G pixel data.

  A filter processing unit in the subsequent stage of the pixel copy processing unit in the image processing apparatus performs one-dimensional filter processing having the number of taps corresponding to the five pixels. In the one-dimensional filter processing, when the pixel data R0 (1101) becomes the target pixel, pixel data of the same color for two pixels on the left and right with the pixel data R0 (1101) as the center is required. For this reason, the pixel copy processing unit extracts and extracts effective pixel data from the end of the image by four pixels on the left and right sides so that pixels of the same color as the target pixel are included in the left and right pixels with respect to the input pixel data. Output is performed after valid image data is expanded as reference pixel data.

FIG. 11B is a diagram illustrating pixel data for one line after the pixel copy process is performed.
At the left end portion of the image, as shown in FIG. 11B, pixel copy processing for four pixels is performed for each of the effective pixel data R0 (1101) and G0 (1102) at the left end portion of the image. As a result, the pixel data R0 (1121), G0 (1122), R0 (1123), and G0 (1124) is subjected to pixel copy processing with reference pixels being expanded outside the left end of the image. Even when the pixel data R0 (1101) becomes a target pixel in the subsequent filter processing unit by the pixel copy processing, the pixel data R0 (1121) and R0 (1123) are used as reference pixels having the same color as the pixel data R0 (1101). Can be used.

On the other hand, at the right end portion of the image, as shown in FIG. 11B, pixel copy processing for four pixels is performed for each of the effective pixel data R98 (1111) and R99 (1112) at the right end portion of the image. As a result, pixel copy processing is performed using the pixel data R99 (1125), G99 (1126), R99 (1127), and G99 (1128) as reference pixels in a form expanded outside the right end of the image. Even when the pixel data G99 (1112) becomes the target pixel in the subsequent filter processing unit by the pixel copy processing, the pixel data G99 (1126) and G99 (1128) are used as reference pixels having the same color as the pixel data G99 (1112). Can be used.
The pixel copy processing method for pixel data read from a line including pixel data of G color and B color is the same.

FIG. 13 shows pixel data for four pixels on the same line as a reference pixel at the left end of an image in pixel data output from an image sensor having a special color filter array disclosed in Non-Patent Document 1. It is a figure which shows the example which implemented the pixel copy process. FIG. 10 is a diagram showing a color filter array disclosed in Non-Patent Document 1. As shown in FIG.
Here, in consideration of the position of the filter coefficient in the subsequent filter processing after the pixel copy processing, the pixel copy processing of the reference pixel is performed while maintaining the color order of the color filter array.
Therefore, in the first line, the B color pixel data 1301 serving as the reference pixel is copied from the same color B effective pixel data 1202 located at the leftmost end of the image in the same line as the pixel data 1301. It will be processed. The period of the color filter array shown in FIG. 10 is a unit of 6 pixels. Therefore, the interval between the B-color pixel data 1301 serving as the reference pixel subjected to the pixel copy process and the B-color effective pixel data 1302 as the copy source is 6 pixels.

In the sixth line, the R pixel data 1303 serving as the reference pixel is subjected to pixel copy processing from the same color R effective pixel data 1304 located at the left end of the image in the same line as the pixel data 1303. Become a thing. The interval between the R-color pixel data 1303 serving as a reference pixel subjected to the pixel copy process and the B-color effective pixel data 1304 of the copy source is 6 pixels.
On the other hand, in the image sensor having the color filter array of the Bayer array shown in FIG. 12, the period of the color filter array is in units of two pixels. For this reason, as shown in FIG. 11, the interval between the pixel data to be the reference pixel that has been subjected to the pixel copy process at the edge of the image and the pixel data to be the copy source is two pixels.

As described above, when the pixel copy processing described above is performed on the pixel data obtained by the color filter of the color filter array other than the Bayer array, the interval between the reference pixel subjected to the pixel copy processing and the effective pixel of the copy source is set to the Bayer array. Longer than that of time. Therefore, the pixel value difference between the pixel data serving as the reference pixel and the effective pixel as the copy source may be larger than that in the Bayer array. Further, the subsequent filter processing unit that processes the pixel data output from the pixel copy processing unit is also affected by the difference in the pixel value of the pixel data serving as the reference pixel.
Under the problems as described above, the present inventors have arrived at an embodiment of the present invention described below.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating an example of a functional configuration of the image processing apparatus 100. Here, a case where the image processing apparatus 100 is a digital camera will be described as an example.
In FIG. 1, an imaging optical system 101 includes a lens, a diaphragm, etc., and performs focus adjustment and exposure adjustment.
The image sensor 102 has a photoelectric conversion function for converting an optical image into an electrical signal, and is configured by a CCD, a CMOS sensor, or the like. The image sensor 102 is provided with R (red), G (green), and B (blue) color filters having a period of 6 × 6 pixels as shown in FIG. The signal level output from each pixel is a signal level corresponding to any of R, G, and B colors. The light incident on the imaging optical system 101 forms an image on the light receiving surface of the image sensor 102 and is output from the image sensor 102 as an analog image signal.

  The A / D converter 103 converts the analog image signal output from the image sensor 102 into a RAW image signal (digital image data) having a color filter array of 6 × 6 pixels shown in FIG.

The image processing circuit 104 acquires the image data output from the A / D converter 103, and performs development processing such as pixel interpolation processing and color conversion processing described later on the acquired image data.
A memory interface (I / F) 105 writes / reads image data processed by the image processing circuit 104 and various control data to / from a memory 106 constituted by a DRAM.
The CPU 107 manages various controls of the image processing apparatus 100.
The display unit 108 includes a liquid crystal monitor or the like. The display unit 108 displays an image based on the image data processed by the image processing circuit 104 as needed so that the user can observe the state of the subject in real time.

FIG. 2 is a diagram illustrating an example of the configuration of the image processing circuit 104 of the image processing apparatus 100.
The offset correction processing unit 201 and the gain correction processing unit 202 correct an analog characteristic variation (such as a column amplifier characteristic variation) of the image sensor 102 and the A / D converter 103 in order to correct an input pixel signal. Perform offset correction and gain correction.

The pixel copy processing unit 203 of the present embodiment performs pixel copy processing described later.
The filter processing unit 204 performs, for example, low-pass filter processing having filter coefficients for seven pixels that require reference pixels for three pixels on the left and right of the target pixel on the pixel data on which the pixel copy processing has been performed.
The synchronization processing unit 205 performs pixel interpolation calculation processing to calculate R, G, and B color signal values in each pixel, and performs the synchronization processing.
The YUV conversion processing unit 206 converts the RGB signal data into YUV signal data composed of the luminance signal Y and the color difference signals U and V.
The gamma correction processing unit 207 performs gamma correction processing for converting the signal value of the YUV signal data output from the YUV conversion processing unit 206 based on the signal value conversion table. Thereafter, various image processing according to the purpose is performed.

FIG. 3 is a diagram illustrating an example of the configuration of the pixel copy processing unit 203 of the image processing circuit 104.
The pixel copy processing control unit 330 controls the entire pixel copy processing unit 203.
The pixel copy processing control unit 330 receives a VALID_IN signal (Sig_302) for notifying that valid pixel data has been input from the previous gain correction processing unit 202. Also, the pixel copy processing control unit 330 outputs a VALID_OUT signal (Sig_305) that notifies the subsequent filter processing unit 204 that valid pixel data is output.
Further, the pixel copy processing control unit 330 receives the processing stop signal STOP_IN signal (Sig_304) from the subsequent filter processing unit 204. When the STOP_IN signal (Sig_304) is at a high level, the pixel copy processing control unit 330 stops the processing of the pixel copy processing unit 203.

  When the STOP_OUT signal (Sig_301) is at a high level, the pixel copy processing control unit 330 stops the input of valid pixel data from the previous gain correction processing unit 202. The pixel copy processing control unit 330 outputs a high-level STOP_OUT signal (Sig_301) to the gain correction processing unit 202 when the high-level STOP_IN signal (Sig_304) is input and during the pixel copy processing of the pixel data copy processing unit 332. To do. As a result, the pixel copy processing control unit 330 stops input of valid pixel data from the gain correction processing unit 202 to the pixel copy processing unit 203.

The line buffer memory unit 331 has as many line buffer memories (FIFO format) as the sum of the number of pixel data of one line and the number of pixel periods of the color filter array (six pixels in the case of the color filter array in FIG. 10). Have
The line buffer memory unit 331 receives a DATA_IN signal (Sig_303) for transferring pixel data from the gain correction processing unit 202 in the previous stage. The line buffer memory unit 331 stores the DATA_IN signal (Sig_303) as effective pixel data when the VALID_IN signal (Sig_302) is at a high level. Each effective pixel data is captured by one pixel per clock cycle when the VALID_IN signal (Sig_302) is at a high level.

The LINEBUF_OUTEN signal (Sig_311) is a read signal of the line buffer memory unit 331 from the pixel copy processing control unit 330. When the LINEBUF_OUTEN signal (Sig_311) is at a high level, valid pixel data is output from the line buffer memory unit 331 as a BUF_DATA_OUT signal (Sig_320).
The pixel data copy processing unit 332 performs pixel copy processing for adding effective pixel data at the edge of the image as reference pixel data outside the edge of the image. Here, the reference pixel data subjected to the image copy process is a pixel of interest of a filter having a filter coefficient for seven pixels used in the low-pass filter process in the filter processing unit 204 is located at the end of the image. This is pixel data referred to at the time.

The pixel data copy processing unit 332 includes effective pixel data at the left end and right end, and reference pixel data at the left end and right end of the next line during output processing, among the effective pixel data for two lines of image data. , And a buffer memory unit 340 for storing.
The buffer memory unit 340 includes buffer memories M_1L (351), M_1R (352), M_2L (353), and M_2R (354) for storing effective pixel data. Furthermore, the buffer memory unit 340 includes buffer memories M_REF_L (355) and M_REF_R (356) for storing reference pixel data.
Each of the buffer memories M_1L (351), M_1R (352), M2_L (353), and M2_R (354) stores pixel data corresponding to the number of pixel periods (for six pixels) of the color filter array shown in FIG.

In the buffer memories M_1L (351) and M_2L (353), the effective pixel data of 6 pixels at the left end portion of the image data of the first line and the second line are respectively stored.
The buffer memories M_1R (352) and M_2R (354) store the effective pixel data of 6 pixels at the right end of the image data of the first line and the second line, respectively.
Further, the pixel data copy processing unit 332 has two buffer memories for storing the reference pixel data in order to output the reference pixel data subjected to the pixel copy process with a delay of one line. In the buffer memories M_REF_L (355) and M_REF_R (356) for storing the reference pixel data, the reference pixel data at the left end and the reference pixel data at the right end of the line output with a delay of one line are stored.

  When the COPYBUF_INEN signal (Sig_340) from the pixel copy processing control unit 330 is at the high level, the pixel data copy processing unit 332 stores the effective pixel data at the end of the image in order in the four buffer memories. The storage order is the order of buffer memory M_1L (351) → M_1R (352) → M_2L (353) → M_2R (354) → M_1L (351) →. When writing of pixel data for 6 pixels is completed in one buffer memory, writing to the next buffer memory is performed.

The pixel copy processing control unit 330 outputs a High level COPY_START signal (Sig_341) for one pulse to the pixel data copy processing unit 332 at the start of the pixel copy processing, and instructs the start of the pixel copy processing.
Upon completion of the pixel copy processing at each of the left and right end portions of the image, the pixel data copy processing unit 332 outputs a High level COPY_END signal (Sig_351) to the pixel copy processing control unit 330, and performs pixel copy processing. Notify completion.

The COPY_VALID_OUT signal (Sig_350) output from the pixel data copy processing unit 332 to the pixel copy processing control unit 330 is at a high level when the reference pixel data is read from the buffer memory unit 340. The read reference pixel data is output to the selector unit 333 as a COPY_DATA_OUT signal (Sig_321).
The pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305) when the COPY_VALID_OUT signal (Sig_350) is at a High level. As a result, as described above, the filter processing unit 204 is notified that the effective pixel data is output.

The color ID search unit 341 of the pixel data copy processing unit 332 searches and sets which effective pixel data is output as the reference pixel data from the effective pixel data stored in the buffer memory unit 340. The operation of the color ID search unit 341 will be described in detail later.
The selector unit 333 outputs the pixel data signal DATA_OUT signal (Sig_306) to the subsequent filter processing unit 204. When the SEL signal (Sig_314) output from the pixel copy processing control unit 330 to the selector unit 333 is at a high level, a COPY_DATA_OUT signal (Sig_321) that is reference pixel data is output as the DATA_OUT signal (Sig_306). On the other hand, when the SEL signal (Sig_314) is at a low level, a BUF_DATA_OUT signal (Sig_320) that is valid pixel data stored in the line buffer memory unit 331 is output as the DATA_OUT signal (Sig_306).

The register unit 334 includes a plurality of registers that write setting values for controlling the pixel copy processing unit 203. Each register is connected to the pixel copy processing control unit 330 and the pixel data copy processing unit 332, and a register value is read out in a timely manner in each processing unit. Before the pixel copy process is started, the set value is read from and written to the register of the register unit 334 by a Read / Write signal from the CPU (system control unit) 107.
Hereinafter, an example of a register included in the register unit 334 will be described.
The number of horizontal pixels and the number of vertical pixels of the image data are set in the H_PIX register and the V_PIX register of the register unit 334.
As the number of reference pixel data to be subjected to pixel copy processing, the number of pixels of the reference image data at the left end is set in the HL_REF_PIX register, and the number of pixels of the reference image data at the right end is set in the HR_REF_PIX register.

As the number of effective pixel data to be copied from the pixel copy process, the number of effective pixel data at the left end is set in the HL_VAL_PIX register, and the number of effective pixel data at the right end is set in the HR_VAL_PIX register. .
FIG. 4A is a diagram illustrating an example of setting values of the color ID register referred to by the color ID search unit 341. In the following description, the color ID register is referred to as a CID register as necessary.

The CID register of the register unit 334 has a register configuration capable of holding set values for 6 × 6 pixels in accordance with the period of the color filter array shown in FIG.
A color ID value corresponding to each color of the color filter is set in the CID register. In this embodiment, a color ID value of “0” is set at the R pixel position, “1” is set at the G pixel position, and “2” is set at the B pixel position. The color ID value in the CID register is referred to when the color ID search unit 341 searches for the pixel position of the effective pixel data that is the copy source of the reference pixel data.

  The CID register has a horizontal color ID address value CIDX and a vertical color ID address value CIDY shown in FIG. 4A, and the position of each color ID can be specified by these. In the following description, a combination of the color ID address value CIDX in the horizontal direction and the color ID address value CIDY in the vertical direction is referred to as a CID address value as necessary. The pixel position of the reference pixel data and the pixel position of the effective pixel data are specified by the CID address value.

The start pixel position of the reference pixel data at the left end of the first line of the image data is set in the L_REF_CIDX register and the L_REF_CIDY register of the register unit 334 using the CID address value.
The starting pixel position of the effective pixel data is set in the L_VAL_CIDX register and the L_VAL_CIDY register of the register unit 334.
Similarly, the start pixel position of the reference pixel data at the right end of the first line of the image data is set in the R_REF_CIDX register and the R_REF_CIDY register of the register unit 334 using the CID address value.

The end pixel position of the valid pixel data is set in the R_VAL_CIDX register and the R_VAL_CIDY register of the register unit 334.
The number of horizontal pixels and the number of vertical pixels of the number of combined pixel blocks of the color ID used for searching the reference image data are set in the BLK_X register and BLK_Y register of the register unit 334.

FIG. 5 is a diagram illustrating an example of a result of pixel copy processing at both ends of an image. The white pixel data in FIG. 5 is reference pixel data extrapolated to the left end portion and the right end portion of the image by pixel copy processing. On the other hand, the pixel data of halftone dots indicates effective pixel data.
Pixel blocks 501, 502, and 503 at the left end of the first line and the second line represent reference pixel data subjected to pixel copy processing.
Pixel blocks 508, 509, and 510 at the right end of the first line and the second line represent reference pixel data subjected to pixel copy processing.
The pixel blocks 504 to 507 in the first line and the second line represent effective pixel data.

By the pixel copy processing described below, the reference pixel data of the pixel blocks 501 and 503 at the left end portion of the image is subjected to copy processing by inverting the R and G effective pixel data of the pixel block 505 upside down. For the reference pixel data of the pixel block 502, the effective pixel data of the G color and the B color of the pixel block 504 are inverted and copied.
For the reference pixel data of the pixel blocks 508 and 510 at the right end of the image, the B and G effective pixel data of the pixel block 506 are inverted and copied. For the reference pixel data of the pixel block 509, the effective pixel data of the G color and the R color of the pixel block 507 are inverted and copied.

For the reference pixel data of the pixel blocks 531 and 533 at the left end of the images of the third line and the fourth line, G-color effective pixel data for two pixels of the pixel block 534 is copied. For the reference pixel data of the pixel block 532, the effective pixel data of the R color and the B color of the pixel block 535 are inverted and copied.
For the reference pixel data of the pixel blocks 538 and 540 at the right end of the images of the third line and the fourth line, the G effective pixel data for two pixels of the pixel block 538 is copied. For the reference pixel data of the pixel block 539, the effective pixel data of the B color and the R color of the pixel block 536 are inverted and copied.

The reference pixel data of the pixel blocks 551 and 553 at the left end of the images of the fifth line and the sixth line are subjected to copy processing by inverting the R and G effective pixel data of the pixel block 554 upside down. For the reference pixel data of the pixel block 552, the effective pixel data of the R color and the G color of the pixel block 554 are inverted and copied.
For the reference pixel data of the pixel blocks 558 and 560 at the right end of the images of the fifth and sixth lines, the G and R effective pixel data of the pixel block 556 are inverted and copied. For the reference pixel data of the pixel block 559, the effective pixel data of the G color and the R color of the pixel block 556 are inverted and copied.

FIG. 6 is a timing chart showing an example of an outline of processing contents of each signal in the pixel copy processing unit 203 shown in FIG.
Hereinafter, with reference to the timing chart of FIG. 6, reference pixel data for three pixels in the horizontal direction is copied to the left end and the right end of four lines (first to fourth lines) shown in FIG. An example of the operation when the processing is performed) will be described.

In the period A in FIG. 6, before valid pixel data is input as the DATA_IN signal (Sig_ 303), the CPU 107 (system control unit) writes the following setting values to each register of the register unit 334.
In the present embodiment, it is assumed that the size of the image data is 1280 × 960 pixels. Therefore, “H_PIX register = 1280, V_PIX register = 960” is set.
Since the reference pixel data for three pixels in the horizontal direction are copied to the left and right end portions of the image (perform pixel copy processing), “HL_REF_PIX register = 3, HR_REF_PIX register = 3” is set.

In the CID register, the value of the color ID shown in FIG. 4A is set in accordance with the combination of the color filter arrays shown in FIG.
The starting point pixel position of the reference pixel data shown in FIG. 5 is the position of the G pixel data at the left end of the first line (the position of the G pixel in the pixel block 501). In the CID register, the CID address value of the color ID corresponding to this position is the pixel position 401 (CIDX = 0, CIDY = 0) in FIG. 4B. Therefore, “L_REF_CIDX register = 0, L_REF_CIDY register = 0” is set.

The start pixel position of the effective pixel data shown in FIG. 5 is the position of the fourth G pixel data on the first line (the position of the G pixel in the pixel block 504). In the CID register, the CID address value of the color ID corresponding to this position is the pixel position 402 (CIDX = 3, CIDY = 0) in FIG. 4B. Therefore, “L_VAL_CIDX register = 3, L_VAL_CIDY register = 0” is set.
The starting point pixel position of the reference pixel data shown in FIG. 5 is the position of the G pixel data at the right end of the first line (the position of the G pixel in the pixel block 508). In the CID register, the CID address value of the color ID corresponding to this position is the pixel position 408 (CIDX = 3, CIDY = 0) in FIG. Therefore, “R_REF_CIDX register = 3, R_REF_CIDY register = 0” is set.

The end pixel position of the effective pixel data shown in FIG. 5 is the position of the fourth G pixel data from the right end of the first line (the position of the G pixel in the pixel block 507). In the CID register, the CID address value of the color ID corresponding to this position is the pixel position 412 (CIDX = 2, CIDY = 0) in FIG. Therefore, “R_VAL_CIDX register = 2, R_VAL_CIDY register = 0” is set.
In the color filter array of the image sensor 102 of this embodiment shown in FIG. 10, the upper two lines are composed of a combination of G color and R color and a combination of B color and G color vertically. The center two lines are composed of a combination of G color and G color and a combination of B color and R color vertically. The two lines at the lower end are composed of a combination of B and G colors and a combination of G and R colors in the vertical direction.

Therefore, if the reference pixel data to be subjected to pixel copy processing is set to a pixel block (size is 1 horizontal pixel and 2 vertical pixels) that is a combination of upper and lower colors across two lines, Effective pixel data of the copy source having the same color combination always exists. As described above, the number of combined pixel blocks of the color ID used for searching the reference pixel data in the color ID search unit 341 is the number of horizontal pixels “1” and the number of vertical pixels “2”. Therefore, “BLK_X register = 1, BLK_Y register = 2” is set.
The color filter array shown in FIG. 10 has a period of 6 pixels. Effective pixel data as a copy source of the pixel copy process always exists within one cycle. Therefore, “HL_VAL_PIX register = 6, HR_VAL_PIX register = 6” is set. As described above, the values of the HL_VAL_PIX register and the HR_VAL_PIX register are the numbers of pixels at the left end and the right end of the effective pixel data that is the copy source of the pixel copy process.

In FIG. 6, pixel data 1L of the DATA_IN signal (Sig_303) indicates 6 pixels of effective pixel data at the left end of the pixel data on the first line. The pixel data for the six pixels becomes effective pixel data to be a copy source in the pixel copy process at the left end portion of the image. Hereinafter, in the notation of FIG. 6, pixel data 2L, 3L,... Indicates 6 pixels of effective pixel data that is a copy source of the left end of the second line, the third line,.
Similarly, in FIG. 6, pixel data 1R of the DATA_IN signal (Sig_303) indicates 6 pixels of effective pixel data at the right end of the pixel data on the first line. The pixel data for the six pixels becomes effective pixel data that is a copy source in the pixel copy process at the right end of the image. Hereinafter, in the notation of FIG. 6, pixel data 2R, 3R,... Indicate effective pixel data of 6 pixels that are the copy source at the right end of the second line, the third line,.

  In the notation of FIG. 6, the pixel data 1M of the DATA_IN signal (Sig_303) is pixel data other than the pixel data 1L and 1R among the pixel data on the first line, and is not used in the pixel copy process. Hereinafter, in the notation of FIG. 6, pixel data 2M, 3M... Is pixel data other than pixel data 2L and 2R, 3L and 3R. Indicates.

On the other hand, in FIG. 6, pixel data 1L ′ of the DATA_OUT signal (Sig_306) indicates three pixels of reference pixel data subjected to pixel copy processing at the left end of the first line. Hereinafter, in the notation of FIG. 6, pixel data 2L ′, 3L ′,... Indicate reference pixel data subjected to pixel copy processing at the left end of the second line, the third line,.
Similarly, in FIG. 6, pixel data 1R ′ of the DATA_OUT signal (Sig_306) indicates three pixels of reference pixel data subjected to pixel copy processing at the right end of the first line. Hereinafter, in the notation of FIG. 6, pixel data 2R ′, 3R ′,... Indicate reference pixel data subjected to pixel copy processing at the right end of the second line, the third line,.

During the period A shown in FIG. 6, the effective pixel data (pixel data 1L, 1M, 1R) for the first line and the six pixels at the left end of the second line are sent as the DATA_IN signal (Sig_303) from the gain correction processing unit 202 in the previous stage. Effective pixel data (pixel data 2L).
The pixel copy processing control unit 330 refers to the register values of the register unit 334 (“H_PIX register = 1280”, “HL_VAL_PIX register = 6, HR_VAL_PIX register = 6”), and performs the following processing. As described above, the H_PIX register is the number of horizontal pixels of the image data. The HL_VAL_PIX register is the number of pixels at the left end of the effective pixel data that is the copy source, and the HR_VAL_PIX register is the number of pixels at the right end of the effective pixel data that is the copy source.

  The pixel copy processing control unit 330 creates control timing by a counter circuit (not shown) based on these register values, and generates a COPYBUF_INEN signal (Sig_340). The COPYBUF_INEN signal (Sig_340) becomes High level when the pixel data 1L, 1R, 2L, 2R, 3L, and 3R are input as the DATA_IN signal (Sig_303). During the period when the COPYBUF_INEN signal (Sig_340) is at the high level, the pixel data copy processing unit 332 stores the pixel data 1L, 1R, 2L, 2R,... In the buffer memory unit 340 in order.

7A to 7D schematically illustrate an example of the contents of pixel data stored in the buffer memories M_1L, M_1R, M_2L, and M_2R and the buffer memories M_REF_L and M_REF_R of the buffer memory unit 340 in each period. FIG. Specifically, FIGS. 7A, 7B, 7C, and 7D show states when the periods A, C, F, and I shown in FIG. 6 are completed.
The address values ML_CIDX and MR_CIDX shown in FIG. 7 indicate the address values of the buffer memories M_1L (351) and M_2L (353), M_1R (352) and M_2R (354), respectively. These address values correspond to the CID address value CIDX of the CID register.

  The position of “ML_CIDX = 0” is the CID address value specified by “L_VAL_CIDX register”, and corresponds to the position of “CIDX = 3” of the CID register from “L_VAL_CIDX register = 3”. As described above, the L_VAL_CIDX register indicates the start pixel position of the effective pixel data at the left end of the first line. Thereafter, “ML_CIDX = 1, 2, 3, 4, 5” sequentially corresponds to “CIDX = 4, 5, 0, 1, 2” of the CID register.

  On the other hand, the position of “MR_CIDX = 5” is the CID address value specified by “R_VAL_CIDX register”, and corresponds to the position of “CIDX = 2” of the CID register from “R_VAL_CIDX register = 2”. As described above, the R_VAL_CIDX register indicates the start pixel position of the effective pixel data at the right end of the first line. Thereafter, “MR_CIDX = 4, 3, 2, 1, 0” sequentially corresponds to “CIDX = 1, 0, 5, 4, 3” of the CID register.

The color ID search unit 341 uses the CID register to determine the position of the effective pixel data that is the copy source of the reference pixel data, and reads the effective pixel data stored at the positions indicated by ML_CIDX and MR_CIDX based on the result. .
In the period A, as shown in FIG. 7A, the effective pixel data 1L, 1R, and 2L are stored in the buffer memories M_1L (351), M_1R (352), and M2_L (353), respectively. The line buffer memory unit 331 stores effective pixel data (1L, 1M, 1R) and effective pixel data 2L for the first line.

Next, in the period B, the pixel copy processing control unit 330 outputs a high-level STOP_OUT signal (Sig_301) to the previous gain correction processing unit 202, and inputs valid pixel data to the DATA_IN signal (Sig_303). Stop.
Next, the pixel copy processing control unit 330 outputs a High level COPY_START signal (Sig_341) to the pixel data copy processing unit 332. Thereby, the pixel data copy processing unit 332 starts the pixel copy process. In period B, pixel copy processing of reference pixel data at the left end of the first line is performed.

Hereinafter, an example of the content of the pixel copy process will be described with reference to FIGS. 4 and 7.
As the first processing in the period B, pixel copy processing of the reference pixel data (G color of the pixel block 501 in FIG. 5) of the first pixel at the left end of the first line is performed.
Here, “L_REF_CIDX register = 0, L_REF_CIDY register = 0” is set. Therefore, the combination of color IDs of the pixel block 501 to be subjected to pixel copy processing corresponds to the combination (1, 0) of the pixel block 411 in FIG. 4B. As described above, the L_REF_CIDX register and the L_REF_CIDY register indicate the start pixel position of the reference pixel data at the left end of the first line.

  In this example, “L_VAL_CIDX register = 3, L_VAL_CIDY register = 0” is set. Therefore, the combination of the color IDs of the pixel block 504 of the effective pixel data that is the copy source of the pixel copy process corresponds to the combination (1, 2) of the pixel block 414 in FIG. As described above, the L_VAL_CIDX register and the L_VAL_CIDY register indicate the start pixel position of the effective pixel data at the left end of the first line.

The CID address value of the pixel block 414 is “CIDX = 3”, and this CID address value corresponds to the position of the address value “ML_CIDX = 0” of the buffer memory in FIG. Therefore, effective pixel data corresponding to the pixel block 414 is effective pixel data of the pixel block 701 stored in the buffer memories M_1L and M_2L.
The color ID search unit 341 starts reading the color ID value from the pixel block 414 of the CID register shown in FIG. 4B, and reads the color ID value of each pixel block for a maximum of six addresses in the right direction in the figure. . In addition, when it moves to the position of “CIDX = 5”, it moves to the position of “CIDX = 0” next.

A comparison circuit (not shown) compares the read color ID value with the color ID combination (1, 0) of the pixel block 411 of the reference pixel data, and has the same color ID combination. Explore.
Since the combination of the color IDs of the pixel block 414 is (1, 2), the process proceeds to the pixel block 415 on the right side.
Since the combination of the color IDs of the pixel block 415 is (0, 1), the comparison circuit determines that the combination is the same as the combination (1, 0) of the color ID of the pixel block 411 of the reference pixel data.

The CID address value of the pixel block 415 is “CIDX = 4”, and the position of the buffer memory address value “ML_CIDX = 1” in FIG. Therefore, the effective pixel data corresponding to the pixel block 415 becomes effective pixel data of the pixel block 702 stored in the line buffer memories M_1L and M_2L.
Next, the G color of the pixel block 702 of the line buffer memory M_2L is output to the COPY_DATA_OUT signal (Sig_321) as reference pixel data (G color of the pixel block 501 in FIG. 5) of the leftmost first pixel on the first line. . Note that during the period B, the selector unit 333 connects the COPY_DATA_OUT signal (Sig_321) to the DATA_OUT signal (Sig_306).

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.
On the other hand, the R color of the pixel block 702 of the line buffer memories M_1L and M_2L is stored in the buffer memory M_REF_L (355) because it becomes the reference pixel data of the first pixel at the left end of the second line.

As the second processing in the period B, pixel copy processing of reference pixel data (B color of the pixel block 502 in FIG. 5) of the second pixel at the left end of the first line is performed.
In the first process in the period B, it is determined that the combination of the color IDs of the pixel block 501 subjected to the pixel copy process corresponds to the combination of the pixel blocks 411 by setting “L_REF_CIDX register = 0, L_REF_CIDY register = 0”. . Therefore, the combination of the color IDs of the pixel block 502 of the reference pixel data to be subjected to pixel copy processing moves to the combination (2, 1) of the pixel block 412 on the right side of the pixel block 411 in FIG.

In this example, “L_VAL_CIDX register = 3, L_VAL_CIDY register = 0” is set. Therefore, as in the first process in the period B, the combination of the color IDs of the pixel blocks 504 of the effective pixel data that is the copy source of the pixel copy process is the combination (1, 2) of the pixel block 414 in FIG. ). The color ID search unit 341 starts reading the color ID value from the pixel block 414 of the CID register shown in FIG. 4B, and reads the color ID value of each pixel block for a maximum of six addresses in the right direction in the figure. .
A comparison circuit (not shown) compares the read color ID value with the color ID combination (2, 1) of the pixel block 412 of the reference pixel data, and has the same color ID combination. Explore.
Since the combination of the color IDs of the pixel block 414 is (1, 2), the comparison circuit determines that the combination is the same as the combination (2, 1) of the color ID of the pixel block 412 of the reference pixel data.

As described above, the pixel block 414 corresponds to the pixel block 701 in the pixel data stored in the line buffers M_1L and M_2L illustrated in FIG.
Next, the B color of the pixel block 701 in the line buffer memory M_2L is output to the COPY_DATA_OUT signal (Sig_321) as the reference pixel data of the second pixel at the left end of the first line (B color of the pixel block 502 in FIG. 5). The

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the output of the High level COPY_DATA_OUT signal (Sig_350) is completed, the Low level COPY_DATA_OUT signal (Sig_350)
Is output.
On the other hand, the B color of the pixel block 701 of the line buffer memories M_1L and M_2L becomes the reference pixel data of the second pixel at the left end of the second line and is stored in the buffer memory M_REF_L (355).

As the third process in the period B, the pixel copy process of the reference pixel data (G color of the pixel block 503 in FIG. 5) of the third pixel at the left end of the first line is performed.
As described above, in the second process in the period B, the combination of the color IDs of the pixel block 502 of the reference pixel data subjected to the pixel copy process corresponds to the combination (2, 1) of the pixel block 412. Therefore, the combination of the color IDs of the pixel block 503 of the reference pixel data to be subjected to pixel copy processing moves to the combination (1, 0) of the pixel block 413 on the right side of the pixel block 412 in FIG.

In this example, “L_VAL_CIDX register = 3, L_VAL_CIDY register = 0” is set. Therefore, as in the first process in the period B, the combination of the color IDs of the pixel blocks 504 of the effective pixel data that is the copy source of the pixel copy process is the combination (1, 2) of the pixel block 414 in FIG. ). The color ID search unit 341 starts reading the color ID value from the pixel block 414 of the CID register shown in FIG. 4B, and reads the color ID value of each pixel block for a maximum of six addresses in the right direction in the figure. .
A comparison circuit (not shown) compares the read color ID value with the color ID combination (1,0) of the pixel block 413 of the reference pixel data, and has the same color ID combination. Explore. Since the combination of the color IDs of the pixel block 415 is (0, 1) as in the first process in the period B, the comparison circuit uses the combination of the color IDs of the pixel block 413 of the reference pixel data (1, 1, 0).

The CID address value of the pixel block 415 is “CIDX = 4”, and the position of the buffer memory address value “ML_CIDX = 1” in FIG. Therefore, the effective pixel data corresponding to the pixel block 415 corresponds to the effective pixel data of the pixel block 702 stored in the line buffer memories M_1L and M_2L.
Next, the G color of the pixel block 702 of the line buffer memory M_2L is output to the COPY_DATA_OUT signal (Sig_321) as reference pixel data (G color of the pixel block 503 in FIG. 5) of the third pixel at the left end of the first line. . Note that during the period B, the selector unit 333 connects the COPY_DATA_OUT signal (Sig_321) to the DATA_OUT signal (Sig_306).

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. Accordingly, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_350). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.
On the other hand, the R color of the pixel block 702 of the line buffer memories M_1L and M_2L is stored in the buffer memory M_REF_L (355) because it becomes the reference pixel data of the third pixel on the left end of the second line.

  This completes the pixel copy processing of the reference pixel data for the leftmost three pixels set in “HL_REF_PIX register = 3”. Therefore, the pixel data copy processing unit 332 outputs a High level COPY_END signal (Sig_351) for one pulse to the pixel copy processing control unit 330.

Next, in the period C, the pixel copy processing control unit 330 outputs a High level LINEBUF_OUTEN signal (Sig_311) to the line buffer memory unit 331. As a result, the pixel data 1L, 1M, and 1R are read from the line buffer memory unit 331 and output as a BUF_DATA_OUT signal (Sig_320). Note that during the period C, the selector unit 333 connects the BUF_DATA_OUT signal (Sig_320) with the DATA_OUT (Sig_306) signal.
Similarly, during the period C, the High level VALID_IN signal (Sig_302) is input to the line buffer memory unit 331, and the pixel data 2M of the second line, 2R, and the pixel data 3L of the third line are stored in the line buffer memory unit 331. The

  FIG. 7B illustrates an example of pixel data stored in the four line buffers M_1L (351), M_1R (352), M_2L (353), and M_2R (354) of the buffer memory unit 340 when the period C is completed. FIG. The line buffer memories M_1L (351), M_1R (352), M_2L (353), and M_2R (354) store pixel data 3L, 1R, 2L, and 2R, respectively. The buffer memory M_REF_L (355) stores the reference pixel data 2L ′ at the left end of the second line.

As the first processing in the period D, pixel copy processing of reference pixel data (G color of the pixel block 508 in FIG. 5) of the first pixel on the right end of the first line is performed.
With the setting of “R_REF_CIDX register = 3, R_REF_CIDY register = 0”, the combination of color IDs of the pixel block 508 subjected to pixel copy processing corresponds to the combination (1, 2) of the pixel block 418 in FIG. In this example, “R_VAL_CIDX register = 2, R_VAL_CIDY register = 0” is set. Therefore, the combination of the color IDs of the pixel block 507 of the effective pixel data that is the copy source of the pixel copy process corresponds to the combination (1, 0) of the pixel block 417 in FIG.

The CID address value of the pixel block 417 is “CIDX = 2”, and this CID address value corresponds to the position of the buffer memory address value “MR_CIDX = 5” in FIG. 7B. Therefore, effective pixel data corresponding to the pixel block 417 is effective pixel data of the pixel block 704 stored in the buffer memories M_1R and M_2R.
The color ID search unit 341 starts reading the color ID value from the pixel block 417 of the CID register in FIG. 4C, and reads the color ID value of each pixel block for up to six addresses in the left direction in the figure. In addition, when it moves to the position of “CIDX = 0”, it moves to the position of “CIDX = 5” next.

In a comparison circuit (not shown), the read color ID value is compared with the color ID combination (1, 2) of the pixel block 418 of the reference pixel data, and the pixel blocks having the same color ID combination Explore.
Since the combination of the color IDs of the pixel block 417 is (1, 0), the process proceeds to the pixel block 416 adjacent to the left.
Since the combination of color IDs of the pixel block 416 is (2, 1), the comparison circuit determines that the combination is the same as the combination (1, 2) of color IDs of the pixel block 418 of the reference pixel data.

The CID address of the pixel block 416 is “CIDX = 1”, and the position of the address value “ML_CIDX = 4” in the buffer memory in FIG. Therefore, the effective pixel data corresponding to the pixel block 416 corresponds to the effective pixel data of the pixel block 703 stored in the buffer memories M_1R and M_2R.
Next, the G color of the pixel block 703 of the line buffer memory M_2R is output to the COPY_DATA_OUT signal (Sig_321) as reference pixel data (G color of the pixel block 508 in FIG. 5) of the first pixel on the right end of the first line. . Note that during the period C, the COPY_DATA_OUT signal (Sig_321) is connected to the DATA_OUT signal (Sig_306) by the selector unit 333.

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.
On the other hand, the B color of the pixel block 703 of the line buffer memories M_1R and M_2R is stored in the buffer memory M_REF_R (356) because it becomes the reference pixel data of the first pixel at the left end of the second line.

As the second processing in the period D, pixel copy processing of reference pixel data (R color of the pixel block 509 in FIG. 5) of the second pixel on the right end of the first line is performed.
In the first processing in the period D, it is determined that the combination of the color IDs of the pixel block 508 subjected to pixel copy processing corresponds to the combination of the pixel blocks 418 by setting “R_REF_CIDX register = 3, R_REF_CIDY register = 0”. . Therefore, the combination of the color IDs of the pixel block 509 of the reference pixel data to be subjected to pixel copy processing moves to the combination (0, 1) of the pixel block 419 on the right side of the pixel block 418 in FIG.

Here, “R_VAL_CIDX register = 2, R_VAL_CIDY register = 0” is set. Therefore, as in the first process in the period D, the combination of the color IDs of the pixel blocks 507 of the effective pixel data that is the copy source of the pixel copy process is the combination (1,0) of the pixel block 417 in FIG. ). The color ID search unit 341 starts reading the color ID value from the pixel block 417 of the CID register shown in FIG. 4C, and reads the color ID value of each pixel block for up to six addresses in the left direction in the figure.
In a comparison circuit (not shown), the read color ID value is compared with the color ID combination (0, 1) of the pixel block 417 of the reference pixel data, and the pixel blocks having the same color ID combination Explore.
Since the combination of the color IDs of the pixel block 417 is (1, 0), the comparison circuit determines that the combination is the same as the combination (0, 1) of the color ID of the pixel block 417 of the reference pixel data.

The pixel block 417 corresponds to the pixel block 704 in the pixel data stored in the line buffers M_1R and M_2R in FIG.
Next, the R color of the pixel block 704 of the line buffer memory M_2R is output to the COPY_DATA_OUT signal (Sig_321) as reference pixel data (R color of the pixel block 509 in FIG. 5) of the second rightmost pixel on the first line. .

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the output of the High level COPY_DATA_OUT signal (Sig_350) is completed, the Low level COPY_DATA_OUT signal (Sig_350)
Is output.
On the other hand, the G color of the pixel block 704 in the line buffer memories M_1R and M2_R is stored in the buffer memory M_REF_R (356) because it becomes the reference pixel data of the second pixel at the right end of the second line.

As the third process in the period D, the pixel copy process of the reference pixel data (G color of the pixel block 510 in FIG. 5) of the third pixel at the right end of the first line is performed.
As described above, in the second process in the period D, the combination of the color IDs of the pixel block 509 of the reference pixel data subjected to the pixel copy process corresponds to the combination (0, 1) of the pixel block 419. Therefore, the combination of the color IDs of the pixel block 510 of the reference pixel data to be subjected to pixel copy processing moves to the combination (1, 2) of the pixel block 420 on the right side of the pixel block 419 in FIG.

Here, “R_VAL_CIDX register = 2, R_VAL_CIDY register = 0” is set. Therefore, as in the first process in the period D, the combination of the color IDs of the pixel blocks 507 of the effective pixel data that is the copy source of the pixel copy process is the combination (1,0) of the pixel block 417 in FIG. ). The color ID search unit 341 starts reading the color ID value from the pixel block 417 of the CID register shown in FIG. 4C, and reads the color ID value of each pixel block for a maximum of six addresses in the right direction in the figure. .
In a comparison circuit (not shown), the read color ID value is compared with the color ID combination (1, 2) of the pixel block 420 of the reference pixel data, and the pixel blocks having the same color ID combination Explore. Similarly to the first process in the period D, the combination of the color IDs of the pixel block 417 is (1, 0), so that the process proceeds to the pixel block 416 adjacent to the left side. Since the combination of color IDs of the pixel block 416 is (2, 1), the comparison circuit determines that the combination is the same as the combination (1, 2) of color IDs of the pixel block 418 of the reference pixel data.

The CID address value of the pixel block 416 is “CIDX = 1”, and the position of the buffer memory address value “ML_CIDX = 4” in FIG. Therefore, the effective pixel data corresponding to the pixel block 416 corresponds to the effective pixel data of the pixel block 703 stored in the line buffer memories M_1L and M_2L.
Next, the G color of the pixel block 703 of the line buffer memory M_2R is output to the COPY_DATA_OUT signal (Sig_321) as reference pixel data (G color of the pixel block 510 in FIG. 5) of the rightmost third pixel on the first line. . Note that during the period D, the COPY_DATA_OUT signal (Sig_321) is connected to the DATA_OUT signal (Sig_306) by the selector unit 333.

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.
On the other hand, the B color of the pixel block 703 of the line buffer memories M_1R and M_2R becomes the reference pixel data of the third pixel at the right end of the second line and is stored in the buffer memory M_REF_R (356).

  Next, in the period E, the pixel data copy processing unit 332 outputs the reference pixel data 2L ′ at the left end of the second line stored in the buffer memory M_REF_L (355) to the COPY_DATA_OUT signal (Sig_321). Note that during the period E, the selector unit 333 connects the COPY_DATA_OUT signal (Sig_321) to the DATA_OUT signal (Sig_306).

At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.
This completes the pixel copy processing of the reference pixel data for the leftmost three pixels and the rightmost three pixels set by “HR_REF_PIX register = 3” and “HL_REF_PIX register = 3”. Therefore, the pixel data copy processing unit 332 outputs a High level COPY_END signal (Sig_351) for one pulse to the pixel copy processing control unit 330.

In the subsequent period F, the pixel copy processing control unit 330 outputs a high-level LINEBUF_OUTEN signal (Sig_311) to the line buffer memory unit 331. As a result, the pixel data 2L, 2M, and 2R are read from the line buffer memory unit 331 and output as a BUF_DATA_OUT signal (Sig_320). During the period F, the selector unit 333 connects the BUF_DATA_OUT signal (Sig_320) to the DATA_OUT signal (Sig_306).
Similarly, during the period F, a High-level VALID_IN signal (Sig_302) is input to the line buffer memory unit 331, and the pixel data 3M and 3R for the third line and the pixel data 4L for the fourth line are stored in the line buffer memory unit 331. The

  FIG. 7C illustrates an example of pixel data stored in the four line buffers M_1L (351), M_1R (352), M_2L (353), and M_2R (354) of the buffer memory unit 340 when the period F is completed. FIG. The line buffer memories M_1L (351), M_1R (352), M_2L (353), and M_2R (354) store pixel data 3L, 3R, 4L, and 2R, respectively. The buffer pixel M_REF_R (356) stores the reference pixel data 2R ′ at the right end of the second line.

In the subsequent period G, the pixel data copy processing unit 332 outputs the reference pixel data 2R ′ at the right end of the second line stored in the buffer memory M_REF_R (356) to the COPY_DATA_OUT signal (Sig_321). Note that during the period G, the COPY_DATA_OUT signal is connected to the DATA_OUT signal by the selector unit 333.
At the same time, the pixel data copy processing unit 332 outputs a High level COPY_VALID_OUT signal (Sig_350) to the pixel copy processing control unit 330. As a result, the pixel copy processing control unit 330 outputs a High level VALID_OUT signal (Sig_305). After the completion of the output of the High level COPY_DATA_OUT signal (Sig_350), the Low level COPY_DATA_OUT signal (Sig_350) is output.

  With the above processing, the output of the pixel data of the first line and the second line with the reference pixel data added to both ends is completed.

Next, in the period H, pixel copy processing of reference pixel data (G color of the pixel block 531, B color of 532, G color of 533) for the leftmost three pixels on the third line is performed.
In the period H, a pixel copy process similar to the process in the period B is performed. However, the reference pixel data subjected to the pixel copy process shifts from the pixel block 501 of the reference pixel data to the pixel block 531. Therefore, the combination of color IDs of the pixel block 531 subjected to pixel copy processing corresponds to the combination (1, 1) of the pixel block 431, not the pixel block 411 of FIG. 4B.
Similarly, the pixel block of the effective pixel data that is the copy source of the pixel copy process moves from the pixel block 504 to the pixel block 534. Accordingly, the combination of the color IDs of the pixel block 534 of the effective pixel data that is the copy source of the pixel copy process corresponds to the combination (1, 1) of the pixel block 434 instead of the pixel block 414 in FIG.

The CID address of the pixel block 434 is “CIDX = 3”, and the CID address value corresponds to the position of the buffer memory address value “ML_CIDX = 0” in FIG. 7C. Therefore, effective pixel data corresponding to the pixel block 434 becomes effective pixel data of the pixel block 705 stored in the buffer memories M_1L and M_2L.
As described above, in the period H, when the pixel copy processing of the reference pixel data for the leftmost three pixels on the third line is completed, the process proceeds to the period I.

In the period I, the pixel data 1L, 1M, and 1R in the process of the period C are replaced with the pixel data 3L, 3M, and 3R, and the pixel data 2M, 2R, and 2L are replaced with the pixel data 4M, 4R, and 5L, respectively. .
Next, in the period J, a pixel copy process similar to the process in the period D is performed. However, the reference pixel data to be subjected to pixel copy processing is transferred from the pixel block 508 of the reference pixel data to the pixel block 538. Therefore, the combination of color IDs of the pixel block 538 to be subjected to pixel copy processing corresponds to the combination (1, 1) of the pixel block 418 to the pixel block 438 in FIG.

Similarly, the pixel block of the effective pixel data that is the copy source of the pixel copy process moves from the pixel block 507 to the pixel block 537. Therefore, the combination of the color IDs of the pixel block 537 of the effective pixel data that is the copy source of the pixel copy process corresponds to the combination (1, 1) of the pixel block 437 instead of the pixel block 417 in FIG.
The CID address value of the pixel block 437 is “CIDX = 2”, and the position of the buffer memory address value “MR_CIDX = 5” in FIG. Therefore, effective pixel data corresponding to the pixel block 437 becomes effective pixel data of the pixel block 706 stored in the buffer memories M_1L and M_2L.
As described above, when the pixel copy process of the reference pixel data for the rightmost three pixels on the third line is completed in the period J, the process proceeds to the period K.

In the period K, the reference pixel data 4L ′ in the buffer memory M_REF_L (355) is output.
In the next period L, the pixel data 2L, 2M, and 2R in the process of the period F are replaced with the pixel data 4L, 4M, and 4R, and the pixel data 3M, 3R, and 4L are replaced with the pixel data 5M, 5R, and 6L, respectively. Is done.
In the last period M, the reference pixel data 4R ′ in the buffer memory M_REF_R (356) is output.
Thereafter, the pixel copy process is executed up to the final line of the image data, and the process of the pixel data copy processing unit 332 is completed.

FIG. 8 is a diagram illustrating an example of a result of pixel copy processing when the left end portion of the image is located at a different position with respect to the filter colors of the color filter array illustrated in FIG. 10. The white pixel data in FIG. 8 is reference pixel data extrapolated to the left end portion of the image by pixel copy processing. On the other hand, the pixel data of halftone dots indicates effective pixel data.
FIG. 8A shows a case where the left end portion of the image (the left end position of the pixel block 807) is a color filter array of G color, B color, G color, G color, R color, and G color from the top. It is.
The pixel blocks 801 and 803 serving as reference pixel data for the first line and the second line are subjected to pixel copy processing, and G pixel data of the pixel block 807 of effective pixel data having the same combination of color filter colors is obtained. And B color pixel data are copied.

R pixel data and G pixel data of the pixel block 808 of effective pixel data 808 having the same combination of color filter colors are copied to the pixel block 802 serving as reference pixel data by pixel copy processing.
In the pixel blocks 804 and 806 serving as reference pixel data, the R color pixel data and the G color pixel data of the pixel block 808 of the effective pixel data 808 having the same color filter color combination are inverted by pixel copy processing. To be copied.

The pixel block 805 serving as reference pixel data is subjected to a copy process of the G pixel data and the B pixel data of the pixel block 807 of the effective pixel data 807 having the same combination of color filter colors by the pixel copy process.
Similarly, pixel copy processing is also performed on the third line, the fourth line, the fifth line, and the sixth line.

  In the pixel block surrounded by the solid line as the reference pixel data, the pixel data of the pixel block surrounded by the solid line having the same color filter color combination, which is the pixel data of the pixel block at the left end of the effective pixel data, is copied. Is done. When the color filter color is inverted vertically between the pixel block of the reference pixel data and the pixel block of the effective pixel data of the copy source, the pixel data of the effective pixel data is inverted up and down, and the pixel block of the reference pixel data To be copied.

  Similarly, a pixel block surrounded by a broken line serving as reference pixel data includes pixel data of a pixel block surrounded by a broken line that has the same color filter color combination and is the pixel data of the pixel block at the left end of the effective pixel data. Data is copied. Similarly, when the color filter color is inverted vertically between the pixel block of the reference pixel data and the pixel block of the effective pixel data of the copy source, the pixel data of the effective pixel data is inverted upside down and the reference pixel data Copied to pixel block.

Note that, when the B-color effective pixel data is subjected to pixel copy processing within the same line in the first line as in the pixel copy processing described above, the B-color pixel data 810 is copied as reference pixel data. Accordingly, the interval between the B-color pixel data is longer than that when the pixel copy processing described in the present embodiment is performed. Therefore, there is a high possibility that the difference between the reference pixel data to be copied as reference pixel data and the pixel value of the effective pixel data of the copy source becomes large.
Similarly, when the effective pixel data of the R color is subjected to the pixel copy process within the same line in the second line as in the pixel copy process described above, the R pixel data 809 is copied as the reference pixel data. . Therefore, the interval between the R-color pixel data is longer than that when the pixel copy processing described in the present embodiment is performed. Therefore, there is a high possibility that the difference between the reference pixel data to be copied as reference pixel data and the pixel value of the effective pixel data of the copy source becomes large.
Similarly, in the case of the pixel data on the second to sixth lines, the distance between the reference pixel data to be copied and the effective pixel data of the copy source when the pixel copy processing described in the present embodiment is performed is set. It is possible to shorten the length (see the double-line square pixels in FIG. 8A).

FIG. 8B is a diagram illustrating a case where the left end portion of the image is an R, G, R, B, G, B color filter array from the top.
FIG. 8C is a diagram illustrating a case where the left end portion of the image is a color filter array of G color, B color, G color, G color, R color, and G color from the top.
In either case, the illustration is the same as in FIG. In the case of performing the pixel copy processing described in the present embodiment, the interval between the reference pixel data to be copied and the effective pixel data of the copy source, compared to the method of performing pixel copy processing on pixel data of the same color in the same line It is possible to shorten the time.

  As described above, in the present embodiment, an area outside the end (left end / right end) of the effective pixel area, which is an area of effective pixel data, is the size necessary for pixel interpolation processing such as low-pass filter processing from the end. A reference pixel area, which is an area of, is set. A color is set for each pixel in the reference pixel region so that a repetitive pattern of the color filter array is held (so as not to be broken). The reference pixel area is divided into pixel blocks composed of a plurality of pixels, and the pixel block in the effective pixel area, the pixel block having the same size as the pixel block in the reference pixel area, the pixel closest to the pixel block in the reference pixel area Select a block. Then, it is determined whether or not the color combinations of these pixel blocks are the same. When the color combination of the pixel block in the effective pixel region and the pixel block in the reference pixel region is not the same, the determination is performed by shifting the pixel blocks in the effective pixel data one by one in the horizontal direction until they are the same. .

  Specifically, when pixel copy processing is performed on pixel data output from the image sensor 102 having a color filter having a color array of M × N (M and N are positive integers), the following is performed. To. That is, a pixel block having the same color combination as the pixel block in the reference pixel area is searched from the effective pixel area in units of P × Q pixel blocks. Here, P is an integer of M or less, Q is an integer of N or less, at least one of P and Q is an integer of 2 or more, and P × Q is smaller than M × N. In addition, the P × Q pixel block has a minimum size in a range in which the color combination of the pixel block in the reference pixel region exists in the effective pixel region in the search direction from the reference pixel region to the effective pixel region. It is preferable to do this.

  As described above, the interval between the reference pixel subjected to the copy process and the effective pixel as the copy source can be shortened. As a result, the difference in pixel value between the reference pixel subjected to the copy process and the effective pixel as the copy source can be reduced as compared with the conventional process. Further, in the subsequent image processing unit (filter processing unit 204) that processes the pixel data output from the pixel copy processing unit 203, it is possible to reduce the influence of the difference in the pixel value of the pixel data serving as the reference pixel.

  In the present embodiment, the procedure of pixel copy processing at the left end portion and the right end portion of the image is described as an example. However, it is possible to apply the pixel copy processing described in the present embodiment to the pixel copy processing at the upper end portion and the lower end portion of the image.

FIG. 9 is a diagram illustrating an example of a result of pixel copy processing of the upper end portion (for four pixels) of an image. The white pixel data in FIG. 9 is reference pixel data extrapolated to the upper end of the image by pixel copy processing. On the other hand, the pixel data of halftone dots indicates effective pixel data.
In the pixel copy process at the upper end of the image, as with the pixel copy process at the left and right ends of the image, the regularity of the color filter array shown in FIG. Perform a search. Therefore, as the pixel size of the pixel block for search, the number of pixels in the horizontal direction is set to “2”, and the number of pixels in the vertical direction is set to “1”.

The pixel block 901 in the first column and the second column, which are reference pixel data, has G pixel data of the pixel block 905, which is effective pixel data having the same color filter color combination, by pixel copy processing. R color pixel data is copied.
In the pixel blocks 902 and 904 serving as reference pixel data, a pixel block 906 that is effective pixel data having the same combination of color filter colors is copied by being horizontally reversed by pixel copy processing.
A pixel block 905, which is effective pixel data having the same combination of color filter colors, is copied to the pixel block 903 serving as reference pixel data while being reversed left and right by pixel copy processing.

  In the third column, the fourth column, the fifth column, and the sixth column, the pixel block surrounded by the solid line as the reference pixel data is the pixel block data at the upper end of the effective pixel data, Pixel data surrounded by a solid line having the same color filter color combination is copied. When the color filter color is horizontally reversed between the pixel block of the reference pixel data and the pixel block of the effective pixel data of the copy source, the pixel data of the effective pixel data is inverted horizontally and becomes the pixel block of the reference pixel data. Copied.

  Similarly, pixel data surrounded by a broken line having the same color combination, which is the data of the pixel block at the upper end of the effective pixel data, is copied to the pixel block surrounded by the broken line as reference pixel data. Similarly, when the color filter color is inverted horizontally between the pixel block of the reference pixel data and the pixel block of the effective pixel data of the copy source, the pixel data of the effective pixel data is inverted horizontally and the reference pixel data Copied to pixel block.

  When the pixel copy processing at the upper end portion of the image and the pixel copy processing at the left end portion and the right end portion of the image are simultaneously executed with the color filter array shown in FIG. 10, the pixel size of the pixel block for search is individually set. Set to. More specifically, in the pixel copy process at the left end and right end of the image, the pixel size of the pixel block for search is set to “1” in the horizontal direction and “2” in the vertical direction. Set. On the other hand, in the pixel copy process at the upper end of the image, the pixel size of the pixel block for search is set to “2” for the number of pixels in the horizontal direction and “1” for the number of pixels in the vertical direction.

  Furthermore, the pixel size of the search pixel block can be changed in at least one of the horizontal direction and the vertical direction depending on the arrangement of the color filter colors in the color filter array of the image sensor. That is, the pixel block sizes (in the same direction) of the search pixel blocks do not have to be the same as in this embodiment, and may differ depending on the location.

Further, in the present embodiment, an example has been described in which a pixel block that is a copy source of pixel copy processing is searched from the end of an image by using a combination of color IDs of reference pixel data for searching. However, for example, the following processing method may be adopted.
If the position of the pixel block of the effective pixel data of the copy source at the edge of the image is known from the position on the color filter array of the pixel block of the reference pixel data, the pixel copy process is performed with the search process omitted It may be configured.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  100: Image processing device 104: Image processing circuit 203: Pixel copy processing unit

Claims (6)

Acquisition of pixel data output from an image sensor having a color filter having a color arrangement of M × N (M and N are positive integers) and having a color arrangement other than a Bayer arrangement Means,
The color is such that a repetitive pattern of the color arrangement in the color filter is held in each of the reference pixels when processing the pixel data, which is an area outside the edge of the image determined by the pixel data. Arranging means for arranging;
The pixel block having the same color combination as the color combination arranged by the arrangement unit with respect to the pixel block in the reference pixel area, which is an area composed of the reference pixels, with the horizontal direction or the vertical direction as the search direction, Search means for searching from an effective pixel area which is an area of an image based on pixel data;
Copy means for copying pixel data included in the pixel block in the effective pixel area searched by the search means as pixel data included in the pixel block in the reference pixel area;
I have a,
The search means is the pixel block having the same color combination as the color combination arranged by the arrangement means with respect to the pixel block in the reference pixel area, and is most similar to the pixel block in the reference pixel area. An image processing apparatus , wherein the pixel block that is near is searched from the effective pixel region . The search means starts the search from a pixel block closest to the pixel block in the reference pixel region among the pixel blocks in the effective pixel region, and detects the color arranged in the pixel block in the reference pixel region. The pixel blocks in the effective pixel region to be compared with the pixel blocks in the reference pixel region are sequentially changed along the search direction until the pixel block having the same color combination as the combination exists. The image processing apparatus according to claim 1 , wherein: The copy means selects, from the effective pixel area searched by the search means, pixel data corresponding to the same color as the color arranged by the arrangement means for the pixels of the pixel block in the reference pixel area to an image processing apparatus according to claim 1 or 2, characterized in that copy. The size of the pixel block, the image processing apparatus according to any one of claim 1 to 3, further comprising a setting means for setting according to the arrangement of the colors in the color filter. Based on the sequence of the colors in the color filter, the pixel size of the pixel block, any claim 1-4, characterized by further comprising a changing means for changing in accordance with the direction of the search in said searching means The image processing apparatus according to claim 1. Acquisition of pixel data output from an image sensor having a color filter having a color arrangement of M × N (M and N are positive integers) and having a color arrangement other than a Bayer arrangement Process,
The color is such that a repetitive pattern of the color arrangement in the color filter is held in each of the reference pixels when processing the pixel data, which is an area outside the edge of the image determined by the pixel data. An arrangement step of arranging,
The pixel block having the same color combination as the color combination arranged in the arrangement step with respect to the pixel block in the reference pixel area, which is an area composed of the reference pixels, with the horizontal direction or the vertical direction as the search direction, A search step for searching from an effective pixel area which is an area of an image based on pixel data;
A copy step of copying pixel data included in the pixel block in the effective pixel region searched in the search step as pixel data included in the pixel block in the reference pixel region;
I have a,
The searching step is the pixel block having the same color combination as the color combination arranged by the arrangement step with respect to the pixel block in the reference pixel region, and is the most similar to the pixel block in the reference pixel region. An image processing method characterized by searching for the near pixel block from the effective pixel region .

Images