TW202335489A - Method and image processor unit for processing raw image data - Google Patents

Abstract

The invention relates to a method for processing raw image data (IMG _RAW) provided by an image sensor, wherein the image data comprising a first array of pixel, said first pixel array being overlaid with a color filter array (3), so the pixel represent color information according to a first specific color pattern defined by the color filter array (3), said first specific color filter pattern comprising blocks of adjacent pixel assigned to the same color characterized by: a) Binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array; b) Demosaicing of the second pixel array according to a second color filter array (3) which is related to the color distribution pattern of the second pixel array; and c) Upscaling the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array (PA _IMD).

Description

Method and image processing unit for processing original image data

The present invention relates to a method for processing raw image data provided by an image sensor, wherein the image sensor includes a first pixel array, the first pixel array overlaps a first color filter array, so that the pixels are arranged according to the A first specific color pattern defined by the color filter array represents color information, the first specific color pattern including blocks of adjacent pixels assigned to the same color.

The present invention further relates to an image processing unit for processing raw image data provided by an image sensor, the image sensor including a first pixel array providing a sensor array including the first pixel data array. An image in which the first pixel array overlaps with the color filter array such that the pixels represent color information according to a first specific color pattern defined by the color filter array, the first specific color pattern including adjacent pixels assigned to the same color. block of pixels.

The invention further relates to a computer program comprising instructions for performing the above method.

The pixel array of a standard digital image sensor, especially a CMOS image sensor, is designed to capture light without being biased towards any particular wavelength. The resulting image will be a monochrome image without any color information. To generate color information, the pixel array overlaps with a color filter array, which is a grid of filters for different wavelengths. The most common color filter array pattern is the Bayer color filter, which consists of 2x2 basic units, containing one red (red, R), two green (green, G) and one blue (blue, B) Pixel unit point. The basic unit is repeated across the entire pixel array of the image sensor, producing an image with R, G, and B pixels. Due to this process, color information is incomplete. The process of reconstructing missing color information, for example, the R, G, and B of each pixel, is called demosaicing.

In addition to Bayer filters, other higher-order color filter array patterns exist, such as QuadBayer, HexaDeca or 6x6 CFA. In low light conditions, pixels acquired with these color filter arrays can be synthesized by combining blocks of pixels of the same color. The individual colored pixels again form a Bayer pattern. This will reduce the image resolution, but will increase the signal to noise ratio (SNR) of the image.

Cui, K.; Jin, Z. & Steinbach, E. In 2018 "Color Image Removal Using 3-Level Convolutional Neural Network Structure" at the 25th IEEE International Conference on Image Processing (ICIP) in 2018 Mosaic" describes a 3-level convolutional neuron network structure for color demosaicing, in which the green channel is independently reconstructed in the first stage. By using the reconstructed green channel as a guide, the red and blue channels are restored in the second stage. Finally, the 3-quality RGB color image is reconstructed in the third stage.

US 2018/0357750 A1 describes a demosaicing method for an image having a multi-unit mosaic pattern, where the multi-unit mosaic image includes a first quarter, a second quarter, a third quarter, and Fourth quarter. Each quarter includes a plurality of cells, and each cell includes a pixel value. An image segmentation process was performed on the image to obtain four Bayer plane images. On each of the four quarter-resolution planar images, a quarter-resolution demosaic process is performed to obtain a quarter-resolution image in each color channel. Finally, the four quarter-resolution images are combined in each color channel to produce a full-resolution image in each color channel.

Hirakawa, K. & Parks, T.W., "Adaptive Uniform Oriented Demosaic Algorithm" in IEEE Transactions on Image Processing, IEEE, 2005, 14, 360~369: "Adaptive Uniform Oriented Demosaic Algorithm" describe the inherent limitations commonly associated with directional interpolation methods of demosaicing algorithms. The problems, namely, misdirection color artifacts, interpolated color artifacts, and aliasing. A demosaicing algorithm is proposed that estimates missing pixels by interpolating in directions with fewer color artifacts. The aliasing problem is solved by applying filter bank technology to directional interpolation. Use a nonlinear iterative process to reduce interpolation artifacts. This is a suitable standard algorithm for demosaicing standard Bayer pattern images.

The problem is dealing with a variety of higher-order color filter array patterns in addition to the standard Bayer color filter array. This requires highly sophisticated hardware and software resources.

It is an object of the present invention to provide an improved method for processing raw image data provided by an image sensor having a first specific color filter pattern using a standard demosaicing process based on a second color filter pattern. The array handles the second array of pixels.

This object is achieved by a method comprising the steps of claim 1, the image processing unit of claim 15, and the computer program of claim 17. Preferred embodiments are described in the accompanying claims.

To process image data including a first array of pixels according to a first specific color pattern, a demosaicing process configured for demosaicing a second array of pixels according to a second color filter array is used. The method consists of the following steps: a) Combine blocks of adjacent pixels assigned to the same color downward to obtain a second pixel array. Compared with the image of the first pixel array, the image size and resolution of the second pixel array are reduced; b) demosaicing the second pixel array according to the second color filter array associated with the color distribution pattern of the second pixel array; and c) Enlarge the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to obtain an enlarged third pixel array.

It is recommended that the pixels are first downcomposited into a low resolution second pixel array based on a second color filter array, e.g. a Bayer pattern image. Next, a standard well-known demosaicing algorithm for the second pixel array according to the second color filter array, for example, standard Bayer demosaicing, is used to demosaic the image data. Then, the demosaiced image is upsampled to the original high resolution of the first pixel array according to the first color filter array to obtain and amplify the third pixel array.

This allows the use and implementation of only one standard process for demosaicing, while the raw image data according to the first color filter array are adapted to the second color filter by individual synthesis of pixels.

Therefore, according to the present invention, the resolution of the original image data is first reduced by the demosaicing process and then increased back by enlarging the demosaiced image.

Preferably, the enlarged third pixel array can be further reconstructed by using the original image data. Therefore, information from the original original image data provided in the first pixel array according to the first color filter array is used to improve the quality of the enlarged image including the third pixel array.

The advantage of this approach is that core standard demosaicing algorithms can be used, for example, the standard Bayer demosaicing process. There is no need to have an algorithm on hand that can demosaic the high-order patterns intact. No frame buffer is required because this method can operate on a collection of several frame lines at once. No specific algorithm is required for native color filter array patterns. A well-known building block algorithm can be used as the core demosaicing process, where the method can still be used for high-order color filter pattern demosaicing.

The method may further comprise the step of filtering the pixel data in the demosaiced second pixel array after step b) to reduce aliasing artifacts which may occur throughout the further method due to e.g. in critical sampling areas Occurs during synthesis. This filter is optional.

Step a) of compositing blocks of adjacent pixels assigned to the same color may be performed by calculating an average of the pixels of a color block in the first pixel array. The average may, for example, be calculated as the mean or median of pixel values in a block of directly adjacent pixels. For example, the pixel values of blocks of the same color pixels, such as 2x2, 3x3, 4x4, 5x5, etc., can be averaged to obtain the resulting pixel values of the second pixel array with reduced image size and resolution.

The blocks of adjacent pixels of the same color assigned to the same pixel array in step a) can be applied to a quad Bayer 4x4 or HexaDeca 8x8 color filter array to down-synthesize such first pixel array into a Bayer 2x2 filter array. The second pixel array of the color device array.

Step c) of upscaling the demosaiced second pixel array may be handled by nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos resampling or spline interpolation, or a combination thereof. These are the preferred methods for scaling up pixel arrays among others.

Before performing the step b) of demosaicing the second pixel array, the step a) of compositing blocks of adjacent pixels assigned to the same color may be repeated. It is obtained after two subsequent steps of a first down-synthesis of a block into a block of an intermediate second pixel array, and a second down-synthesis of this block of intermediate second pixels into a final second pixel array. By repeating the compositing step a) at least twice, the high-order pixel array based on the high-order color filter array can be down-composited into a low-order pixel array based on the low-order color filter array, for example, HexaDeca 8x8 color filter The filter array is combined down into a Bayer 2x2 color filter array.

Step c) of upscaling the demosaiced second pixel array can be repeated in two subsequent steps: - first upscaling the demosaiced final pixel array to an intermediate third pixel array having a size and color distribution pattern according to the intermediate second pixel array, and - Enlarge the middle third pixel array for a second time to the size and color distribution pattern of the first pixel array to obtain an enlarged third pixel array.

Therefore, in the step of amplifying the low-order pixel array according to the low-order color filter array, for example, the Bayer 2x2 color filter array, the amplification can also be performed according to the high-order color filter array, for example, the HexaDeca 8x8 color filter array. Return to the higher-order pixel array.

After each upscaling step, the resulting pixel array can be reconstructed by using the original image data of the final third pixel array and by using the intermediate second pixel array for the intermediate third pixel array obtained in the first synthesis step. Therefore, the initial image information generated from the original image data is reused to improve the image quality of the enlarged intermediate second pixel array, resulting in an improved image including the final third pixel array.

The enlarged intermediate third pixel array obtained after step c) can be reconstructed by replacing the pixels in the enlarged third pixel array with available pixels from the relevant image data (ie, the original image data or the intermediate second pixel array) , having the same pixel positions while maintaining the color channel differences present in the amplified third pixel array.

The reconstruction of the enlarged (middle) third pixel array can be performed by regional edge detection of the block surrounding the third pixel array based on the relevant image data (i.e., the original image data or the middle second pixel array), and by using Interpolation of blocks in individual color channels is performed by detecting edge orientation and color channel differences present in the amplified third pixel array.

The reconstruction step as described in claims 11 and 12 may each comprise the step of determining a block error metric indicating the adaptation of the replaced low-order block to its neighborhood, and applying the block error metric as described in claims 11 and 12 The results of the reconstructed amplified third pixel array are combined into a final reconstructed pixel array in which pixels or blocks are selected based on the associated block error metric.

This allows processing of at least two specific methods for replacing pixels and estimating which reconstruction strategy is best for individual blocks.

An optional step of filtering the pixel array may be performed after individual reconstruction steps and/or by filtering the final reconstructed pixel array.

The image processing unit may be designed as image processing hardware, such as an ASIC, FPGA, or a suitable program microprocessor, and the hardware is configured to process image data by executing the above method steps.

The method may be implemented as a computer program containing instructions. When the program is executed by a processing unit, the instructions cause the processing unit to process the original image data provided by the image sensor using the steps of the method according to the invention.

Figure 1 illustrates a schematic block diagram of an image camera 1 including a photoelectric sensor 2. The photoelectric sensor 2 includes a first pixel array, and the first pixel array overlaps with the color filter array 3.

Such a color filter array 3 on top of the photo sensor 2 optically blocks certain wavelengths to obtain color information for specific pixels. The color filter array 3 is a grid of filters for different wavelengths. By using such a color filter array 3, the monochromatic pixels of the photo sensor 2 are assigned to specific colors. The pixel value of an individual pixel of the photoelectric sensor 2 indicates the brightness of the color of the image at that individual pixel position.

The original image data IMG _RAW of the photoelectric sensor 2 is provided to the image processing unit 4. The image processing unit 4 is configured to demosaic the original image data IMG _RAW to obtain the final image IMG in a specific manner as explained in more detail below. _FIN .

Figure 2 shows different pixel arrays assigned to different color filter arrays.

At a), a well-known low-order Bayer pixel array assigned to a Bayer color filter array is shown. Many 2x2 blocks including red (red, R), green (green, G), and blue (blue, B) pixels are repeated on the 8x8 pixel array. Several well-proven algorithms for demosaicing this Bayer pixel array are available in the prior art and can be used in the present method of processing raw image data.

At b), a 6x6 CFA pattern is shown, which forms a higher order color filter array pattern compared to the Bayer color filter array pattern. The 6x6 block contains four 3x3 blocks of individual colors red (red, R), green (green, G), and blue (blue, B). The 3x3 blocks of color-specific RGB are configured according to the color configuration of the 2x2 blocks in the Bayer color filter array pattern (R-G-G-B).

At c), a four-Bayer color filter array pattern is shown. High-order pixel arrays contain pixels in an 8x8 matrix with repeated 4x4 blocks, with each 4x4 block formed from four 2x2 blocks for individual colors R, G, and B. The 2x2 blocks of individual color RGB are arranged in a low-level Bayer pattern R-G-G-B.

At d), a HexaDeca color filter array pattern formed by an 8x8 matrix is presented. Here, four blocks of 4x4 array size are assigned to individual colors R, G, and B.

The four blocks of individual colors are arranged in an order similar to the low-order Bayer color filter array pattern shown at a).

Figure 3 shows a flow chart of a method of processing raw image data IMG _RAW provided by an image sensor, and the raw image data IMG _RAW includes a first pixel array, wherein each pixel is defined by a specific first color filter array The first specific color pattern represents color information.

The original image data IMG _RAW can be, for example, the high-order color filter array shown in b), c) and d) of Figure 2, that is, the first pixel array of a 6x6 CFA pixel array, a quad Bayer pixel array or a HexaDeca pixel array .

The original image data IMG _RAW can also be the first pixel array assigned to another more complex color filter array, such as Sony® RGBW, RYYB, MYYC, Fuji® X-Trans® and the like.

The original image data IMG _RAW is processed in the first step a) by down-compositing blocks of adjacent pixels assigned to the same color to obtain a second pixel array. Compared with the image of the first pixel array, the second pixel array is obtained. The image size and resolution of the two-pixel array are reduced.

For example, the 2x2, 3x3 or 4x4 blocks of the same color in the high-order pixel array in b), c) or d) of Figure 2 can be down-synthesized into a low-order pixel array (e.g., a in Figure 2) A pixel value for an individual color R, G or B of a Bayer pixel array).

A set of pixel values of a block of pixels assigned to the same color may be averaged to receive a resulting pixel value, for example, a related pixel of a low-order pixel array.

In the second step b), demosaicing of the second pixel array is performed, the second pixel array being configured according to the second color filter array in relation to the color distribution pattern of the second pixel array. This allows the use of standard procedures for demosaicing low-level pixel arrays containing color information according to low-level specific color patterns defined by individual second color filter arrays. For example, the demosaic step may be performed on a second array of pixels configured according to a low-order Bayer color filter array pattern.

An optional step (e) of filtering the pixel data obtained by demosaicing step b) can be performed to reduce aliasing artifacts.

Then in step c) the demosaiced and optionally filtered second pixel array is enlarged to the size of the color distribution pattern of the first pixel array to obtain an enlarged third pixel array PA _IMD . Therefore, the second array of pixels received after reducing the original image data by synthesis step a) is again re-amplified to the pixel size specified in the first array of pixels in the associated first specific color pattern of the first color filter array and color distribution.

In the final step d), the enlarged third pixel array is reconstructed by using the original image data. Therefore, the initial pixel information of the original image data IMG _RAW is used to improve the image received after step c) of enlarging the second pixel array of demosaicing.

The result is a final demosaiced image containing a resolution based on the first pixel array of the original raw image data.

Figure 4 shows a block diagram of a method for processing the original image data IMG _RAW according to Figure 3, but repeating the steps a1) for compositing the original image data IMG _RAW and a2 for compositing the intermediate second pixel array IMG _IMD ), IMG _IMD is received after the first step a1), which combines blocks of adjacent pixels of the same color assigned to the first pixel array downward to obtain an intermediate image with reduced size and resolution. Second pixel array.

The second subsequent step a2) of compositing the intermediate second pixel array IMG _IMD is to combine pixels of the same color that are arranged adjacent to each other in the same sub-block to receive a second pixel array with a standard color distribution. The subsequent step b) of demosaicing this second pixel array applies to a specific standard pixel array having the color distribution of the respective standard second color filter array. Therefore, by performing the two subsequent steps a1) and a2) of down-synthesizing the initial raw image data IMG _RAW into image data based on the second pixel array, the color distribution pattern is adapted to the standard second color filter array. This allows the standard process of demosaicing the second pixel array according to the second color distribution pattern on the original image data IMG _RAW with a different color distribution pattern.

After the optional step (e) of filtering the enlarged second pixel array, a first step of enlarging the demosaiced second pixel array to the intermediate pixel array PA _IMD1 according to the size and resolution of the intermediate color pixel distribution pattern is performed. c1). In step d1), the obtained intermediate third pixel array IMG IMD is modified by using the intermediate image data IMG _IMD1 (ie, the intermediate second pixel array IMG _IMD obtained after the first step a1 of synthesizing the initial raw image data IMG _RAW ). Pixel array is reconstructed.

Then in step c2), the obtained intermediate third pixel array is enlarged into an intermediate pixel array PA _IMD2 according to the size and color distribution pattern, corresponding to the original first pixel array including the pixel distribution pattern according to the first color filter array.

Then in step d2), the enlarged pixel array PA _IMD2 (corresponding to the original image data demosaiced according to the first pixel array) is reconstructed by using the original image data IMG _RAW to receive the final image including the third pixel array. Image img _fin .

Therefore, the third pixel array received by this method is (almost) similar to direct demosaicing of the original image data including the first pixel array according to the specific first color pattern.

Figure 5 schematically illustrates the two subsequent steps of synthesizing blocks of the first pixel array according to the HexaDeca 8x8 RGB pattern into an intermediate second pixel array according to the quad Bayer 4x4 pattern and a second pixel array according to the Bayer 2x2 pattern. The first step in compositing the HexaDeca 8x8 pixel array a1) reduces the array of size mxn (8x8) to half size m/2xn/2 (i.e. 4x4). The second step a2) of synthesizing four Bayer 4x4 pixel arrays into a final second pixel array according to the Bayer 2x2 pattern reduces the size to m/4xn/4. In this example, each synthesis step reduces the size of the input pixel array by half.

The compositing step is performed by combining 2x2 pixel blocks of the same color into a single pixel value. For example, the compositing process can start with the first and second upper lines on the left, where four pixel values are assigned to the red R, ie, the first 2x2 block. For example, pixel values are combined by estimating the mean or median of four pixels to obtain one pixel value for red at the upper left pixel position of a four-Bayer 4x4 intermediate pixel array. Repeat this process for the following 2x2 blocks assigned to red, then green G, and finally blue B.

In the same way, the second step a2) of the synthesis combines the 2x2 blocks of pixels assigned to the same color R, G or B into one pixel value assigned to the same color R, G and B to obtain the Bayer 2x2 according to the standard Pattern the second pixel array.

The standard demosaicing process in step b) is then performed on this reduced standard second pixel array.

Figure 6 shows a block diagram of the block-by-block processing of step d) of reconstructing the enlarged third pixel array after the demosaic process has been performed and the resulting pixel array is enlarged to a high-order color filter array pattern.

The enlarged pixel array PA _IMD and the original image data IMG _RAW of the individual pixel array are forwarded as input to the processing block R1_a) for pixel replacement. Here, the pixels of the enlarged third pixel array (ie, the intermediate pixel array PA _IMD ) are replaced by individual pixel values of the original image data IMG _RAW while maintaining the color difference values estimated in the previous demosaicing process. Therefore, the detail reconstruction process uses the original image data IMG _RAW according to the first specific color pattern to reconstruct the details in the enlarged image PA _IMD . However, the raw image data contains only block-by-block information for only one color at a time (e.g., 2x2 R-pixels for Quad Bayer or 4x4 R-pixels for HexaDeca). Processing is performed block by block, for example, sequential 2x2 Quad Bayer or 4x4 HexaDeca blocks. There are two paths that are combined in step R_d) for the final output, where the left path is a replacement strategy using the original original image data IMG _RAW , and the right path is an interpolation strategy that takes into account the surrounding color filter array.

In step R1_a) of the right path, the pixels in the upscaled image PA _IMD are replaced with available pixels from the original image data of the initial color filter array, while maintaining the color channel differences present in the upscaled image. This will be explained in more detail later in conjunction with Figure 7.

After the optional filtering step R1_b), in step R1_c) a block error metric is calculated to see if the replaced block (e.g. 2x2 Quad Bayer or 4x4 HexaDeca) fits its neighborhood well.

The left path performs regional edge detection on the surrounding blocks based on the initial raw image data IMG _RAW according to the first color filter array in step R2_a), and takes into account the detected edge orientation in individual color channels in step R2_b) blocks are interpolated. Also here, the color channel differences from the upscaled image are used to recover full color information, for example, RGB information. In this path, another optional step R2_c) of filtering is also possible before calculating the block error metric in step R2_d).

Based on the previously calculated error metric, the final decision is made whether to obtain the pixel value from the right or left path. The merging of the two paths in step R_d) may be a hard decision or a method similar to soft mixing.

Final post-processing filtering of color pixels in step R_e) may optionally be used to enhance image quality.

The result is the final image IMG _FIN .

Pixel replacement in step R1_a) can be performed, for example, using the following equation:

For selected green pixel positions within individual color filter arrays: d _{G -R} =G _Dem -R _Dem d _{G -B} =G _Dem -B _Dem G _replace =G _RAW R _replace =G _RAW -d _{G -R} B _replace =G _RAW -d _{G -B}

For selected red pixel positions within individual color filter arrays: d _{R -B} =R _Dem -G _Dem d _R -B =R _{Dem -B Dem} R _replace =R _RAW G _replace =R _RAW -d _{R -G} B _replace =R _RAW -d _{R -B}

For selected blue pixel positions within individual color filter arrays: d _BG =B _Dem -G _Dem d _BR =B _Dem -R _Dem B _replace =B _RAW R _replace =B _RAW -d _BR G _replace =B _RAW -d _BG

For the green pixel G position, the green pixel is replaced by the original green pixel data G _RAW , and the demosaiced red pixels R and blue pixels B used for the original green pixel position are subtracted from the original green pixel G _RAW by individual differences d _{G -R} or d _{G -B} to replace.

The difference between the green, red or blue pixels is determined by the demosaiced green pixel G _Dem , the red pixel R _Dem , the blue pixel B _Dem as the basic pixel and the demosaiced associated demosaiced red or demosaiced red pixel at the position of the basic green pixel. Calculated by the difference of blue pixels (existing in the demosaiced image PA_IMD). In the case where a basic red pixel exists in the original image data, a difference value is calculated based on the demosaiced red pixel R _Dem and the demosaiced green and blue pixels for this basic red pixel. Similarly, for the blue pixel B _RAW , the difference is calculated by the demosaic blue pixel B _Dem and the associated demosaic red pixel R _Dem or green pixel G _Dem .

Therefore, when a 1x1 pixel block of a specific color RGB is obtained from a high-order pixel block of the same color (for example, a 4x4 pixel array), the color vectors G _Dem , R _Dem and B _Dem . This color vector is combined with the original array of pixels assigned to the original image data for a specific color (eg, green G). Enlarging the demosaiced 1x1 pixel block produces a higher-order pixel array, for example, a 4x4 pixel array, and then combining the 4x4 pixel array with the original pixel array for that color. The color of the initial pixel array specifies the basic color, thereby specifying the above equation for the raw image data G _RAW , R _RAW or B _RAW for green, red or blue.

Figure 7 shows a block diagram of a functional circuit for performing pixel replacement in step R1_a) of Figure 6.

Raw image data for blocks in the form of CFA data R _CFA , G _CFA and B _CFA for three colors RGB are provided to the addition logic ADD, the subtraction logic SUB and the multiplexers MUX1, MUX2 and MUX3. In addition, the enlarged RGB images of the blocks in these three channels R _Upscale , G _Upscale and B _Upscale are input to the subtraction logic SUB to calculate the differences d _{G -R} , d _{G -B} , by using the above equations. _dBG , _dBR , _{dR -G} , _{dR -B} .

Figure 7 shows an example of a block diagram for blue block B, where the differences d _{G -B} and d _{G -R} (GB _Upscale and GR _Upscale ) are calculated. Multiplexers MUX1, MUX2 and MUX3 are controlled together independently of the block type to provide raw image data R _CFA , G _CFA for red block type, green block type or blue block type at MUX1, MUX2 or MUX3 or B _CFA . Furthermore, corrected pixel values for other colors are provided at the output of the multiplexers MUX1, MUX2 or MUX3 for which the original raw material R _CFA , G _CFA or B _CFA is not selected. Because pixel values may leave the allowed value range through this process, pixel values need to be clipped to zero and to the maximum. This is controlled by CLIP logic at the outputs of individual multiplexers MUX1, MUX2 and MUX3.

Figure 8 is related to the "interpolation" step R2_b) in Figure 6. In an exemplary case, the edge is not detected by the area edge detection. In this regard, it takes the blue block as an example to propose an example of estimating the surrounding blocks on the left path in Figure 6. The green pixel G _I can be estimated by the average of the four neighboring pixel values around the green pixel G _I .

This can be expressed by the following equation: G ₀ =(t ₀ +t ₁ +I ₀ +I ₁ )/4 G ₁ =(t ₀ +t ₁ +r ₀ +r ₁ )/4 G ₂ =(I ₀ +I ₁ +b ₀ +b ₁ )/4 G ₃ =(b ₀ +b ₁ +r ₀ +r ₁ )/4

For blue pixels, the same scheme applies: B ₀ =(t ₀ +t ₁ +I ₀ +I ₁ )/4 B ₁ =(t ₀ +t ₁ +r ₀ +r ₁ )/4 B ₂ =(I ₀ +I ₁ +b ₀ +b ₁ )/4 B ₃ =(b ₀ +b ₁ +r ₀ +r ₁ )/4

This works with a color quad Bayer 2x2 pixel array.

The red pixel values r _{0 , 1 , 2 , 3} can be calculated by using the pixel replacement method R1_a) on the right path in Figures 6 and 7 by the following equation: R ₀ =G ₀ -(GR _Upscale ) R ₁ =G ₁ -(GR _Upscale ) etc.

For other high-order pixel array blocks, such as HexaDeca, the pixel values at the outer corners can be calculated as described above, such as G ₀ =(t ₀ +t ₁ +I ₀ +I ₁ )/4, etc.

Calculate the inner pixel value based on the outer surrounding pixels, for example G ₆ =(t ₂ +t ₃ +r ₀ +r ₁ )/4 G ₁₀ =(r ₂ +r ₃ +b ₂ +b ₃ )/4.

The outer middle pixel value can be estimated as the average of three adjacent surrounding pixels, for example, G ₁₄ =(b ₁ +b ₂ +b ₃ )/3 G ₈ =(I ₁ +I ₂ +I ₃ )/3.

The same applies to blue pixels.

For an exemplary four-Bayer 2x2 block of a specific color, the error metric for the scheme on the left in Figure 6 can be calculated by the following equation: E=|G0-t0|+|G0-I0|+|G1-t1|+|G1-r0|+|G2-I1|+|G2-b0|+|G3-r1|+|G3-bl|.

The error value can be calculated as follows

In case this error value ε is greater than a predetermined threshold value for a specific color, for example t _G for green pixel G or t _B for blue pixel B, the block is replaced by this reconstruction scheme.

1: Image camera 2: Photoelectric sensor 3: Color filter array 4: Image processing unit B _CFA , G _CFA , R _CFA : CFA data B _Upscale , G _Upscale , R _Upscale : Channel GB _Upscale : Difference d _{G - B} GR _Upscale : difference d _{G -R} IMG _FIN : final image IMG _IMD : middle second pixel array IMG _RAW : original image data PA _IMD : amplified third pixel array PA _IMD1 , PA _IMD2 : middle pixel array

The invention is explained in more detail by means of exemplary embodiments with accompanying drawings. It shows:

Figure 1: Schematic block diagram of the image processing unit;

Figure 2: An example of a pixel array based on a specific color filter array;

Figure 3: Flowchart of the method used to process the original image data to demosaic the image data to obtain the final image;

Figure 4: Flowchart of the method for the two subsequent steps of synthesizing blocks and reconstructing individually enlarged pixel arrays;

Figure 5: An exemplary pixel array for performing steps of synthesizing blocks of adjacent pixels of a high-order color filter array into a low-order color filter array;

Figure 6: Flowchart of a method for reconstructing an enlarged pixel array;

Figure 7: Exemplary hardware logic for reconstructing an amplified pixel stream of a pixel array;

Figure 8: Four Bayer blocks and HexaDeca blocks for estimation from surrounding pixels.

Domestic storage information (please note in order of storage institution, date and number) without Overseas storage information (please note in order of storage country, institution, date, and number) without

1:Image camera

2: Photoelectric sensor

3: Color filter array

4:Image processing unit

IMG _FIN : final image

IMG _RAW : original image data

Claims (17)

A method for processing raw image data (IMG _RAW ) provided by an image sensor, wherein the image data includes a first pixel array overlapping a color filter array (3), such that the pixel represents color information according to a first specific color pattern defined by the color filter array (3), said first specific color filter pattern including a plurality of blocks of adjacent pixels assigned to the same color, The method is characterized by the following steps: a) Down-synthesize the blocks of adjacent pixels assigned to the same color to obtain an image with reduced size and resolution compared to the image of the first pixel array. a second pixel array; b) demosaicing the second pixel array according to a second color filter array (3) associated with the color distribution pattern of the second pixel array; and c) demosaicing the demosaiced The second pixel array is enlarged to the size of the color distribution pattern of the first pixel array to obtain an enlarged third pixel array (PA _IMD ). The method of claim 1, characterized by the following steps: d) reconstructing the amplified third pixel array (PA _IMD ) by using the original image data (IMG _RAW ). The method according to claim 1 or 2, characterized by the following steps: e) After step b), filter the pixel data in the demosaiced second pixel array to reduce multiple aliasing artifacts. The method according to any one of claims 1 to 3, characterized in that the comparison of adjacent pixels assigned to the same color is performed by calculating an average value of the pixels of a color block in the first pixel array. Step a) of synthesizing the blocks of pixels. The method of claim 4, wherein the average value is calculated as a mean or median value of the pixel values in a block of directly adjacent pixels. Method according to one of the preceding claims, characterized in that in step a) a plurality of the same color assigned to the first pixel array according to a quad Bayer 4x4 or HexaDeca 8x8 color filter array pattern (3) The blocks of adjacent pixels are combined down into a second pixel array according to the Bayer 2x2 color filter array pattern (3). The method according to one of the preceding claims, wherein the step c) of amplifying the demosaiced second pixel array is performed by nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos re-interpolation Processed by sampling or spline interpolation. The method according to one of the preceding claims, characterized in that, before performing the step b) of demosaicing the second pixel array, the regions of adjacent pixels assigned to the same color are repeatedly de-mosaicd. In the step a) of block synthesis, the second pixel array is composed of the blocks of the first pixel array for the first time into an intermediate second pixel array (IMG IMD), and the intermediate second pixel array (IMG _IMD ). The blocks of the pixel array (IMG _IMD ) are obtained after a second down-synthesis of the two subsequent steps of the final second pixel array. The method of claim 8, characterized in that step c) of amplifying the demosaiced second pixel array is repeated in the following two subsequent steps: a first amplification of the demosaiced final second pixel array to an intermediate third pixel array having a size and a color distribution pattern according to the intermediate second pixel array (IMG _IMD ), and enlarging the intermediate third pixel array to the size of the first pixel array for a second time and the color distribution pattern to obtain the amplified third pixel array (PA _IMD ). The method of claim 9, wherein after each upscaling step, the resulting pixel array is obtained by using the original image data (IMG _RAW ) for the final third pixel array and by using the IMG RAW in the first synthesis step. The intermediate second pixel array (IMG _IMD ) obtained for the intermediate third pixel array is reconstructed. The method according to one of the preceding claims, characterized in that the enlarged middle third pixel array is reconstructed by: using data from the relevant image data (ie, the original image data (IMG _RAW ) or The available pixels of the intermediate second pixel array (IMG _IMD ) replace the pixels in the amplified third pixel array (PA _IMD ) with the same pixel position while maintaining the amplified third pixel array (PA _{IMD )} . ) exists in multiple color channel differences. The method according to one of the preceding claims, characterized in _that by mapping an individual pixel _or Area edge detection is performed around the block of pixels, and by using the detected edge orientation and the color channel differences present in the amplified third pixel array (PA _IMD ), The step of interpolating the block in the individual color channel to reconstruct the enlarged (middle) third pixel array. A method as claimed in claims 11 and 12 in combination, wherein the reconstruction steps as claimed in claims 11 and 12 each comprise the step of determining a number indicative of the adaptation of the replaced low-order blocks to their neighbours. block error metrics, and combining the results of the reconstructed upscaled third pixel array (PA _IMD ) obtained by the reconstruction according to claims 11 and 12 into a selection based on the associated block error metrics A final reconstructed pixel array of the pixels or the blocks. Method according to one of claims 11 to 13, characterized by the step of filtering the pixel array obtained after the individual reconstruction step and/or filtering the final reconstructed pixel array. An image processing unit (4) for processing raw image data (IMG _RAW ) provided by an image sensor. The image sensor includes a sensor array including a first pixel array. An image of data from a pixel array overlapping a first color filter array (3) such that the pixels are represented according to a first specific color pattern defined by the color filter array (3) Color information, the first specific color pattern includes multiple blocks of adjacent pixels assigned to the same color, and the image processing unit (4) is characterized in that it is configured to: a) convert the adjacent pixels assigned to the same color The blocks are combined downward to obtain a second pixel array with reduced image size and resolution compared to the image of the first pixel array; b) according to the color distribution pattern associated with the second pixel array a second color filter array (3) demosaicing the second pixel array; and c) enlarging the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to obtain An amplified third pixel array (PA _IMD ). The image processing unit of claim 15, characterized in that the image processing unit (4) is configured to process image data by performing the method steps of one of claims 1 to 14. A computer program comprising instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the method as described in one of claims 1 to 14.