WO2023155999A1

WO2023155999A1 - Method and image processor unit for processing raw image data

Info

Publication number: WO2023155999A1
Application number: PCT/EP2022/054104
Authority: WO
Inventors: Gregor Schewior; Julian HARTIG; Martin Köhler
Original assignee: Dream Chip Technologies Gmbh
Priority date: 2022-02-18
Filing date: 2022-02-18
Publication date: 2023-08-24
Also published as: TW202335489A

Abstract

The invention relates to a method for processing raw image data (IMGRAW) provided by an image sensor, wherein the image data comprising a first array of pixel, said first pixel array being overlaid with a color filter array (3), so the pixel represent color information according to a first specific color pattern defined by the color filter array (3), said first specific color filter pattern comprising blocks of adjacent pixel assigned to the same color characterized by: a) Binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array; b) Demosaicing of the second pixel array according to a second color filter array (3) which is related to the color distribution pattern of the second pixel array; and c) Upscaling the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array (PAIMD).

Description

Method and image processor unit for processing raw image data

The invention relates to a method for processing raw image data provided by an image sensor, wherein the image sensor comprises a first array of pixel, said first pixel array being overlaid with a first color filter array, so that the pixel represents color information according to a first specific color pattern defined by the color filter array, said first specific color pattern comprises blocks of adjacent pixel assigned to the same color.

The invention further relates to an image processor unit for processing raw image data provided by an image sensor, said image sensor comprises a sensor array of a first pixel array providing an image comprising a first array of pixel data, said first pixel array being overlaid with a color filter array so that the pixel represents color information according to a first specific color pattern defined by the color filter array, said first specific color pattern comprises blocks of adjacent pixel assigned to the same color.

The invention further relates to a computer program comprising instructions to execute the above-referenced method.

The pixel array of a standard digital image sensor, in particular CMOS image sensor, is designed to capture light without favoring any specific wavelengths. The resulting image would be a monochromatic image without any color information. To generate the color information, the pixel array is overlaid with a color filter array which is a grid of optical filters for different wavelengths. The most common color filter array pattern is the Bayer color filter which consists of 2x2 basic cell containing one red (R), two green (G) and one blue (B) pixel cell point. A basic cell is repeated over the entire pixel array of the image sensor generating an image with R, G and B pixels. Due to this process, the color information is incomplete. The process of reconstructing the missing color information, e.g. reconstructing R and G and B for every pixel is known as demosaicing.

Next to the Bayer color filter, other higher-order color filter array patterns like QuadBayer, HexaDeca or 6x6 CFA exist. Under low light conditions, the pixels acquired with these color filter arrays can be binned by combining blocks of pixel of the same color. Pixels of the respective color form again a Bayer pattern. This reduces image resolution but increases the signal to noise ratio (SNR) of the image.

Cui, K.; Jin, Z. & Steinbach, E.: „Color image demosaicing using a 3-stage convolutional neural network structure”, 2018 25th IEEE International Conference on Image Processing (ICIP), 2018 describes a 3-stage convolutional neural network structure for color demosaicing wherein the green-channel is reconstructed independently in a first stage. By using the reconstructed green-channel as guidance, the red- and blue-channels are recovered in the second stage. Finally, 3-quality RGB color images are reconstructed in the third stage.

US 2018/0357750 A1 describes a demosaicing method for an image having a multicell mosaic pattern, wherein the multi-cell mosaic pattern includes a first quarter, a second quarter, a third quarter, and a fourth quarter. Each quarter includes multiple cells, and each cell includes a pixel value. An image-dividing process is performed on the image to obtain four Bayer plane images. On each of the four quarter-resolution plane images, a quarter-resolution demosaicing process is performed to obtain a quarter-resolution image in each color channel. Finally, the four quarter-resolution images are combined in each color channel to generate a full-resolution image in each color channel.

Hirakawa, K. & Parks, T. W.: “Adaptive homogeneity-directed demosaicing algorithm”, IEEE Transactions on Image Processing, IEEE, 2005, 14, 360-369 describes inherent problems often associated with directional interpolation approach to demosaicing algorithms, namely misguidance color artifacts, interpolation color artifacts and aliasing. A demosaicing algorithm is proposed, which estimates missing pixels by interpolating in the direction with fewer color artifacts. The aliasing problem is addressed by applying filter bank techniques to directional interpolation. The interpolation artifacts are reduced using a non-linear iterative procedure. This is an applicable standard algorithm for demosaicing of a standard Bayer pattern image.

The problem is to process a variety of higher-order color filter array patterns in addition to the standard Bayer color filter array. This requires highly complex hard- and software resources.

The object of the present invention is to provide an improved method for processing raw image data provided by an image sensor with a first specific color filter pattern with a standard process of demosaicing directed to process a second pixel array according to a second color filter array.

The object is achieved by the method comprising the steps of claim 1 and the image processor unit according to claim 15 and the computer program according to claim 17. Preferred embodiments are described in the dependent claims.

In order to process image data comprising a first array of pixel according to a first specific color pattern by use of a demosaicing process arranged for demosaicing a second pixel array according to a second color filter array. The method comprises the steps of: a) Binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array; b) Demosaicing of the second pixel array according to a second color filter array which is related to the color distribution pattern of the second pixel array; and c) Upscaling the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array.

It is proposed to first bin the pixels down to a low resolution into a second pixel array according to a second color filter array, e.g. a Bayer pattern image. Then, the standard well-known demosaicing algorithm for the second pixel array according to the second color filter array, e.g. a standard Bayer demosaicing is used for demosaicing the image data. The demosaiced image is then upsampled to the initial high resolution of the first array of pixel according to the first color filter array to achieve and upscaled third pixel array.

This allows the use and implementation of only one standard procedure for demosaicing, while the raw image data according to the first color filter array is adapted to the second color filter array by respective binning of pixels.

Thus, according to the present invention the resolution of the raw image data is first reduced for the demosaicing process and then increased back by upscaling the demosaiced image.

Preferably, the upscaled third pixel array can be further reconstructed by use of the raw image data. Thus, the information in the original raw image data provided in the first array of pixel according to the first color filter array is used to increase the quality of the upscaled image comprising the third pixel array.

The method has the advantage that a core standard demosaicing algorithm can be used, for example a standard Bayer demosaicing process. It is not required to have an algorithm at hand, which demosaices the higher-order pattern as is. No frame buffer is required, as the method is able to operate on a set of a few frame lines at a time. No specific algorithm for native color filter array patterns is required. Well- known building block algorithms can be used as core demosaicing process, wherein the method is still enabled for higher-order color filter pattern demosaicing.

The method can further include the step of filtering of the pixel data in the demosaiced second pixel array after step b) to reduce aliasing artefacts, which can occur throughout the further method due to binning e.g. in critically sampled regions. This filter is optional.

The step a) of binning the blocks of adjacent pixel assigned to the same color can be performed by calculating an average value of the pixels of one color block in the first pixel array. The average value can be for example calculated as mean value or median value of the pixel values in the block of directly adjacent pixels. For example, the pixel values of a block of pixels of the same color, e.g. 2x2, 3x3, 4x4, 5x5 etc. block can be averaged to achieve a resulting pixel value for a second pixel array of reduced size and resolution of the image.

The blocks of adjacent pixels assigned to the same color of the same pixel array in step a) can be applied to QuadBayer 4x4 or HexaDeca 8x8 color filter array to bin such first pixel arrays down to a second pixel array according to a Bayer 2x2 color filter array.

The step c) of upscaling the demosaiced second pixel array can be processed by nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos resampling or spline interpolation or a combination thereof. These are preferred methods for upscaling pixel arrays besides other methods.

The step a) of binning of the blocks of adjacent pixel assigned to the same color can be repeated before performing the step b) of demosaicing the second pixel array achieved after two subsequent steps of first binning the blocks of the first pixel array down to an intermediate second pixel array, and second binning of the blocks of this intermediate second pixel array down to the final second pixel array. By repeating the step a) of binning at least two times, a higher-order pixel array according to a higher- order color filter array can be binned down to a low-order pixel array according to a low-color filter array, e.g. to bin a HexaDeca 8x8 color filter array down to a Bayer 2x2 color filter array.

The step c) of upscaling the demosaiced second pixel array can be repeated in two subsequent steps of:

- First upscaling the demosaiced final pixel array to an intermediate third pixel array having a size and color distribution pattern according to the intermediate second pixel array, and

- Second upscaling the intermediate third pixel array to the size and color distribution pattern of the first pixel array to achieve the upscaled third pixel array.

Thus, also in the step of upscaling a low-order pixel array according to a low-order color filter array, e.g. the Bayer 2x2 color filter array, can be upscaled back to a higher-order pixel array according to a higher-order color filter array, e.g. HexaDeca 8x8 color filter array.

After each step of upscaling, the resulting pixel array can be reconstructed by use of the raw image data for the final third pixel array and by use of the intermediate second pixel array achieved in the first binning step for the intermediate third pixel array. Thus, original image information arising from the raw image data are re-used to improve the image quality of the upscaled intermediate second pixel array to achieve an improved image comprising the final third pixel array.

The upscaled intermediate third pixel array achieved after step c) can be reconstructed by replacing the pixels in the upscaled third pixel array with the available pixels from the related image data, i.e. the raw image data or intermediate second pixel array, having the same pixel location while maintaining color channel differences being present in the upscaled third pixel array.

The reconstruction of the upscaled (intermediate) third pixel array can be performed by local edge detection of the blocks surrounding the third pixel array based on the related image data, i.e. the raw image data or the intermediate second pixel array, and interpolating the block in the respective color channel by use of the detected edge direction, and the color channel differences being present in the upscaled third pixel array.

The reconstruction steps according to claims 11 and 12 can each comprise the step of determining block error metrics indicating the fit of the replaced lower order blocks into their neighborhood and merging the results of the reconstructed upscaled third pixel array achieved by the reconstruction according to claims 11 and 12 into a final reconstructed pixel array selecting the pixels or blocks based on the related block error metrics.

This allows to process at least two specific methods for replacing pixels and to estimate, which of the reconstruction strategies are most applicable for a respective block. The optional step of filtering the pixel array can be achieved after the respective step of reconstruction and/or by filtering the final reconstructed pixel array.

The image processor unit can be designed as an image processing hardware, e.g. ASIC, FPGA or suitable programmed microprocessor, said hardware being arranged to process the image data by performing the above described method steps.

The methods can be implemented into a computer program comprising instructions which, when the program is executed by a processing unit, causes the processing unit to process raw image data provided by an image sensor with the steps of the method according to the present invention.

The invention is explained in more detail by way of an exemplary embodiment with the enclosed figures. It shows:

Figure 1 - schematic block diagram of an image processor unit;

Figure 2 - examples of pixel arrays according to a specific color filter array;

Figure 3 - flow diagram of the method for processing raw image data for demosaicing image data to achieve a final image;

Figure 4 - flow diagram of a method of two subsequent steps of binning of blocks and reconstructing the respectively upscaled pixel array;

Figure 5 - exemplary pixel arrays for performing the step of binning blocks of adjacent pixel of a higher-order color filter array to a lower-order color filter array;

Figure 6 - flow diagram of a method for reconstructing the upscaled pixel array;

Figure 7 - exemplary hardware logic for reconstructing an upscaled flow of pixels of a pixel array;

Figure 8 - QuadBayer and HexaDeca blocks for estimating from surrounding pixels.

Figure 1 illustrates a schematic block diagram of an image camera 1 comprising an optoelectronic sensor 2 comprising a first pixel array, which is overlaid with the color filter array 3. Such color filter arrays 3 on top of the optoelectronic sensor 2 optically masks out certain wave lengths to get color information to a certain pixel. The color filter array 3 is a grid of optical filters for different wave lengths. By use of such color filter array 3, the monochromatic pixels of the optoelectronic sensor 2 are assigned to a specific color. The pixel value of a respective pixel of the optoelectronic sensor 2 indicate the brightness of the color of the image at this respective pixel location.

The raw image data IMGRAW of the optoelectronic sensor 2 is provided to an image processor unit 4, which is arranged for demosaicing the raw image data IMGRAW to achieve a final image IMGFIN in a specific way as it is explained in more detail in the following.

Figure 2 presents different pixel arrays assigned to different color filter arrays.

At a), the well-known low-order Bayer pixel array assigned to the Bayer color filter array is shown. A number of 2x2 blocks including red-(R), green-(G) and blue-(B) pixels are repeated over the 8x8 pixel array. A variety of well-proven algorithms for demosaicing such a Bayer pixel array is available in the prior art and can be used for the present method for processing the raw image data.

At b), a 6x6 CFA pattern is shown, which forms a higher-order color filter array pattern compared to the Bayer color filter array pattern. A 6x6 block contains four 3x3 blocks of a respective color red (R), green (G) and blue (B). The 3x3 blocks of a specific color RGB are arranged according to the color arrangement of a 2x2 block in the Bayer color filter array pattern (R-G-G-B).

At c), a QuadBayer color filter array pattern is shown. The higher-order pixel array contains pixels in a 8x8 matrix, wherein a 4x4 block is repeated, when each 4x4 block is formed by four 2x2 blocks of a respective color R, G and B. The 2x2 blocks of a respective color RGB are arranged in the way of the lower-order Bayer pattern R-G-G-B.

At d), a HexaDeca color filter array pattern is presented, which is formed by a 8x8 matrix. Here, four blocks of 4x4 array size are assigned to a respective color R, G, B. The four blocks of the respective color are arranged in the order like the low-order Bayer color filter array pattern shown at a).

Figure 3 shows a flow diagram of the method or processing raw image data IMGRAW provided by an image sensor, when said raw image data IMGRAW comprises a first array of pixel, where each pixel represents color information according to a first specific color pattern defined by a specific first color filter array.

The raw image data IMGRAW can be for example a first pixel array of a higher-order color filter array as shown in figure 2 b), c) and d), i.e. a 6x6 CFA pixel array, a QuadBayer pixel array or a HexaDeca pixel array.

The raw image data IMGRAW can also be a first pixel array assigned to another color filter array of a more complex structure, like Sony® RGBW, RYYB, MYYC, Fuji® X- Trans® and the like.

The raw image data IMGRAW are processed in the first step a) by binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array.

For example, the 2x2, 3x3 or 4x4 blocks of the same color of the higher-order pixel arrays in figure 2 b), c) or d) can be binned down to one pixel value of the respective color R, G or B of the low-order pixel array, e.g. the Bayer pixel array of figure 2 a).

The set of pixel values of the block of pixels assigned to the same color can be averaged to receive a resulting pixel value, for example one related pixel of the lower-order pixel array.

In the second step b) demosaicing of the second pixel array, which is arranged according to a second color filter array, which is related to the color distribution pattern of the second pixel array is performed. This allows the use of a standard process for demosaicing a lower-order pixel array comprising color information according to the lower-order specific color pattern defined by the respective second color filter array. For example, the step of demosaicing can be taken place on a second pixel array arranged according to the lower-order Bayer color filter array pattern.

An optional step (e) of filtering the pixel data achieved by the step of b) of demosaicing can be performed to reduce aliasing artefacts.

The demosaiced and optionally filtered second pixel array is then upscaled in step c) to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array PAIMD. Thus, the second pixel array received after downsizing the raw image data by the step a) of binning is rescaled again to the size and color distribution of the pixels as specified in the first array of pixel in the related first specific color pattern of the first color filter array.

In a final step d) the upscaled third pixel array is reconstructed by use of the raw image data. Thus, the original pixel information of the raw image data IMGRAW are used to improve the image received after the step c) of upscaling the demosaiced second pixel array.

The result is a final demosaiced image comprising a resolution according to the first array of pixel of the original raw image data.

Figure 4 shows a block diagram for the method for processing raw image data IMGRAW according to figure 3, but with repeated steps of a1 ) and a2) of binning the raw image data IMGRAW and an intermediate second pixel array IMGIMD received after the first step a1 ) of binning the blocks of adjacent pixel assigned to the same color of the first pixel array down to achieve an intermediate second pixel array of reduced size an resolution.

The second subsequent step a2) of binning the intermediate second pixel array IMGIMD merges pixel of the same color arranged adjacent to each other in a same sub-block to receive a second pixel array of a standard color distribution. The following step b) of demosaicing this second pixel array is adapted to the specific standard pixel array having the color distribution of the respective standard second color filter array. Thus, by performing the two subsequent steps a1 ) and a2) of binning the original raw image data IMGRAW down to image data according to the second pixel array, the color distribution pattern is adapted to the standard second color filter array. This allows the standard process of demosaicing the second pixel array according to the second color distribution pattern on the raw image data IMGRAW having a different color distribution pattern.

After the optional step (e) of filtering the upscaled second pixel array, a first step of c1) of upscaling the demosaiced second pixel array to an intermediate pixel array PAIMDI according to a size and resolution of the intermediate color pixel distribution pattern is performed. The resulting intermediate third pixel array is reconstructed in step d1) by use of the intermediate image data IMGIMD-I , i.e. the intermediate second pixel array IMGIMD achieved after the first step a1 ) of binning the original raw image data IMGRAW.

The resulting intermediate third pixel array is then upscaled in step c2) to an intermediate pixel array PAIMD2 according to a size and color distribution pattern, which corresponds to the original first pixel array comprising a pixel distribution pattern according to the first color filter array.

The upscaled pixel array PAIMD2 is then reconstructed in step d2) by use of the raw image data IMGRAW to receive the final image IMGFIN comprising the third pixel array, which corresponds to the demosaiced raw image data according to the first pixel array.

Thus, the third pixel array received by the method is (almost) similar to directly demosaicing the raw image data comprising a first pixel array according to the specific first color pattern.

Figure 5 exemplarily illustrates two subsequent steps of binning of blocks of a first pixel array according to the HexaDeca 8x8 RGB pattern into an intermediate second pixel array according to the QuadBayer 4x4 pattern and the second pixel array according to the Bayer 2x2 pattern. The first step a1 ) of binning the HexaDeca 8x8 pixel array reduces the array of the size of m x n (8x8) to half of the size m/2 x n/2 (i.e. 4x4). The second step a2) of binning the QuadBayer 4x4 pixel array into the final second pixel array according to the Bayer 2x2 pattern reduces the size to m/4 x n/4. Each binning step reduces the input pixel array size by the half in this example.

The binning steps are performed by merging a 2x2 pixel block of the same color into one pixel value. The binning procedure can start for example with the first and second upper line at the left side with the four pixel values assigned to the red color R, i.e. the first 2x2 block. The pixel values are merged e.g. by estimating the mean value of the four pixels or the median value to achieve one pixel value for the red color at the upper left pixel location of the QuadBayer 4x4 intermediate pixel array. The procedure is repeated for the following 2x2 block assigned to the red color, then for the green color G and finally for the blue color B.

In the same way, the second step a2) of binning merges a 2x2 block of pixels assigned to the same color R, G or B to one pixel value assigned to the same color R, G or B to achieve the second pixel array according to the standard Bayer 2x2 pattern.

The standard process of demosaicing in step b) is then performed on this reduced standard second pixel array.

Figure 6 shows a block diagram of the blockwise processing of step d) of reconstructing the upscaled third pixel array after the demosaicing process is performed and the resulting pixel array is upscaled to the higher-level color filter array pattern.

The upscaled pixel array PAIMD is forwarded to a processing block R1_a) for pixel replacement as input together with the raw image data IMGRAW for the respective pixel array. Here, pixel of the upscaled third pixel array, i.e. the intermediate pixel array PAIMD are replaced by respective pixel values of the raw image data IMGRAW while maintaining color differences estimated in the preceding demosaicing process. Thus, the detail reconstruction procedure uses the raw image data IMGRAW according to the first specific color pattern to reconstruct details in the upscaled image PAIMD. However, the raw image data only contains blockwise information for only one color at the same time (e.g. 2x2 R pixels for QuadBayer or 4x4 R pixels or HexaDeca).

The processing is performed blockwise, e.g. 2x2 QuadBayer or 4x4 HexaDeca block per sequence. There are two paths which are merged in step R_d) for the final output, wherein the left path is a replacement strategy using the original raw image data IMGRAW and the right path is an interpolation strategy taking the color filter array surrounding into account.

In step R1_a) of the right path, the pixels in the upscaled image PAIMD are replaced with the available pixels from the raw image data of the original color filter array while maintaining color channel differences, which are present in the upscaled image. This is explained in more detail with regard to figure 7 later on.

After an optional filtering step R1_b), block error metrics are computed in step R1_c) to see, whether the replaced blocks (e.g. 2x2 QuadBayer or 4x4 HexaDeca) fit well into their neighborhood.

The left path does in step R2_a) a local edge detection of the block surrounding based on the original raw image date IMGRAW according to the first color filter array and an interpolation of the block in the respective color channel in step R2_b) taking the detected edge direction into account. Also here, the color channel differences from the upscaled image are used to restore full color information, e.g. RGB information. In this path, also a further optional step R2_c) of filtering is possible before block error metrics are computed in step R2_d).

The final decision, whether to take either the pixel values from the right or the left path is taken based on the previously computed error metrics. The merging of the two paths in step R_d) can be either a hard decision or a soft blending-like approach.

A final post-processing filter in step R_e) on the color pixels can be optionally used to enhance the image quality.

The result is the final image IMGFIN. The pixel replacement in step R1_a) can be performed for example with the following equations:

For a selected Green pixel position within the respective color filter array: dc-R ⁼ GDem- RDem dG-B ⁼ GDem - BDem Greplace ⁼ GRAW

Rreplace = GRAW - dG-R

Breplace = GRAW - dG-B

For a selected Red pixel position within the respective color filter array: dR-B ⁼ RDem - GDem dR-B ⁼ RDem - BDem

Rreplace ⁼ RRAW

Greplace = RRAW - dR-G

Breplace = RRAW - dR-B

For a selected Blue pixel position within the respective color filter array: dB-G ⁼ BDem- GDem dB-R ⁼ BDem - RDem

Breplace ⁼ BRAW

Rreplace = BRAW - dB-R

Greplace = BRAW - dB-G

For the green pixels G position, the green pixel is replaced by the original green raw pixel data GRAW, while the demosaiced red and blue pixels R, B for said original green pixel position are replaced by the original green pixel GRAW subtracted by the respective difference dc-R or dc-B.

The difference for a green, red or blue pixel is calculated by the difference of the respective demosaiced green, red or blue pixel Goem, Roem, Boem as basic pixel and the related demosaiced red or blue pixel demosaiced at the basic green pixel position, which is present in the demosaiced image PAJMD. In case that a basic red pixel is present in the raw image data, the difference is calculated based on the demosaiced red pixel Roem and the demosaiced green and blue pixel for this basic red pixel. Similar, for a blue pixel BRAw the difference is calculated by the demosaiced blue pixel Boem and the related demosaiced red or green pixel Roem and Goem.

Thus, when achieving a 1x1 pixel block of a specific color RGB from a higher-order pixel block of the same color, e.g. 4x4 pixel array, a color vector Goem, Roem and Boem is achieved for this 1x1 pixel block by the step of demosaicing. This color vector is combined with the original pixel array of the raw image data assigned to one specific color, e.g. green color G. Upscaling the demosaiced 1x1 pixel block results in a higher-order pixel array, e.g. 4x4 pixel array, which is then combined with the original pixel array of this one color. The color of the original pixel array specifies the basic color and thus the formula specified above for green, red or a blue color used for the raw image data GRAW, RRAW or BRAW.

Figure 7 shows a block diagram of a functional circuit for performing the pixel replacement in step R1_a) of figure 6.

The raw image data for a block in form of CFA data RCFA, GCFA and BCFA for the three colors RGB are provided to an adding logic ADD, subtracting logic SUB and multiplexers MUX1 , MUX2 and MUX3. Further, the upscaled RGB image of a block in this three channels Rupscaie, Gupscaie and Bupscaie are input to subtracting logics SUB in order to calculate the differences do-R, do-B, dB-G, dB-R, dR-G, dR-B by use of the equations as stated above.

Figure 7 shows the example of the block diagram for the blue block B where the differences dc-B and dc-R are calculated (GBupscaie and GRupscaie). The multiplexers MUX1 , MUX2 and MUX3 are together controlled independent on the block type to provide either the raw image data RCFA, GCFA or BCFA for the red block type, green block type or blue block type at MUX1 , MUX2 or MUX3. Further, the corrected pixel values for the other colors are provided at the output of the multiplexers MUX1 , MUX2 or MUX3, on which the original raw data RCFA, GCFA or BCFA are not selected. Because the pixel values might leave the allowed value range by this processing, clipping to zero and to a maximal pixel value is required. This is controlled by the CLIP logic at the output of the respective multiplexers MLIX1 , MLIX2 and MLIX3.

Figure 8 relates to step R2_b) of “Interpolation” in Figure 6 for the exemplary case, that the local edge detection detected no edge. In this regard, it presents an example for estimating the block surrounding at the left path of figure 6 for the blue block as example. The green pixels Gi can be estimated by the average of the four adjacent pixel values in the surrounding of the green pixel Gi.

This can be represented by the following equations:

Go ⁼ (to + ti + Io + h)/4

Gi ⁼ (to + ti + ro + n)/4

G2 ⁼ (Io + I1+ bo + bi)/4

G3 ⁼ (bo + bi+ ro + n)/4

For the blue pixels, the same scheme applies:

Bo ⁼ (to + ti + Io + h)/4

Bi = (to + ti + ro + ri)/4

B2 ⁼ (Io + I1+ bo + bi)/4

B3 = (bo + bi+ ro + ri )/4

This applies for QuadBayer 2x2 pixel array of one color.

The red pixel values ro.i ,2,3 can be calculated by use of the pixel replacement method R1_a) in figure 6 and 7 at the right path by the equations:

Ro ⁼ Go - (GRupscale) Ri ⁼ Gi - (GRupscale) etc.

For other higher order pixel array blocks, e.g. HexaDeca the pixel values at the outer corners can be calculated as described above, e.g. Go — (to + ti + Io + h)/4 etc.

The inner pixel values are calculated based on the outer surrounding pixels, e.g.

Ge ⁼ (t2 + ts + ro + n)/4

Gio ⁼ (r2 + r3 + b2 + bs)/4.

The outer middle pixel values can be estimated as average of the adjacent three surrounding pixels, e.g.

G14 = (bi + b2 + bs)/3

G₈ = (h + l₂ + l₃)/3.

The same applies for the blue pixels.

The error metric for the left scheme in figure 6 can be calculated for an exemplary QuadBayer 2x2 block of a specific color by the formula: 11 1 + | G2 - b0 | + | G3 - r1 1 +

An error value can be calculated by

In case that this error value £ is greater than a predefined threshold value for the specific color, e.g. to for green pixel G or tB for blue pixel B, the block is replaced by this reconstruction scheme.

Claims

1 . Method for processing raw image data (IMGRAW) provided by an image sensor, wherein the image data comprising a first array of pixel, said first pixel array being overlaid with a color filter array (3), so that the pixel represent color information according to a first specific color pattern defined by the color filter array (3), said first specific color filter pattern comprising blocks of adjacent pixel assigned to the same color characterized by: a) Binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array; b) Demosaicing of the second pixel array according to a second color filter array (3) which is related to the color distribution pattern of the second pixel array; and c) Upscaling the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array (PAIMD).

2. Method according to claim 1 , characterized by the step of: d) Reconstructing the upscaled third pixel array (PAIMD) by use of the raw image data (IMGRAW).

3. Method according to claim 1 or 2, characterized by the step of: e) Filtering of the pixel data in the demosaiced second pixel array after step b) to reduce aliasing artefacts. Method according to one of claims 1 to 3, characterized in, that the step a) of binning of the blocks of adjacent pixel assigned to the same color is performed by calculating an average value of the pixels of one color block in the first pixel array. Method according to claim 4, wherein the average value is calculated as mean value or median value of the pixel values in a block of directly adjacent pixels. Method according to one of the preceding claims characterized by binning the blocks of adjacent pixels assigned to the same color of the first pixel array in step a) according to QuadBayer 4x4 or HexaDeca 8x8 color filter array pattern (3) down to a second pixel array according to a Bayer 2x2 color filter array pattern (3). Method according to one of the preceding claims wherein the step c) of upscaling the demosaiced second pixel array is processed by nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos resampling or spline interpolation. Method according to one of the preceding claims, characterized by repeating the step a) of binning of the blocks of adjacent pixel assigned to the same color before performing the step b) of demosaicing the second pixel array achieved after two subsequent steps of first binning the blocks of the first pixel array down to an intermediate second pixel array (IMGIMD), and second binning of the blocks of the intermediate second pixel array (IMGIMD) down to the final second pixel array. Method according to claim 8, characterized by repeating the step c) of upscaling the demosaiced second pixel array in two subsequent steps of:

- First upscaling the demosaiced final second pixel array to an intermediate third pixel array having a size and color distribution pattern according to the intermediate second pixel array (IMGIMD), and

- Second upscaling the intermediate third pixel array to the size and color distribution pattern of the first pixel array to achieve the upscaled third pixel array (PAIMD). Method according to claim 9 wherein after each step of upscaling the resulting pixel array is reconstructed by use of the raw image data (IMGRAW) for the final third pixel array and by use of the intermediate second pixel array (IMGIMD) achieved in the first binning step for the intermediate third pixel array. Method according to one of the preceding claims, characterized by reconstructing the upscaled intermediate third pixel array by:

- Replacing the pixels in the upscaled third pixel array (PAIMD) with the available pixels from the related image data, i.e. the raw image data (IMGRAW) or intermediate second pixel array (IMGIMD), having the same pixel location while maintaining color channel differences being present in the upscaled third pixel array (PAIMD). Method according to one of the preceding claims, characterized by reconstructing the upscaled (intermediate) third pixel array by local edge detection of the block surrounding a respective pixel or block of pixels based on the related image data, i.e. the raw image data (IMGRAW) or the intermediate second pixel array (IMGIMD), and interpolating the block in the respective color channel by use of the detected edge direction, and the color channel differences being present in the upscaled third pixel array (PAIMD). Method according to the combination of claims 11 and 12, wherein the reconstruction steps according to claims 11 and 12 each comprises the step of determining block error metrics indicating the fit of the replaced lower order blocks into their neighborhood and merging the results of the reconstructed upscaled third pixel array (PAIMD) achieved by the reconstruction according to claims 11 and 12 into a final reconstructed pixel array selecting the pixels or blocks based on the related block error metrics. Method according to one of claims 11 to 13, characterized by filtering the pixel array achieved after the respective step of reconstruction and/or by filtering the final reconstructed pixel array. Image processor unit (4) for processing raw image data (IMGRAW) provided by an image sensor, said image sensor comprises a sensor array of a first pixel array providing an image comprising a first array of pixel data, said first pixel array being overlaid with a first color filter array (3) so that the pixel represent color information according to a first specific color pattern defined by the color filter array (3), said first specific color pattern comprising blocks of adjacent pixel assigned to the same color, characterized in that the image processor unit (4) is arranged to: a) Binning of the blocks of adjacent pixel assigned to the same color down to achieve a second pixel array of reduced size and resolution of the image compared to the image of the first pixel array; b) Demosaicing of the second pixel array according to a second color filter array (3) which is related to the color distribution pattern of the second pixel array; and c) Upscaling the demosaiced second pixel array to the size of the color distribution pattern of the first pixel array to achieve an upscaled third pixel array (PAIMD). Image processor unit according to claim 15, characterized in that the image processor unit (4) is arranged for processing image data by performing the method steps of one of claims 1 to 14. Computer program comprising instructions, which, when the program is executed by a processing unit, causes the processing unit to carry out the steps of the method of one of claims 1 to 14.