WO2020007990A1 - Compression of a raw image - Google Patents

Compression of a raw image Download PDF

Info

Publication number
WO2020007990A1
WO2020007990A1 PCT/EP2019/068013 EP2019068013W WO2020007990A1 WO 2020007990 A1 WO2020007990 A1 WO 2020007990A1 EP 2019068013 W EP2019068013 W EP 2019068013W WO 2020007990 A1 WO2020007990 A1 WO 2020007990A1
Authority
WO
WIPO (PCT)
Prior art keywords
color
image
raw image
pixel cell
cell block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2019/068013
Other languages
French (fr)
Inventor
Siegfried FÖSSEL
Thomas Richter
Joachim Keinert
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Original Assignee
Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV filed Critical Fraunhofer Gesellschaft zur Foerderung der Angewandten Forschung eV
Publication of WO2020007990A1 publication Critical patent/WO2020007990A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/64—Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/64—Systems for the transmission or the storage of the colour picture signal; Details therefor, e.g. coding or decoding means therefor
    • H04N1/646—Transmitting or storing colour television type signals, e.g. PAL, Lab; Their conversion into additive or subtractive colour signals or vice versa therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component

Definitions

  • the present invention relates to concepts for efficient compression of raw images such as raw images of a Bayer pattern.
  • Image capturing devices as cameras are often based on single or multiple sensor chips.
  • the sensors of the cameras based on a single sensor chip may have multiple photosensitive pixel cells.
  • Said pixel cells may each be provided with color filters having different or similar colors on top of each of the photosensitive pixel cells.
  • the filters enable each pixel cell of the image sensor to capture multiple light spectra or color components for each pixel as a basic image component.
  • One of the frequently used color filter pattern is a so called "Bayer" pattern, wherein a color filter array is provided for arranging red, green and blue (RGB ) color filters on a square grid of photosensitive pixel cells.
  • the Bayer pattern is provided with a set of red, green and blue color filters on top of a four pixel cell block each to create one red, one blue and two green sensitive pixels.
  • the captured values from these pixel cells are called raw data.
  • each pixel Since each pixel is filtered to record one of three colors, the data from each pixel cannot fully specify each of the red, green, and blue values on its own. However for displaying a color image at least 3 color values for each pixel cell position are necessary. To obtain a full color image, an algorithm can be used to interpolate a set of red, green, and blue values for each pixel. These algorithms make use of the surrounding pixels of the corresponding colors to estimate the values for a particular pixel.
  • the raw data can be processed to three color components values (typically RGB values) for each pixel cell position.
  • RGB values typically RGB values
  • This process is known e.g. as a “demosaicing” process. This can be done by linear interpolation of same colored pixel cells or a more sophisticated method. In this process the raw data will disadvantageously be expanded to an RGB image and the amount of data is increased by a factor of 3.
  • An embodiment may have a raw image encoder for encoding a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, the raw image encoder configured to subject the raw image to a color transformation to obtain a color image, by mapping, for each pixel cell block, the four pixels of the pixel cell block onto a color component quadruple forming a sample of the color image, the color component quadruple comprising three color component values of a target color space, and one pseudo color component value, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.
  • Another embodiment may have a decoder for reconstructing from a compressed data stream a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, configured to decode from a compressed data stream by multi-component picture decoding a color i
  • Another embodiment may have a raw image encoder for encoding a Bayer pattern raw image, configured to map each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G 2 (green), and B (blue), onto a color component quadruple ⁇ Cr, Cb, Y, DY) according to
  • Another embodiment may have a decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple ⁇ Cr, Cb, Y, DY ⁇ according to
  • Another embodiment may have a Decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple ⁇ Cr, cb, Y, DY ⁇ according to
  • a method for encoding a Bayer pattern raw image may have the steps of: mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G 2 (green), and B (blue), onto a color component quadruple ⁇ Cr, Cb, Y, DY ⁇ according to
  • a method for reconstructing from a compressed data stream a Bayer pattern raw image may have the steps of: decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple ⁇ Cr, Cb, Y, DY ⁇ according to
  • Another embodiment may have a data stream having encoded thereonto a color image generated using a method for encoding a Bayer pattern raw image, the method having the steps of: mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G 2 (green), and B (blue), onto a color component quadruple ⁇ Cr, Cb, Y, DY ⁇ according to
  • Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method for reconstructing from a compressed data stream a Bayer pattern raw image, the method having the steps of: decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple (Cr, Cb, Y, DY ⁇ according to onto pixels R (red), Gi (green), Gz (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, when said computer program is run by a computer, wherein b may have the value of Ma.
  • An advantage of concepts of the present application stems from an efficient way of handling the pixel cell blocks of raw images. They are transformed to a color component quadruple forming of a sample of a color image, the quadruple
  • the color transformation fulfills two tasks concurrently: a decorrelation of the information of the raw data values is achieved - at least for homogenous areas which is the most part of images, thereby reducing the entropy and allowing better compression efficiency.
  • the overall compression results in a lossless transformation of data into a data stream from which either the original raw image may be derived, or an image corresponding to the raw image which shows the content in the target color space, but is of half width and height compared to the raw image, i.e. has half resolution.
  • FIG. 1 shows schematically a pixel cell block of an image sensor subjected to a demosaicing step
  • FIG. 2 illustrates schematically a first and a second apparatus configured to perform color transforming a color image, according to a first and a second embodiment of the present invention
  • FIG. 3 illustrates schematically a raw image encoder, according to a specific embodiment of the present invention.
  • FIG. 4 illustrates schematically a decoder, according to a specific embodiment of the present invention
  • Fig. 1 shows schematically a pixel cell block of a commonly used image sensor equipped with a color filter pattern, the so called Bayer pattern.
  • This pattern is a set of (one) red, (two) green and (one) blue color filters (RGB filters) on top of a pixel cell block formed by four pixels, each to create one red, one blue and two green sensitive pixels.
  • the captured values from these pixel cells are the so called raw data.
  • RGB color values
  • the raw data may be processed - this process is called demosaicing - to three color components values (typically RGB or YCt > C r values) for each pixel cell position. This can be done by linear interpolation.
  • the first step in compressing an RGB color image is typically a color transformation, as the information in the color components are strongly correlated and this redundancy should be minimized.
  • the result is typically a luminance component and two color difference components YUV or YCbC r .
  • the decorrelated components are now one Red-Chrominance Cr, one Blue- Chrominance Cb, one Luminance Y and one pseudo Delta-Luminance DY.
  • the first three components C r , C b , Y can be used directly to transform them into an RGB image with half width and half height by using G as mean value of Gi and G 2 . This is equivalent to consider one Bayer pixel cell block as one RGB macro pixel.
  • the three components C r , C b and Y can be considered as color image with a residual DY.
  • the raw image can be reconstructed.
  • the residual DY and the half resolution image together may be used to reconstruct the Bayer raw data to full resolution image.
  • FIG. 2 illustrates schematically a color transformation and its retransformation which underlies embodiments for a raw image encoder and its pendant, the decoder,
  • the forward color transform 5 is for color transforming of a raw image 10 into a color image 18.
  • the color image 10 is formed by a number of X lines by Y columns of pixel cell blocks 12, wherein each cell blocks 12 comprises four pixels 14, arranged as a
  • the raw image 10 has N times M pixels clustered into N/2
  • pixel cell blocks 12 M/2 superpixels, called pixel cell blocks 12.
  • a color decorrelation is realized by the color transformation 5.
  • image 18 and such as RGB images for example, the neighbored four color
  • the color transform 5 is designed to deal with a raw image 10 of such a color pattern.
  • each pixel cell block 12 comprising four pixels 14, each pixel 14 being associated with one of three base colors so that each pixel cell block 12 comprises, for each of the three base colors, at
  • the four pixels 14 of the pixel cell block 12 are mapped onto a color component quadruple 16 forming a sample of the color image 18.
  • the mapping 5 may be done according to (1) : each pixel cell block 12 of a Bayer pattern raw image 10 which comprises four pixels R red, Gi green, G 2 green, and B blue, is mapped onto a color component quadruple ⁇ C r , C b , Y, DY ⁇ 16 according to (1), with the quadruple forms a sample of the color image 18.
  • the color component quadruple 16 comprises three color component values 20 of a target color spaces and one pseudo color component value 22.
  • the decorrelated components color component quadruple 16 may be referred as e.g. color component values 20 being one Red-Chrominance C r , one Blue-Chrominance C b , one
  • Luminance Y as well as the pseudo color component value 22 as a pseudo Delta- Luminance DY, That is, the afore-mentioned target color space may be elected to be a YC r C b color space.
  • the first three component values (20) C r , C b , Y can be used e.g. to be directly transformed into an RGB image with half width and half height. This is equivalent to consider one pixel cell block 12 as one RGB macro pixel. However, alternatives using other color spaces are feasible as well
  • the four color components Cr, Cb, Y and DY may be transformed back lossless to the color components R, G-i, G 2 and B. This is done by the color retransformation 25.
  • Function (1) and (2) may, thus, be based on a linear mapping.
  • each of the three color component values 20 may result from a weighted sum of the four pixels 14 of the pixel cell block 12.
  • the weights of the weighted sum of the two pixels of the pixel cell block may be associated accordingly with the same base color are equal to each other. See, for instance, the middle columns in the first three lines of the matrix of (1).
  • the linear mapping 5 may be such that the one pseudo color component value 22 results from a further weighted sum of the two pixels of the pixel cell block associated with the same base color, wherein only the weights of the weighted sum of the two pixels being of opposite sign, See the lower row in (1), for instance.
  • FIG. 2 illustrates schematically also the color retransforming procedure 25 from color image 18 to raw image 10.
  • the transform 25 maps, for each sample of the color image 18, the color component quadruple 16 representing this sample, onto the four pixels 14 of the corresponding pixel cell block 12, That is, image 18 comprises as many samples as raw image 10 has pixel cell blocks 12.
  • the color component quadruple 16 comprises three color component values 20 of a predetermined color space and a pseudo color component value 22.
  • the retransform 25 maps each color component quadruple ⁇ C r , C b , Y, DY ⁇ 16 according to (2) onto pixels R red, Gi green, G2 green, and B blue of a corresponding pixel cell block 12 of the Bayer pattern raw image 10,
  • the mapping may be a linear mapping where the two pixels of the pixel cell block 12 being associated with the same base color respectively, i.e. G1 and G2, result from a weighted sum of the color component quadruple 16 with weights of the weighted sum for the two pixels being equal. See the first three columns in the middle two lines of the matrix in (2), for instance.
  • the weights of the weighted sum of the one pseudo color component value may be of opposite sign. See the rightmost column in the middle two lines of the matrix in (2), for instance.
  • the predetermined color space may be selected to be a YC r C b color space or differently.
  • FIG. 3 illustrates schematically an encoding device 28 comprising the raw image encoder 5 and a multi-component picture encoder 30 configured to encode a color image 18 output by the apparatus into a compressed multi-component data stream 32.
  • the multi-component picture encoding performed may comprise lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.
  • FIG. 4 illustrates schematically a raw image decoding device 40 configured for reconstructing a raw image 10 from a multi-component data stream 32 comprising a decoder 25 and a multi-component picture decoder 42 configured to decode a color image 18 from the multi-component data stream 32.
  • the multi-component picture decoding may comprise spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Color Television Systems (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)

Abstract

The invention relates to a raw image encoder for encoding a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, the raw image encoder configured to subject the raw image to a color transformation to obtain a color image, by mapping, for each pixel cell block, the four pixels of the pixel cell block onto a color component quadruple forming a sample of the color image, the color component quadruple comprising three color component values of a target color space, and one pseudo color component value, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.

Description

COMPRESSION OF A RAW IMAGE
FIELD OF THE INVENTION
The present invention relates to concepts for efficient compression of raw images such as raw images of a Bayer pattern.
BACKGROUND
Image capturing devices as cameras are often based on single or multiple sensor chips. The sensors of the cameras based on a single sensor chip may have multiple photosensitive pixel cells. Said pixel cells may each be provided with color filters having different or similar colors on top of each of the photosensitive pixel cells. The filters enable each pixel cell of the image sensor to capture multiple light spectra or color components for each pixel as a basic image component.
One of the frequently used color filter pattern is a so called "Bayer" pattern, wherein a color filter array is provided for arranging red, green and blue (RGB ) color filters on a square grid of photosensitive pixel cells. The Bayer pattern is provided with a set of red, green and blue color filters on top of a four pixel cell block each to create one red, one blue and two green sensitive pixels. The captured values from these pixel cells are called raw data.
Since each pixel is filtered to record one of three colors, the data from each pixel cannot fully specify each of the red, green, and blue values on its own. However for displaying a color image at least 3 color values for each pixel cell position are necessary. To obtain a full color image, an algorithm can be used to interpolate a set of red, green, and blue values for each pixel. These algorithms make use of the surrounding pixels of the corresponding colors to estimate the values for a particular pixel.
For this reason the raw data can be processed to three color components values (typically RGB values) for each pixel cell position. This process is known e.g. as a “demosaicing” process. This can be done by linear interpolation of same colored pixel cells or a more sophisticated method. In this process the raw data will disadvantageously be expanded to an RGB image and the amount of data is increased by a factor of 3.
It is a purpose of the present disclosure to overcome the drawbacks associated with raw images and to allow for an easier handling thereof in order to, for instance, allow for an easier encoding of raw data images.
SUMMARY
An embodiment may have a raw image encoder for encoding a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, the raw image encoder configured to subject the raw image to a color transformation to obtain a color image, by mapping, for each pixel cell block, the four pixels of the pixel cell block onto a color component quadruple forming a sample of the color image, the color component quadruple comprising three color component values of a target color space, and one pseudo color component value, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.
Another embodiment may have a decoder for reconstructing from a compressed data stream a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, configured to decode from a compressed data stream by multi-component picture decoding a color i
image, samples of which are respectively formed by a color component quadruple, and subject the color image to a color transformation to obtain the raw image, by mapping, for each sample of the color image, the color component quadruple to obtain the four pixels of a corresponding pixel cell block, wherein the color component quadruple comprises three color component values of a predetermined color space and pseudo color component value. Another embodiment may have a raw image encoder for encoding a Bayer pattern raw image, configured to map each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY) according to
Figure imgf000005_0001
which forms a sample of the color image, and subject the color image to a multi- component picture encoding to obtain a compressed data stream.
Another embodiment may have a decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000005_0002
onto pixels R (red), Gi (green), G2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, wherein b may have the value of 1/a. i
Another embodiment may have a Decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple {Cr, cb, Y, DY} according to
Figure imgf000006_0001
onto R (red), G (green) and B (blue) components of a sample of an RGB image, wherein b may have the value of 1 la.
According to another embodiment, a method for encoding a Bayer pattern raw image may have the steps of: mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000006_0002
which forms a sample of the color image, and subjecting the color image to a multicomponent picture encoding to obtain a compressed data stream.
According to another embodiment, a method for reconstructing from a compressed data stream a Bayer pattern raw image may have the steps of: decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to
z
Figure imgf000007_0001
onto pixels R (red), Gi (green), G2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, wherein b may have the value of 1 la.
Another embodiment may have a data stream having encoded thereonto a color image generated using a method for encoding a Bayer pattern raw image, the method having the steps of: mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000007_0002
which forms a sample of the color image, and subjecting the color image to a multi- component picture encoding to obtain a compressed data stream.
Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the method for reconstructing from a compressed data stream a Bayer pattern raw image, the method having the steps of: decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple (Cr, Cb, Y, DY} according to
Figure imgf000008_0001
onto pixels R (red), Gi (green), Gz (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, when said computer program is run by a computer, wherein b may have the value of Ma.
An advantage of concepts of the present application stems from an efficient way of handling the pixel cell blocks of raw images. They are transformed to a color component quadruple forming of a sample of a color image, the quadruple
comprising three color values of a target color space and a pseudo color component. This may be done lossless. The color transformation fulfills two tasks concurrently: a decorrelation of the information of the raw data values is achieved - at least for homogenous areas which is the most part of images, thereby reducing the entropy and allowing better compression efficiency. In the case of a mathematically lossless coding the resulting color image, the overall compression results in a lossless transformation of data into a data stream from which either the original raw image may be derived, or an image corresponding to the raw image which shows the content in the target color space, but is of half width and height compared to the raw image, i.e. has half resolution.
BRIEF DESCRIPTION OF THE DRAWINGS
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
FIG. 1 shows schematically a pixel cell block of an image sensor subjected to a demosaicing step;
FIG. 2 illustrates schematically a first and a second apparatus configured to perform color transforming a color image, according to a first and a second embodiment of the present invention; FIG. 3 illustrates schematically a raw image encoder, according to a specific embodiment of the present invention; and
FIG. 4 illustrates schematically a decoder, according to a specific embodiment of the present invention;
DETAILED DESCRIPTION OF THE INVENTION
The subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It is understood that subject matter embodiments can be practiced without these specific details.
In particular, the description starts with presenting a specific example, sets out advantages and then provides further embodiments rendering clear that the advantages may also be achieved by slightly deviating from the specific example outlined before.
Many cameras today are built with only one image sensor chip for image capturing. These sensors are equipped with color filters on top of the photosensitive pixel cells to capture multiple light spectra or color components. Fig. 1 shows schematically a pixel cell block of a commonly used image sensor equipped with a color filter pattern, the so called Bayer pattern. This pattern is a set of (one) red, (two) green and (one) blue color filters (RGB filters) on top of a pixel cell block formed by four pixels, each to create one red, one blue and two green sensitive pixels. The captured values from these pixel cells are the so called raw data. However for displaying a color image at least 3 color values (RGB) for each pixel cell position are required. For this reason the raw data may be processed - this process is called demosaicing - to three color components values (typically RGB or YCt>Cr values) for each pixel cell position. This can be done by linear interpolation.
An effective compression of Bayer raw data is, however, feasible as outlined now.
The first step in compressing an RGB color image is typically a color transformation, as the information in the color components are strongly correlated and this redundancy should be minimized. The result is typically a luminance component and two color difference components YUV or YCbCr.
To compress Bayer raw data also a color decorrelation should be used. In comparison to standard RGB images however we can use the neighbored four color components R, Gi, G2 and B on slightly different positions. The color decorrelation itself should be reversible to recreate the original raw image. If we neglect the slightly different position, we can use the following formula:
Figure imgf000010_0001
The decorrelated components are now one Red-Chrominance Cr, one Blue- Chrominance Cb, one Luminance Y and one pseudo Delta-Luminance DY.
The following decorrelation has multiple advantages.
Cr = R - Y R = Cr + Y
Cb = B - Y B = Cb + Y
Y = (R + G1 + G2 + B)/4 G1 = -1/2 Cr -1/2 Cb + Y +1/2 DY
DY = Gt - G2 G2 = G1-DY
Figure imgf000010_0002
Firstly, it is reversible. That means the four components Cr, Cb, Y and DY can be transformed back lossless to R, G-i, G2, B. Only a lossy compression between forth and back transformation would result in losses.
Secondly, the first three components Cr, Cb, Y can be used directly to transform them into an RGB image with half width and half height by using G as mean value of Gi and G2. This is equivalent to consider one Bayer pixel cell block as one RGB macro pixel.
Figure imgf000011_0001
Third, the three components Cr, Cb and Y can be considered as color image with a residual DY. By using all 4 components the raw image can be reconstructed.
Summarizing, using the above color transform for compression of raw data we are
able to achieve high compression efficiency owing to the good decorrelation
behavior. Nevertheless, it is easy to use the transformed Cr, Cb, Y data after
transformation as RGB image with half size and half width without greater efforts.
The residual DY and the half resolution image together may be used to reconstruct the Bayer raw data to full resolution image.
FIG. 2 illustrates schematically a color transformation and its retransformation which underlies embodiments for a raw image encoder and its pendant, the decoder,
embodiments of which are further outlined below.
The forward color transform 5 is for color transforming of a raw image 10 into a color image 18. The color image 10 is formed by a number of X lines by Y columns of pixel cell blocks 12, wherein each cell blocks 12 comprises four pixels 14, arranged as a
2x2 sub-block. That is, the raw image 10 has N times M pixels clustered into N/2
times M/2 superpixels, called pixel cell blocks 12. To compress the raw data in image 10 which might correspond to a Bayer pattern, a color decorrelation is realized by the color transformation 5. In comparison to standard multi-component images such as image 18 and such as RGB images, for example, the neighbored four color
components R, G-i , G2 and B in a pixel cell block 12 of raw image 10 reside on
slightly different positions.
The color transform 5 is designed to deal with a raw image 10 of such a color pattern.
As said, the raw image 10 is partitioned into pixel cell blocks 12, each pixel cell block 12 comprising four pixels 14, each pixel 14 being associated with one of three base colors so that each pixel cell block 12 comprises, for each of the three base colors, at
< !east one pixel. For each pixel cell block 12, the four pixels 14 of the pixel cell block 12 are mapped onto a color component quadruple 16 forming a sample of the color image 18.
The mapping 5 may be done according to (1) : each pixel cell block 12 of a Bayer pattern raw image 10 which comprises four pixels R red, Gi green, G2 green, and B blue, is mapped onto a color component quadruple {Cr, Cb, Y, DY} 16 according to (1), with the quadruple forms a sample of the color image 18.
The color component quadruple 16 comprises three color component values 20 of a target color spaces and one pseudo color component value 22. The decorrelated components color component quadruple 16 may be referred as e.g. color component values 20 being one Red-Chrominance Cr, one Blue-Chrominance Cb, one
Luminance Y as well as the pseudo color component value 22 as a pseudo Delta- Luminance DY, That is, the afore-mentioned target color space may be elected to be a YCr Cb color space. The first three component values (20) Cr, Cb, Y can be used e.g. to be directly transformed into an RGB image with half width and half height. This is equivalent to consider one pixel cell block 12 as one RGB macro pixel. However, alternatives using other color spaces are feasible as well
The four color components Cr, Cb, Y and DY may be transformed back lossless to the color components R, G-i, G2 and B. This is done by the color retransformation 25. Function (1) and (2) may, thus, be based on a linear mapping. In transform 5 each of the three color component values 20 may result from a weighted sum of the four pixels 14 of the pixel cell block 12. The weights of the weighted sum of the two pixels of the pixel cell block may be associated accordingly with the same base color are equal to each other. See, for instance, the middle columns in the first three lines of the matrix of (1). Further, the linear mapping 5 may be such that the one pseudo color component value 22 results from a further weighted sum of the two pixels of the pixel cell block associated with the same base color, wherein only the weights of the weighted sum of the two pixels being of opposite sign, See the lower row in (1), for instance.
FIG. 2 illustrates schematically also the color retransforming procedure 25 from color image 18 to raw image 10. The transform 25 maps, for each sample of the color image 18, the color component quadruple 16 representing this sample, onto the four pixels 14 of the corresponding pixel cell block 12, That is, image 18 comprises as many samples as raw image 10 has pixel cell blocks 12. The color component quadruple 16 comprises three color component values 20 of a predetermined color space and a pseudo color component value 22.
In the example set out below, the retransform 25 maps each color component quadruple {Cr, Cb, Y, DY} 16 according to (2) onto pixels R red, Gi green, G2 green, and B blue of a corresponding pixel cell block 12 of the Bayer pattern raw image 10, Again, other examples would be feasible as well. In particular, the mapping may be a linear mapping where the two pixels of the pixel cell block 12 being associated with the same base color respectively, i.e. G1 and G2, result from a weighted sum of the color component quadruple 16 with weights of the weighted sum for the two pixels being equal. See the first three columns in the middle two lines of the matrix in (2), for instance. The weights of the weighted sum of the one pseudo color component value may be of opposite sign. See the rightmost column in the middle two lines of the matrix in (2), for instance. The predetermined color space may be selected to be a YCr Cb color space or differently.
FIG. 3 illustrates schematically an encoding device 28 comprising the raw image encoder 5 and a multi-component picture encoder 30 configured to encode a color image 18 output by the apparatus into a compressed multi-component data stream 32. The multi-component picture encoding performed may comprise lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.
FIG. 4 illustrates schematically a raw image decoding device 40 configured for reconstructing a raw image 10 from a multi-component data stream 32 comprising a decoder 25 and a multi-component picture decoder 42 configured to decode a color image 18 from the multi-component data stream 32. The multi-component picture decoding may comprise spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.

Claims

1. Raw image encoder for encoding a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, the raw image encoder configured to subject the raw image to a color transformation to obtain a color image, by mapping, for each pixel cell block, the four pixels of the pixel cell block onto a color component quadruple forming a sample of the color image, the color component quadruple comprising three color component values of a target color space, and one pseudo color component value, and subject the color image to a multi-component picture encoding to obtain a compressed data stream.
2. Raw image encoder according to claim 1 , configured to use, for the mapping, a reversible function.
3. Raw image encoder according to claim 1 , configured to use for the mapping, a linear mapping so that each of the three color component values results from a weighted sum of the four pixels of the pixel cell block in a manner so that weights of the weighted sum of the two pixels of the pixel cell block being associated with the same base color are equal to each other.
4. Raw image encoder according to claim 3, configured so that, according to the linear mapping, the one pseudo color component value results from a further weighted sum of the two pixels of the pixel cell block associated with the same base color, only, the weights of the weighted sum of the two pixels being of opposite sign.
5. Raw image encoder according to claim 3, wherein the multi-component picture encoding comprises lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.
6. Decoder for reconstructing from a compressed data stream a raw image of a predetermined color pattern according to which the raw image is partitioned into pixel cell blocks, each pixel cell block comprising four pixels, each pixel being associated with one of three base colors so that each pixel cell block comprises, for each of the three base colors, at least one pixel, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and subject the color image to a color transformation to obtain the raw image, by mapping, for each sample of the color image, the color component quadruple to obtain the four pixels of a corresponding pixel cell block, wherein the color component quadruple comprises three color component values of a predetermined color space and pseudo color component value.
7. Decoder according to claim 6, configured to use, for the mapping, a reversible function.
8. Decoder according) to claim 6, configured to use for the mapping a linear mapping so that the two pixels of the pixel cell block being associated with the same base color respectively result from a weighted sum of the color component quadruple with weights of the weighted sum for the two pixels being equal.
9. Decoder according to claim 7, wherein weights of the weighted sum of the one pseudo color component value are of opposite sign.
10 Decoder according to claim 6, the multi-component picture decoding
comprises spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.
11. Raw image encoder for encoding a Bayer pattern raw image, configured to
map each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000016_0001
which forms a sample of the color image, and subject the color image to a multi-component picture encoding to obtain a compressed data stream,
G
12. Raw image encoder according to claim 11 , wherein the multi-component picture encoding comprises lossy coding, spatial transform coding, entropy coding, spatial predictive coding, residual transform coding, and/or motion compensation predictive coding.
13. Decoder for reconstructing from a compressed data stream a Bayer pattern raw image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple {Cr, Cb, Y, DY) according to
Figure imgf000017_0001
onto pixels R (red), Gi (green), 62 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image.
14. Decoder according to claim 13, the multi-component picture decoding comprises spatial transform decoding, entropy decoding, spatial predictive decoding, residual transform decoding, and/or motion compensation predictive decoding.
15. Decoder according to claim 13, configured to, in a coding mode option, reconstruct from the compressed data stream an RGB image, by decoding from the compressed data stream by multi-component picture decoding the color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple (Cr, Cb, Y, DY} according to
Figure imgf000018_0001
IDY1 onto R (red), G (green) and B (blue) components of a sample an RGB image.
Decoder for reconstructing from a compressed data stream an RGB image, configured to decode from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and map each color component quadruple (Cr, Cb, Y, DY} according to
cy
Figure imgf000018_0002
onto R (red), G (green) and B (blue) components of a sample an RGB image.
17. Method for encoding a Bayer pattern raw image, comprising
mapping each pixel cell block of the Bayer patern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000019_0002
which forms a sample of the color image, and subjecting the color image to a multi-component picture encoding to obtain a compressed data stream.
18. Method for reconstructing from a compressed data stream a Bayer pattern raw image, comprising decoding from a compressed data stream by multi-component picture decoding a color image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000019_0001
Figure imgf000020_0001
onto pixels R (red), Gi (green), G2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image.
19. Data stream having encoded thereonto a color image generated using a method for encoding a Bayer pattern raw image, comprising
mapping each pixel cell block of the Bayer pattern raw image which comprises four pixels R (red), Gi (green), G2 (green), and B (blue), onto a color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000020_0002
which forms a sample of the color image, and subjecting the color image to a multi-component picture encoding to obtain a compressed data stream.
20. A non-transitory digital storage medium having a computer program stored thereon to perform the method for reconstructing from a compressed data stream a Bayer pattern raw image, comprising decoding from a compressed data stream by multi-component picture decoding a coior image, samples of which are respectively formed by a color component quadruple, and mapping each color component quadruple {Cr, Cb, Y, DY} according to
Figure imgf000021_0001
onto pixels R (red), Gi (green), G2 (green), and B (blue) of a pixel cell block of the Bayer pattern raw image, when said computer program is run by a computer.
PCT/EP2019/068013 2018-07-06 2019-07-04 Compression of a raw image Ceased WO2020007990A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/029,343 2018-07-06
US16/029,343 US10721470B2 (en) 2018-07-06 2018-07-06 Compression of a raw image

Publications (1)

Publication Number Publication Date
WO2020007990A1 true WO2020007990A1 (en) 2020-01-09

Family

ID=67185031

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2019/068013 Ceased WO2020007990A1 (en) 2018-07-06 2019-07-04 Compression of a raw image

Country Status (2)

Country Link
US (1) US10721470B2 (en)
WO (1) WO2020007990A1 (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3367659A1 (en) * 2017-02-28 2018-08-29 Thomson Licensing Hue changing color gamut mapping
CN111654705B (en) * 2020-06-05 2022-11-11 电子科技大学 Mosaic image compression method based on color space conversion
WO2022141515A1 (en) * 2020-12-31 2022-07-07 深圳市大疆创新科技有限公司 Video encoding method and device and video decoding method and device
CN114189677B (en) * 2021-12-20 2024-06-28 深圳市同为数码科技股份有限公司 Color uniformity evaluation method, device, computer equipment and medium
KR102650664B1 (en) * 2022-09-02 2024-03-25 삼성전자주식회사 Apparatus and method for obtaining image emplying color separation lens array
IT202300016971A1 (en) * 2023-08-08 2025-02-08 St Microelectronics Int Nv METHOD AND SYSTEM FOR THE EFFICIENT GENERATION OF MONOCHROME VIDEO DATA, ESPECIALLY FOR LASER BEAM SCANNING APPLICATIONS
CN118741135B (en) * 2024-09-03 2024-12-03 浙江华感科技有限公司 Video code stream encoding method and video code stream decoding method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6134347A (en) * 1997-09-30 2000-10-17 Sharp Kabushiki Kaisha Image filming and compression system, image filming and compression method, and recording medium storing processing program therefor
EP1575262A1 (en) * 2002-12-18 2005-09-14 Nokia Corporation Image data compression device and decompression device and image data compression program and decompression program
EP1657909A2 (en) * 2004-10-19 2006-05-17 Microsoft Corporation System and method for encoding mosaiced colour image data employing a reversible color transform

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3971065A (en) 1975-03-05 1976-07-20 Eastman Kodak Company Color imaging array
JP2003284084A (en) * 2002-03-20 2003-10-03 Sony Corp Image processing apparatus and method, and method of manufacturing image processing apparatus
US10521920B2 (en) * 2017-09-21 2019-12-31 Intel Corporation Reducing textured IR patterns in stereoscopic depth sensor imaging

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6134347A (en) * 1997-09-30 2000-10-17 Sharp Kabushiki Kaisha Image filming and compression system, image filming and compression method, and recording medium storing processing program therefor
EP1575262A1 (en) * 2002-12-18 2005-09-14 Nokia Corporation Image data compression device and decompression device and image data compression program and decompression program
EP1657909A2 (en) * 2004-10-19 2006-05-17 Microsoft Corporation System and method for encoding mosaiced colour image data employing a reversible color transform

Also Published As

Publication number Publication date
US10721470B2 (en) 2020-07-21
US20200014923A1 (en) 2020-01-09

Similar Documents

Publication Publication Date Title
US10721470B2 (en) Compression of a raw image
US11297337B2 (en) Compression and decoding of single sensor color image data
US10038909B2 (en) Compression of light field images
CN109842799B (en) Intra-frame prediction method and device of color components and computer equipment
CN110612722A (en) Method and device for encoding and decoding digital light field images
EP1173005B1 (en) Method and apparatus for color image data processing and compression
Hernandez-Cabronero et al. Lossless compression of color filter array mosaic images with visualization via JPEG 2000
CN114788280A (en) Video coding and decoding method and device
WO2019237659A1 (en) Blind compressive sampling method, apparatus and imaging system
JP2011015347A (en) Apparatus and method for processing image, program and recording medium
US20130027584A1 (en) Method and apparatus for frame rotation in the jpeg compressed domain
JP5309046B2 (en) Encoding device and program
US20160286097A1 (en) Methods and System for Sparse Blue Sampling
EP4052216B1 (en) Image processor
Bazhyna et al. Lossless and near lossless compression of real color filter array data
US20260039969A1 (en) Color filter array image data processing
CN113170160B (en) ICS frame transformation method and device for computer vision analysis
CN108989820B (en) Data compression method and device adopting respective corresponding chroma sampling formats at all stages
JP6655790B2 (en) Imaging device and imaging system
JP2008124530A (en) Raw data compression method
Cuce et al. Compression of images in CFA format
Bazhyna et al. Lossless compression of Bayer pattern color filter arrays
CN117915087A (en) Multispectral remote sensing image compression method, storage medium and device based on Bayer encoding
TW202533570A (en) Neural network video coding in-loop filtering in transform domain
Buemi et al. Bayer pattern compression by prediction errors vector quantization

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19736688

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19736688

Country of ref document: EP

Kind code of ref document: A1