EP1697897B1 - Image processing - Google Patents

Image processing Download PDF

Info

Publication number
EP1697897B1
EP1697897B1 EP04749156.8A EP04749156A EP1697897B1 EP 1697897 B1 EP1697897 B1 EP 1697897B1 EP 04749156 A EP04749156 A EP 04749156A EP 1697897 B1 EP1697897 B1 EP 1697897B1
Authority
EP
European Patent Office
Prior art keywords
intensity
color
image
representation
codeword
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP04749156.8A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP1697897A1 (en
Inventor
Jacob STRÖM
Tomas AKENINE-MÖLLER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telefonaktiebolaget LM Ericsson AB
Original Assignee
Telefonaktiebolaget LM Ericsson AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget LM Ericsson AB filed Critical Telefonaktiebolaget LM Ericsson AB
Priority to PL13172504T priority Critical patent/PL2688042T3/pl
Priority to EP13172504.6A priority patent/EP2688042B1/en
Publication of EP1697897A1 publication Critical patent/EP1697897A1/en
Application granted granted Critical
Publication of EP1697897B1 publication Critical patent/EP1697897B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/98—Adaptive-dynamic-range coding [ADRC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component

Definitions

  • the present invention generally refers to image processing, and in particular to methods and systems for encoding and decoding images.
  • 3D graphics and images have a number of appealing applications on such terminals, including games, 3D maps and messaging, screen savers and man-machine interfaces.
  • a 3D graphics rendering process typically comprises three sub-stages. Briefly, a first stage, the application stage, creates several triangles. The corners of these triangles are transformed, projected and lit in a second stage, the geometry stage. In a third stage, the rasterization stage, images, often denoted textures, can be "glued" onto the triangles, increasing the realism of the rendered image. The third stage also performs sorting using a z-buffer.
  • rendering of images and textures, and in particular 3D images and graphics is a computationally expensive task in terms of memory bandwidth and processing power required for the graphic systems.
  • textures are costly both in terms of memory, the textures must be placed on fast on-chip memory, and in terms of memory bandwidth, a texture can be accessed several times to draw a single pixel.
  • an image (texture) encoding method or system is typically employed.
  • Such an encoding system should result in more efficient usage of expensive on-chip memory and lower memory bandwidth during rendering and, thus, in lower power consumption and/or faster rendering.
  • This reduction in bandwidth and processing power requirements is particularly important for thin clients, such as mobile units and telephones, with a small amount of memory, little memory bandwidth and limited power (powered by batteries).
  • BTC block truncation coding
  • Their scheme encodes gray scale images by considering a block of 4 pixels ⁇ 4 pixels at a time. For each such block, two 8-bit gray scale values are generated and each pixel in the block then uses a single bit to index one of these gray scales. This results in a compression rate of 2 bits per pixel (bpp).
  • BTC suffers from production of artifacts, above all in regions around edges and in low contrast areas containing a sloping gray level. Furthermore, edges in a gray scale image processed by BTC have a tendency to be ragged.
  • CCC color cell compression
  • Iourcha et al. disclose a texture compression scheme called S3TC (S3 Texture Compression) or DXTC (DirectX Texture Compression), that can be seen as an extension of CCC.
  • An image is decomposed into a number of image blocks of 4 pixels ⁇ 4 pixels.
  • Each such image block is encoded into a bit sequence of 64 bits, thus resulting in a compression rate of 4 bpp.
  • the 64-bit sequence comprises two basic colors or color codewords (16 bits each) and a 32-bit sequence of 2-bit indices, one index for each pixel in the image block.
  • a color palette of four colors is generated.
  • the first two RGB (red, green and blue) colors of the palette correspond to the two basic colors (codewords).
  • the two additional colors, situated between the basic colors in the RGB space, are then interpolated therefrom.
  • Each 2-bit index then identifies, for each pixel, one of the four colors of the palette to use for that pixel.
  • the S3TC scheme works fairly well for computer terminals, it is not well adapted for mobile units and other thin clients. Such mobile units typically only have memory busses of 16 or 32 bits at best. Thus, at least two, and possibly up to four, memory accesses are required to read out the 64-bit compressed version of an image block, if S3TC is implemented in a mobile unit. In addition, during the interpolation of the two additional colors of the color palette, multiplication by 1/3 and 2/3 is performed, which is not ideal in hardware. The compression using S3TC is also relatively time consuming, at least on a mobile terminal.
  • POOMA a variant of S3TC
  • an image block of 3 pixels ⁇ 2 pixels is encoded into 32 bits, giving 5.33 bpp.
  • the encoded 32-bit representation of an image block is adapted for the memory busses of mobile phones, which typically are 32 bits at best.
  • a pixel can be rendered using only one memory access compared to two accesses for S3TC.
  • POOMA uses two base colors but only one additional color interpolated between the base colors, resulting in a color palette of three colors.
  • a major disadvantage of POOMA is the block size of 3 ⁇ 2 pixels.
  • calculation of the block address for a particular pixel or texel (texture element) requires division by 3, which is not ideal for hardware design.
  • the widths and heights of textures (images) in graphics are typically always a power of two, which means that a block width of 3 is inconvenient.
  • S3TC the encoding using POOMA is relatively time consuming, in particular when implemented on a mobile terminal.
  • Fenny discloses an image-encoding scheme used in the MBX graphics hardware platform for mobile phones. This scheme uses two low-resolution images, where one image is usually a low-pass filtered version of the original image. During decoding, a bilinear magnification (upscaling) of these two images is created. Each pixel also stores a blend factor between these two upscaled images. 64 bits are used for encoding each image block and a compression rate of 2 bpp and 4 bpp are described. Information from neighboring image blocks is needed, which complicates decoding.
  • the present invention overcomes these and other drawbacks of the prior art arrangements.
  • Yet another object of the invention is to provide image encoding and decoding adapted for usage in thin clients with low memory and memory bandwidth capacity.
  • a further object of the invention is to provide image encoding and decoding adapted for three-dimensional (3D) graphics and images.
  • the present invention involves image processing in the form of encoding (compressing) an image and decoding (decompressing) an encoded (compressed) image.
  • an image to be encoded is decomposed into a number of image blocks comprising multiple image elements (pixels or texture elements, texels).
  • Each image element in a block is characterized by a color, e.g. a 12-bit RGB (red, green, blue) color.
  • a color codeword is determined for the image block.
  • the color codeword is a representation of the colors of the image elements of the image block.
  • a preferred representation is an average value of the colors of the image elements in block, quantized into 12 bits (4 bits for each of the three color components for a RGB color).
  • one and the same color codeword i.e. color representation
  • an intensity codeword is provided for the image block.
  • This intensity codeword is a representation of a set of multiple intensity modifiers that are used (during decoding) for modifying or modulating the intensity of the image elements in the image block.
  • intensity representations for the image elements in the image block are selected. Each such intensity representation is associated with an intensity modifier from the intensity modifier set. In other words, the intensity representation allows identification of which intensity modifier from the set to use for a specific image element of the block.
  • the resulting size of an encoded image block is, thus, only 32 bits and a compression rate of 4 bits per pixel (image element) is obtained. This small 32-bit size is well adapted for thin clients, such as mobile units and telephones, which typically have memory busses of 16 or 32 bits. As a result, only one or at worst two memory accesses are then needed to read out the encoded image block from a memory location.
  • the intensity codeword is an intensity index allowing identification of an intensity modifier set. This index could then identify or point to the set in a table or codebook comprising several different intensity modifier sets. Each set preferably comprises four (mathematically) complementary modifier values. In such a case, the modifier sets stored in the table only have to comprise two different intensity modifier values each and then the other two (complementary) values of the set could be calculated therefrom.
  • the intensity table preferably comprises both sets that include small intensity modifier values, which are adapted for allowing representation of smoothly changing surfaces, and sets that include large intensity modifier values, which are adapted for allowing representation of sharp edges.
  • the encoded image block(s) that should be decoded is identified and fetched from e.g. a memory location. Once the correct encoded image block is identified an intensity modifier set is provided. This modifier set is provided based on the intensity codeword in the encoded image block. This set provision is preferably performed by identifying, by means of the intensity codeword, an intensity modifier set from an intensity table comprising multiple modifier sets.
  • a color representation is generated for at least one of the image elements of the image block. This color generation is performed based on the color codeword in the encoded block representation.
  • the intensity modifier to use for the image element that should be decoded is then selected.
  • the modifier value is selected from the provided modifier set based on the intensity representation associated with the image element and found in the representation sequence of the encoded image block. Once the correct intensity modifier value is selected, the intensity of the image element is modified with this value.
  • the selection of intensity modifier and modification of the intensity are preferably performed for all image elements that should be decoded in the current encoded image block.
  • the block decoding is then preferably repeated for all image blocks that comprises image elements that should be decoded. Thereafter, a decoded representation of an original image, or a portion thereof, can be generated based on the decoded image elements and blocks.
  • the color representation is preferably generated by expanding the three 4-bit color components of the color codeword into three 8-bit components.
  • the resulting 24-bit color is then assigned to the image element(s) of the image block that are to be decoded.
  • the intensity of the image elements is preferably modified by adding or multiplying the intensity modifier to each color component, or each color component is otherwise modulated with the intensity modifier. Thereafter, the resulting intensity modified color components are clamped between a minimum and maximum threshold value.
  • the image encoding and decoding of the invention can be applied to several different types of images, including 1D, 2D and 3D "synthetic" images, photos, text, games, 3D maps and scenes, 3D messages, e.g. animated messages, screen savers, man-machine interfaces (MMIs), etc.
  • 1D, 2D and 3D "synthetic" images photos, text, games, 3D maps and scenes
  • 3D messages e.g. animated messages, screen savers, man-machine interfaces (MMIs), etc.
  • the invention is well adapted for thin clients with limited memory capacity and bandwidth.
  • the encoding is very fast so that it can be performed also in terminals with low clock frequencies.
  • the decoding can extremely simply be implemented in e.g. hardware using only a few standard components.
  • the present invention relates to image and graphic processing, and in particular to encoding or compressing images and decoding or decompressing encoded (compressed) images.
  • an image is decomposed or divided into a number of image blocks.
  • Each such image block then comprises multiple image elements having, among others, a certain color.
  • the image blocks are then encoded to generate an encoded representation of the image.
  • an encoded image or graphic primitive subsequently is to be rendered, e.g. displayed on a screen
  • the relevant image elements of the encoded image blocks are identified and decoded. These decoded image elements are then used to generate a decoded representation of the original image or graphics primitive.
  • the present invention is well adapted for usage with three-dimensional (3D) graphics, such as games, 3D maps and scenes, 3D messages, e.g. animated messages, screen savers, man-machine interfaces (MMIs), etc., but is not limited thereto.
  • 3D graphics such as games, 3D maps and scenes, 3D messages, e.g. animated messages, screen savers, man-machine interfaces (MMIs), etc.
  • MMIs man-machine interfaces
  • the invention could also be employed for encoding other types of images or graphics, e.g. one-dimensional (1D) or two-dimensional (2D) images.
  • image element refers to an element in an image block or encoded representation of an image block.
  • This image block corresponds to a portion of an image or texture.
  • an image element could be a texel (texture element) of a (1D, 2D or 3D) texture or a pixel of a (1D, 2D or 3D) image.
  • an image element is characterized by certain image-element properties, such as a color value.
  • image is used to denote any 1D, 2D or 3D image or texture that can be encoded and decoded by means of the present invention, including but not limited to photos, game type textures, text, drawings, etc.
  • Fig. 1 illustrates a (lossy) method of encoding an image according to the present invention.
  • a first step S1 the image is decomposed or divided into a number of image blocks. Each such image block then comprises multiple image elements.
  • Figs. 2 and 3 schematically illustrate two examples of an image block 600 with eight image elements 610 according to the present invention. In Fig. 2 , the height is two image elements 610 and the width is four image elements 610, i.e.
  • the whole image block is preferably decomposed into (nonoverlapping) image blocks in step S1.
  • a portion of an image is encoded and, thus, only this portion is decomposed into image blocks.
  • a color codeword is determined for an image block.
  • This color codeword is a representation of the colors of the image elements in the image block.
  • the color codeword is a representation of an average color of the image elements of the block.
  • the color could be a RGB (Red, Green, Blue) color, a color in the YUV space or YCrCb space, or any other proprietary color space used in image and graphics processing and management.
  • the color codeword is preferably in the same color format (space) as the image. However, in some cases, it may be useful to convert the image to a different color format, i.e.
  • the color codeword is preferably a 12-bit color-representation sequence.
  • a RGB color codeword could comprise 4 bits for the red color component, 4 bits for the green component and 4 bits for the blue component.
  • a YUV color codeword could include 6 bits, 3 bits and 3 bits, respectively, for the three different components.
  • one and the same color codeword (i.e. color representation) is generated for an image block, i.e. for all the image elements of the image block.
  • This intensity codeword is a representation of a set of multiple intensity modifiers that are used (during decoding) for modifying the intensity of the image elements in the image block.
  • the intensity codeword is an intensity index allowing identification of an intensity modifier set. This index could then identify or point to the set in a table or codebook comprising several different intensity modifier sets. Each set comprises two or more intensity modifier values, preferably at least four modifier values.
  • the modifier values of a set are preferably mathematically complementary values, i.e. each set is preferably symmetrical.
  • a possible intensity modifier set could be [-a, -b, b, a], where a and b are positive integers and a>b.
  • the intensity table preferably comprises sets including small intensity modifier values, which are adapted for allowing representation of smoothly changing surfaces.
  • the table preferably also comprises sets that include large intensity modifier values, which are adapted for allowing representation of sharp edges.
  • the actual intensity modifier values of the sets in the table can be found by starting with random values and then optimizing these values using a number of different optimization schemes and algorithms, such as versions of the LBG-algorithm (Linde, Buzo and Gray) [6], simulated annealing and coordinate search, which are known to a person skilled in the art.
  • LBG-algorithm Lide, Buzo and Gray
  • simulated annealing and coordinate search which are known to a person skilled in the art.
  • a handful of images of different types e.g. photos, game type textures, text, etc., can be used as training data.
  • the intensity modifiers of a set can be forced to be symmetrical, as was discussed above, and/or the intensity modifiers of a given set could be a copy of intensity modifiers of another set modified by a factor, e.g. two.
  • Table 1 illustrates a presently preferred example of an intensity table comprising 16 sets of intensity modifiers, with four modifier values in each set.
  • Table 1 Set Codeword Intensity modifier value 1 0000 bin -8 -2 2 8 2 0001 bin -12 -4 4 12 3 0010 bin -31 -6 6 31 4 0011 bin -34 -12 12 34 5 0100 bin -50 -8 8 50 6 0101 bin -47 -19 19 47 7 0110 bin -80 -28 28 80 8 0111 bin -127 -42 42 127 9 1000 bin -16 -4 4 16 10 1001 bin -24 -8 8 24 11 1010 bin -62 -12 12 62 12 1011 bin -68 -24 24 68 13 1100 bin -100 -16 16 100 14 1101 bin -94 -38 38 94 15 1110 bin -160 -56 56 160 16 1111 bin -254 -84 84 254
  • the intensity modifier sets 9-16 are a copy of sets 1-8 multiplied by a factor of two.
  • the intensity codeword is preferably a 4-bit index (0000 bin - 1111 bin ) identifying one of the (16) sets, e.g. [-8, -2, 2, 8] for codeword 0000 bin (0000 base 2), of the table. Due to careful choice of the modifier values in the sets (symmetrical sets and half of the sets are a factor two of the remaining half), the entire Table 1 can be reconstructed using only 16 modifier values, and the remaining 48 values could be calculated therefrom.
  • the present invention is, though, not limited to usage of Table 1, but could use other tables with other intensity modifier sets and values.
  • the size of the intensity codeword might have to be changed.
  • the codeword size could be limited to one bit (two bits, three bits or more than four bits).
  • the number of intensity modifier values per set could differ from four, e.g. five values could be used per set, giving an example of [-8, -2, 0, 2, 8].
  • the intensity values of the sets in the table could be determined using several different types of images as training data, as was discussed above.
  • the modifier values could be determined using training data corresponding to that image type, i.e. giving an intensity table dedicated for a specific image type. It could also be possible to have an intensity table with intensity modifier values adapted for a specific image. In these cases, i.e. table dedicated for image or image type, it might be necessary to include the intensity modifier values of the table in the compressed file of encoded image blocks or otherwise associate them therewith.
  • the intensity codeword does not have to be an index or pointer to an intensity modifier set in a table, but could actually be an intensity modifier set itself, e.g. comprises two modifier values, such as 2 and 8, and where the other modifier values, such as -2 and -8, can be determined from these two values.
  • a next step S4 selects intensity representations for the image elements in the image block.
  • Each such intensity representation is associated with one intensity modifier value from the intensity modifier set provided in step S3.
  • the intensity representation allows identification of which intensity modifier of the set to use for a specific image element of the block.
  • the intensity representation could be a 2-bit sequence identifying one of these four values, e.g. 11 bin corresponds to -8, 10 bin corresponds to -2, 00 bin corresponds to 2 and 01 bin corresponds to 8. If more than four intensity modifier values are used per set, then more than two bits are required for each intensity element to identify correct modifier.
  • Step S4 is preferably repeated for all image elements in the image block (schematically illustrated by line 1).
  • the result of the encoding of steps S2 to S4 is an encoded image block or more precisely an encoded (compressed) representation of the image block.
  • Such an encoded block representation 700 is illustrated in Fig. 4 .
  • the representation 700 (encoded image block) comprises the color codeword 710, the intensity codeword 720 and a sequence or bitmap 730 of intensity representations (preferably one intensity representation for each image element in the block). Note that the mutual order of the color codeword 710, intensity codeword 720 and intensity-representation sequence 730 of the encoded image block 700 may differ from what is illustrated in the figure.
  • the size of the sequence 730 is 16 bits. Furthermore, assume that corresponding sizes of the color and intensity codewords are 12 and 4 bits, respectively.
  • the total size of the encoded representation 700 of the image block is then 32 bits and a compression rate of 4 bits per pixel (image element) (bpp) is obtained.
  • This small (32-bit) size of the representation 700 is well adapted for thin clients, such as mobile units, which typically have memory busses of 16 or 32 bits. As a result, only one or at worst two memory accesses are then needed to read out the encoded representation 700.
  • steps S2 to S4 are preferably repeated for all image blocks provided during the decomposing of step S1 (schematically illustrated by line 2).
  • the result is then a sequence or file of encoded image blocks.
  • the resulting encoded image blocks (encoded representations of the image blocks) could be ordered in a file from left to right and top to bottom in the same order in which they were broken down in the block decomposing of step S1. The method then ends.
  • the encoded image could then be provided to a memory for storage therein until a subsequent rendering, e.g. display, of the image. Furthermore, the encoded image could be provided as a signal of encoded block representations to a transmitter for (wireless or wired) transmission to another unit.
  • Fig. 5 illustrates an embodiment of step S2 of Fig. 1 in more detail.
  • step S10 an average color of the image elements in the image block is determined.
  • the color of a pixel or texel (image element) of an image is represented by 24 bits of RGB color, i.e. 8 bits of the red component, 8 bits or the green component and 8 bits of the blue component.
  • the invention is not limited to this particular example, but can be applicable to any color representation of pixels and texels.
  • a next step S11 quantizes the average color.
  • the (24-bit) average color is preferably quantized into a 12-bit sequence (color codeword).
  • each 8-bit average component is quantized into a 4-bit average component.
  • the method then continues to step S3 of Fig. 1 .
  • Fig. 6 illustrates an embodiment of steps S3 and S4 of Fig. 1 in more detail.
  • Step S20 investigates the different intensity modifier sets of the table and the different modifier values of the sets and calculates an error value for each such modifier set and modifier value test. Based on these error values, a modifier set and intensity modifier values of the set that results in a smallest error value is selected in step S21. This is described in more detail below. The method then ends.
  • a color codeword and a correct intensity modifier set are selected. Once this is done, encoding of each image element in the image block is done by trying all the four intensity modifiers of the set and calculating the error. Assume that the original (24-bit) color of an image element is (R, G, B) and that the color codeword (quantized average color, 12 bits) is ( R ⁇ , ⁇ , B ⁇ ) and that the chosen modifier set is [-a, -b, b, a].
  • ⁇ 2 R ⁇ ⁇ + ⁇ - R 2 + G ⁇ ⁇ + ⁇ - G 2 + B ⁇ ⁇ + ⁇ - B 2 , where ⁇ ⁇ [-a, -b, b, a].
  • the ⁇ that minimizes the error ⁇ 2 is selected. This could be implemented by calculating the error value for a first selection of modifier values for the image elements and store this first error value (and the chosen selection of modifier values). Thereafter, an error value is calculated for a different selection of modifier values. This error value is then compared to the stored value. If it is smaller than the stored value, the stored value is replaced by this new error value and the modifier values used in the calculation of this error value are also stored. This is then repeated for all combinations of modifiers and sets.
  • Equation (2) will give the best performance in terms of Peak Signal/Noise Ratio (PSNR), since it will minimize the mean square error in the image. Perceptually, however, it may not produce the best result. For some pixels (image elements) p 1 and p 2 , where p 1 seems brighter than p 2 in the original image, p 2 may seem brighter than p 1 in a decoded version of the image. The reason for this is that the red, green and blue components do not give equal contributions to what the human visual system perceives as intensity. Since the green component gives a disproportionate contribution to the perception of intensity, its approximation should be represented in higher precision (more accurate) than those of red and blue.
  • PSNR Peak Signal/Noise Ratio
  • w G is preferably larger than w R and w B .
  • w R 5 16
  • w R 0.299
  • the quantized average color was simply used as a representation (color codeword) of the colors of the image elements in the image block.
  • both the colors and the intensity modifier sets are chosen, i.e. every possible combination is tried.
  • a further iteration through all 12 bits of color is added in addition to the previous iteration of all 4 bits of intensity modifier set and the 2 bits of intensity representation, which together gives 2 18 steps.
  • Encoding an image of 128 ⁇ 128 pixels takes approximately 5 minutes using the same PC laptop as for the simple compression. Although this might be far too long for run-time applications, it is not prohibitive for off-line encoding.
  • this embodiment of the present invention starts with a (24-bit) average color ( R , G , B ), but the color components of this average color are quantized together with the intensity components, i.e. selection of intensity modifier sets and values.
  • R low and R high denote the 4-bit quantization levels or values that are directly below and above R , respectively so that R low ⁇ R ⁇ R high .
  • the task is then to choose R ⁇ as either R low or R high .
  • the same is true for the green and blue components.
  • ⁇ the intensity modifier
  • Table 2 Color codeword Error value ⁇ 2 R low , G low , B low 2 3 ⁇ ⁇ R high , G low ,B low 2 3 ⁇ ⁇ + 17 ⁇ 17 + 2 ⁇ ⁇ R - ⁇ G - ⁇ B R low , G high , B low 2 3 ⁇ ⁇ + 17 ⁇ 17 + 2 ⁇ ⁇ G - ⁇ R - ⁇ B R low , G low , B high 2 3 ⁇ ⁇ + 17 ⁇ 17 + 2 ⁇ ⁇ B - ⁇ R - ⁇ G R low , G high , B high 2 3 ⁇ ⁇ + 17 ⁇ 17 - 2 ⁇ ⁇ R + ⁇ G + ⁇ B R high , G low , B high 2 3 ⁇ ⁇ + 17 ⁇ 17 - 2 ⁇ ⁇ R + ⁇ G + ⁇ B R high , G low , B high 2 3 ⁇ ⁇ + 17 ⁇ 17 - 2 ⁇ ⁇ R + ⁇ G + ⁇ B R
  • ⁇ is not required to be explicitly calculated, only the expressions (error representations) in the straight brackets of Table 2 have to be calculated in order to select the quantization levels (color codeword) to use. Further note that color codeword (R low , G low , Blow) and (R high , G high , B high ) give the same error value. This is under the assumption that any ⁇ (intensity modifier value) can be reached. In reality, however, ⁇ is limited to the intensity modifier values of the used modifier set(s), e.g. the modifier values of Table 1.
  • the total encoding time was not measurably changed compared with the simple encoding, i.e. an image of 128 ⁇ 128 pixels is still compressed in about 60 ms.
  • Another way to choose color codeword among the neighboring quantized colors is to compress the image block with each such quantized color and select the one associated with a smallest error.
  • Such an approach is somewhat slower than by simply using the error representations of Table 2 or 3 for determining a suitable color codeword since the block then has to be compressed eight times compared to one time.
  • Fig. 7 illustrates a flow diagram of a method of decoding an encoded image or encoded version of an original image according to the present invention.
  • the encoded image basically comprises several encoded representations of image blocks, such as representations 700 of Fig. 4 . These encoded block representations are preferably generated by the image encoding method discussed above in connection with Fig. 1 .
  • the method generally starts by identifying encoded image block(s) to decode. It could be possible that all encoded image blocks of an encoded image should be decoded to generate a decoded representation of the original image. Alternatively, only a portion of the original image is to be accessed. As a consequence, only a selected number of image blocks have to be decoded (or more precisely, a selected amount of image elements of certain image blocks have to be decoded).
  • step S30 provides an intensity modifier set.
  • This modifier set is provided based on the intensity codeword in the encoded representation. This set provision is preferably performed by identifying, by means of the intensity codeword, an intensity modifier set from a table, e.g. Table 1 above, comprising multiple modifier sets. However, in some applications it might be possible that the intensity codeword itself comprises the modifier set and that no table look-up is required.
  • a color representation is generated for at least one of the image elements of the image block (i.e. for the image element(s) that should be decoded). This color generation is performed based on the color codeword in the encoded block representation.
  • the intensity modifier to use for the image element that should be decoded is selected. The modifier value is selected from the modifier set provided in step S30 based on the intensity representation associated with the image element and found in the representation sequence of the encoded block representation. Once the correct intensity modifier value is selected in step S32, the intensity of the image element is modified or modulated with this value in step S33.
  • Intensity modification refers to modifying, e.g. adding or multiplying, all color components of the color representation by the (possibly weighted) intensity modifier value.
  • Steps S32 and S33 could be performed for several image elements in the image block (schematically illustrated by line 3). It is anticipated by the invention that in some applications, only a single image element is decoded from a specific image block, multiple image elements of a specific image block are decoded and/or all the image elements of a specific block are decoded.
  • Steps S30 to S33 are then preferably repeated for all image blocks that comprise image elements that should be decoded (schematically illustrated by line 4). This means that the loop of steps S30 to S33 could be performed once, but most often several times for different encoded image blocks and/ or several times for a specific encoded image block.
  • a decoded representation of the original image, or a portion thereof is generated based on the decoded image elements and blocks. Note that in some applications, several image elements have to be decoded in order to render a single pixel of the decoded representation. For example, during trilinear interpolation, eight neighboring image elements are decoded and for bilinear interpolation the corresponding number is four image elements, which is well known to the person skilled in the art. The method then ends.
  • Fig. 8 illustrates an embodiment of steps S30 and S31 of Fig. 7 in more detail.
  • a correct intensity modifier set is identified and selected from the intensity table by means of the intensity codeword. If the intensity modifier set stored in the intensity table comprises a first sub-set of modifier values, e.g. [a, b], a second sub-set of intensity modifier values can be determined from the values of the first sub-set, e.g. [-a, -b].
  • the quantized color of the color codeword preferably 12 bits, is expanded or extended into, preferably, 24 bits. For the case with a RGB color, each quantized 4-bit color component of the color codeword is then expanded into an 8-bit color component.
  • a 4-bit color component of 1010 bin is expanded into 1010 1010 bin .
  • the color codeword instead comprises 15 bits (5 bits for each R, G, B component)
  • the expansion would result in a 30-bit color.
  • the expanded color is then assigned to the image elements of the image block, which are to be decoded. The method then continues to step S32 of Fig. 7 .
  • Fig. 9 illustrates an embodiment of step S33 in Fig. 7 in more detail.
  • the identified and selected intensity modifier value is added to the expanded color.
  • this modifier value is added to all components (all three for a RGB color) of the color for the image element.
  • This could be implemented as a simple addition of the modifier value to all color components.
  • it could be preferred to weight the modifier value before adding it to the components. In such a case, different weights can be employed for the different color components.
  • another type of modification than a simple addition could be employed, e.g. multiplication, XOR or another modification.
  • the same modulation is performed to all components of the expanded color using one and the same intensity modifier value, although this value may be different weighted for the components.
  • the resulting intensity-modified color component values are clamped between a minimum color threshold and a maximum color threshold. For example, if after adding the (possibly weighted) intensity modifier value to a color component, the resulting value is smaller than the minimum threshold, the value is clamped to the value of this threshold.
  • the value of the threshold should instead be used for that component.
  • a non-limiting example of a minimum and maximum threshold is 0 and 255, respectively, for the case with 256 different colors. Corresponding thresholds for 1024 colors could be 0 and 1023.
  • the encoded representation of the image block is according to 1010 0101 0001 0111 11 01 10 00 10 01 00 00 bin (a517D890 hex ), where the bit 0-3 is the red component of the color codeword, bit 4-7 is the green component of the color codeword, bit 8-11 is the blue component of the color codeword, bit 12-15 is the intensity codeword and bit 16-31 is the sequence of intensity representations for the image elements of the block.
  • the color codeword is then decoded (expanded) to generate the color representation of the image block.
  • the correct intensity modifier set to use is selected from Table 1 based on the intensity codeword. As is seen in Table 1 an intensity codeword of 0111 bin corresponds to intensity modifiers [-127, -42, 42, 127].
  • the partly decoded image block is now according to: (43, 0, 0) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17) (170, 85, 17)
  • the intensity representation is 01 bin , i.e. the intensity modifier 127 should be added to all three color components.
  • the result after clamping is (255, 212, 144). Repeating this procedure for all image elements in the block would create the final decoded image block shown below: (43, 0, 0) (255, 212, 144) (128, 43, 0) (212, 127, 59) (128, 43, 0) (255, 212, 144) (212, 127, 59) (212, 127, 59) (212, 127, 59)
  • the image encoding (image block encoding) and image decoding (image block decoding or processing) scheme according to the present invention could be provided in a general data processing system, e.g. in a user terminal or other unit configured for processing and/or rendering images.
  • a terminal could be a computer.
  • the invention is well adapted for thin clients, such as Personal Digital Assistance (PDA), mobile units and telephones.
  • PDA Personal Digital Assistance
  • Such terminals are typically characterized by limited memory capacity and memory bandwidth, and are powered by batteries, i.e. also limited power supply. Since both encoding and decoding according to the present invention can be implemented very simply in hardware, software or a combination of hardware and software and an encoded image block preferably only has a maximum size of 32 bits, the invention could with advantage be applied to a thin client.
  • Fig. 10 illustrates a user terminal 100 represented by a mobile unit.
  • the invention is not limited to mobile units by could be implemented in other terminals and data processing units. Only means and elements in the mobile unit 100 directly involved in the present invention are illustrated in the figure.
  • the mobile unit 100 comprises a (central) processing unit (CPU) 200 for processing data, including image data, within the mobile unit 100.
  • a graphic system 130 is provided in the mobile unit 100 for managing image and graphic data.
  • the graphic system 130 is adapted for rendering or displaying images on a connected screen 120 or other display unit.
  • the mobile unit 100 also comprises a storage or memory 140 for storing data therein.
  • image data may be stored, in particular encoded image data (encoded image blocks) according to the present invention. Due to the small total size of image blocks (32 bits) and high compression rate (4 bpp), image data can efficiently be stored in the memory 140 also in cases with a mobile unit 100 with limited memory capacity.
  • An image encoder 210 is provided in the mobile unit 100.
  • This encoder 210 is configured for encoding an image or texture into an encoded representation of the image (or texture). As was discussed above, such an encoded representation comprises a sequence or file of multiple decoded image blocks.
  • This image encoder 210 may be provided as software running on the CPU 200, as is illustrated in the figure. Alternatively, or in addition, the encoder 210 could be arranged in the graphic system 130 or elsewhere in the mobile unit 100.
  • An encoded representation of an image from the block encoder 210 may be provided to the memory 140 over a (memory) bus 150, for storage therein until a subsequent rendering of the image.
  • the encoded image data may be forwarded to an input and output (I/O) unit 110 for (wireless or wired) transmission to other external terminals or units.
  • This I/O unit 110 can also be adapted for receiving image data from an external unit.
  • This image data could be an image that should be encoded by the image encoder 210 or encoded image data that should be decoded. It could also be possible to store the encoded image representation in a dedicated texture memory provided, for example, in the graphic system 130. Furthermore, portions of the encoded image could also, or alternatively, be (temporarily) stored in a texture cache memory, e.g. in the graphic system 130.
  • the (memory) bus 150 has a maximal bandwidth of 32 bits, a single memory access is required to fetch or read out an encoded image representation of the invention from the memory 140. If however, the bus 150 has larger bandwidth capacity, e.g. 64 bits or even 128 bits, multiple encoded image representations could be fetched in a single memory access. For example assume a 64-bit bus 150 and image block size according to Fig. 2 . If the image blocks are piled "on top of each other", an image block together with the subsequent image block in the memory 140 will form a 4x4 square of image elements. However, if the blocks are positioned "side by side", the image block together with the following block will form a 2x8 box. A 4x4 square is more preferred, since the probability of finding a desired image element in the 4x4 square is larger than for the 2x8 box, if some form of texture caching system is employed, which is well known to the person skilled in the art.
  • An image decoder 220 is provided in the mobile unit 100 for decoding an encoded image in order to generate a decoded image representation.
  • This decoded representation could correspond to the whole original image or a portion thereof.
  • the image decoder 220 provides decoded image data to the graphic system 130, which in turn typically processes the data before it is rendered or presented on the screen 120.
  • the image decoder 220 can be arranged in the graphic system 130, as is illustrated in the figure. Alternatively, or in addition, the decoder 200 can be provided as software running on the CPU 200 or elsewhere in the mobile unit 100.
  • the mobile unit 100 could be equipped with both an image encoder 210 and an image decoder 220, as is illustrated in the figure. However, for some terminals 100 it could be possible to only include an image encoder 210. In such a case, encoded image data could be transmitted to another terminal that performs the decoding and, possibly, rendering of the image. Correspondingly, a terminal 100 could only include an image decoder 220, i.e. no encoder. Such a terminal 100 then receives a signal comprising encoded image data from another terminal and decodes it to generate a decoded image representation. Thus, the encoded image signal could be wirelessly be transmitted between terminals using radio transmitter and receiver.
  • the units 110, 130, 200, 210 and 220 of the mobile unit 100 may be provided as software, hardware or a combination thereof.
  • Fig. 11 illustrates a block diagram of an embodiment of an image encoder 210 according to the present invention.
  • the encoder 210 typically comprises an image decomposer 215 for decomposing or dividing an input image into several image blocks.
  • the decomposer 215 is preferably configured for decomposing the image into image blocks comprising eight image elements (pixels or texels), i.e. having a general size of 8 ⁇ 1 or 8 ⁇ 1 image elements, more preferably 4x2 or 2x4 image elements.
  • This decomposer 215 could be adapted for decomposing different input images into image blocks with different sizes. For example, for a first image type an image block size of 4x2 image elements is used, whereas for a second type a block size of 8 ⁇ 1 could be used.
  • the decomposer 215 preferably receives input information, enabling identification of which image block format to use for a given image.
  • This embodiment of the image encoder 210 comprises a single block encoder 300.
  • This block encoder 300 encodes the image block(s) received from the image decomposer to generate encoded block representation(s).
  • Such an image block representation comprises a color codeword, an intensity codeword and a sequence of intensity representations.
  • the overall size of the block representation is much smaller than the corresponding size of the uncoded image block.
  • the block encoder 300 is preferably configured for processing (encoding) each image block from the decomposer 215 sequentially.
  • the block encoder 300 preferably comprises, or has access to, an intensity table 500 comprising multiple intensity modifier sets.
  • the modifier sets of the table 500 are used during the encoding for the generation of the intensity, and possibly color, codeword.
  • the intensity table 500 could be arranged in the block encoder 300 or elsewhere in the image encoder 210.
  • the image encoder 210 could comprise a single intensity table 500.
  • several different tables could be arranged in the encoder 210, where the intensity modifiers of the tables are adapted for different image types or a table could be adapted for a specific image.
  • a first intensity table could be used during encoding of a first image type, e.g. photo
  • a second table is used for encoding an image of a second type, e.g. text.
  • a single intensity table 500 generated with training data from several different image types is preferably employed in the encoder 210.
  • the units 215 and 300 of the image encoder 210 may be provided as software, hardware or a combination thereof.
  • the units 215, 300 and 500 may be implemented together in the image encoder 210.
  • a distributed implementation is also possible with some of the units provided elsewhere in the mobile unit.
  • Fig. 12 illustrates a block diagram of another embodiment of the image encoder 210 according to the present invention.
  • This image encoder 210 comprises an image decomposer 215 as the embodiment of Fig. 11 , which is not further discussed.
  • the encoder 210 includes multiple (M, where M is a positive integer larger than one) block encoders 300-1 to 300-M. Each such block encoder 300-1 to 300-M basically corresponds to the block encoder of the image encoder in Fig. 11 .
  • multiple image blocks from the decomposer 215 may be processed (encoded) in parallel, which reduces the total image encoding time.
  • Each block encoder 300-1 to 300-M could comprise an intensity table 500.
  • the intensity tables 500 in the different encoders 300-1 to 300-M could all include identical intensity modifier values.
  • different block encoders could include different tables. In such a case, one or several block encoders could be adapted for a certain image type, whereas other block encoders are adapted for other image type(s).
  • a single intensity table 500 is arranged in the image encoder 210 and connected to all block encoders 300-1 to 300-M.
  • the units 215 and 300-1 to 300-M of the image encoder 210 may be provided as software, hardware or a combination thereof.
  • the units 215, 300-1 to 300-M and 500 may be implemented together in the image encoder 210.
  • a distributed implementation is also possible with some of the units provided elsewhere in the mobile unit.
  • Fig. 13 illustrates a block diagram of an embodiment of a block encoder 300 according to the present invention, such as the block encoder of the image encoder in Fig. 11 or one of the block encoders in the image encoder of Fig. 12 .
  • the encoder 300 comprises a color quantizer 310 that determines a color representation of the colors of the image elements in the image block and quantizes this color representation.
  • the color representation is preferably a 24-bit average color of the image element and is subsequently quantized to a 12-bit color representation, i.e. the color codeword, by the quantizer 310.
  • An intensity quantizer 320 is provided in the block encoder 300 for identifying an intensity modifier set to use for a current image block.
  • the quantizer 320 is preferably configured for selecting this modifier set from an associated intensity table 500.
  • the quantizer 320 then generates an intensity codeword that is associated with the selected modifier set.
  • the encoder 300 further includes an intensity selector 330 that selects, for the image elements in the image block, an intensity modifier from the identified intensity modifier set.
  • the color quantizer 310, intensity quantizer 320 and intensity selector 330 are preferably configured for combined quantization of the color and intensity codeword, as was discussed in detail in the foregoing.
  • the units 310, 320 and 330 of the block encoder 300 may be provided as software, hardware or a combination thereof.
  • the units 310, 320, 330 and 500 may be implemented together in the block encoder 300. Alternatively, a distributed implementation is also possible with some of the units provided elsewhere in the image encoder.
  • Fig. 14 illustrates a block diagram of another embodiment of a block encoder 300 according to the present invention.
  • This block encoder 300 comprises an error estimator 340 for estimating error values for the purpose of selecting intensity modifier set and values, and possibly quantized color value, to use for an image block.
  • This estimator 340 is preferably configured for calculating an error value for a first selection of an intensity modifier set and values (and color in the case of exhaustive encoding) for the image elements in the image block. This first error value is then stored. The error calculation is then repeated for all possible selections of modifier set and values (and color), and after each calculation the estimated error value is compared to the stored error. If it is smaller than the stored value, it replaces the previously stored error.
  • modifier set and values (and color) associated with the error value are also stored. Once all combinations have been tested, the selection resulting in the smallest error is used for generation of intensity (and color) codeword and intensity representations.
  • An intensity modifier set selector 322 and intensity selector 330 selects the modifier set and modifier values that are associated with the smallest error.
  • the remaining units of the block encoder 300 have correspondences in Fig. 13 and are not further discussed.
  • this embodiment of the block encoder 300 could operate according to the combined quantization as described above. In such a case, the color quantizer 310 provides the neighboring quantized colors of the calculated average color of the image block.
  • the error estimator 340 determines error representations for each such neighboring quantized color, e.g. as illustrated in Table 2 or 3, and the neighboring quantized color that results in a smallest error representation is selected as the color codeword for the image block.
  • the units 310, 320, 322, 330 and 340 of the block encoder 300 may be provided as software, hardware or a combination thereof.
  • the units 310, 320, 322, 330, 340 and 500 may be implemented together in the block encoder 300. Alternatively, a distributed implementation is also possible with some of the units provided elsewhere in the image encoder.
  • the quantizer 310 comprises means 312 configured for determining an average of the colors of the image elements in the image block. This average color is preferably a RGB color, but could be any other color format used in image processing. This determined average color is then provided to quantizing means 314, which quantizes the average color.
  • the quantizer 314 is preferably configured for quantizing a 24-bit average RGB color from the color averager 312 into a 12-bit RGB color.
  • the units 312 and 314 of the color quantizer 310 may be provided as software, hardware or a combination thereof.
  • the units 310 and 314 may be implemented together in the color quantizer 310.
  • a distributed implementation is also possible with some of the units provided elsewhere in the block encoder.
  • Fig. 16 illustrates a block diagram of an embodiment of a block decoder 220 according to the present invention.
  • the block decoder 220 preferably comprises a block selector 222 that is adapted for selecting, e.g. from a memory, which encoded image block(s) that should be provided to a block decoder 400 for decoding.
  • the block selector 222 preferably receives input information associated with the encoded image data, e.g. from a header or a rendering engine.
  • An address of an encoded image block having the desired image element(s) is then computed based on the input information. This computed address is preferably dependent upon the image-element (pixel or texel) coordinates within an image.
  • the block selector 222 identifies the encoded image block from the memory. This identified encoded image block is then fetched from the storage and provided to the block decoder 400.
  • the (random) access to image elements of an image block advantageously enables selective decoding of only those portions of an image that are needed.
  • the image can be decoded in any order the data is required. For example, in texture mapping only portions of the texture may be required and these portions will generally be required in a non-sequential order.
  • the image decoding of the present invention can with advantage by applied to process only a portion or section of an image.
  • the selected encoded image block is then forwarded to the block decoder 400.
  • the decoder 400 preferably receives information specifying which image elements of the block that should be decoded. The information could specify that the whole image block, i.e. all image elements therein, should be decoded.
  • the block decoder 400 then generates a decoded representation of the image element(s) in the block.
  • This decoded representation is preferably a P-bit color, where P is the number of bits per image element in the original image, e.g. a 24-bit RGB color.
  • the block decoder 400 preferably comprises an intensity table 500 that is used during the decoding procedure. Alternatively, this intensity table 500 could be provided elsewhere in the image decoder 220. Usage of different intensity tables for different image types, discussed above in connection to Fig. 11 , also applies to the image decoder 220.
  • An optional image composer 224 could be provided in the decoder 220. This composer receives the decoded image elements from the block decoder 400 and composes them to generate a pixel that can be rendered or displayed on a screen. The composer 224 could require several input image elements to generate a single pixel. This image composer 224 could alternatively be provided in the graphic system.
  • the units 222, 224 and 400 of the block decoder 220 may be provided as software, hardware or a combination thereof.
  • the units 222, 224, 400 and 500 may be implemented together in the block decoder 220. Alternatively, a distributed implementation is also possible with some of the units provided elsewhere in the mobile unit.
  • Fig. 17 illustrates a block diagram of another embodiment of an image decoder 220 according to the present invention.
  • the block selector 222 and image composer 224 are similar to corresponding units in Fig. 16 and are not further discussed.
  • the image decoder 220 comprises multiple block decoders 400-1 to 400-Q (Q is a positive integer larger than one). By having access to multiple block decoders 400-1 to 400-Q, the image decoder 220 can process (decode) multiple encoded image blocks in parallel. These multiple block decoders 400-1 to 400-Q allow for parallel processing that increases the processing performance and efficiency of the image decoder 220. For example, one decoded image element is generally sufficient for nearest neighbor interpolation, whereas four (eight) image element are need for bilinear (trilinear) interpolation. Each block decoder 400-1 to 400-Q could comprise an intensity table 500 used for the decoding.
  • a single table 500 is arranged in the image decoder 220 and connected to all block decoders 400-1 to 400-Q.
  • the units 222, 224 and 400-1 to 400-Q of the image decoder 220 may be provided as software, hardware or a combination thereof.
  • the units 222, 224, 400-1 to 400-Q and 500 may be implemented together in the image decoder 220. Alternatively, a distributed implementation is also possible with some of the units provided elsewhere in the mobile unit.
  • Fig. 18 is an illustration of an embodiment of a block decoder 400 according to the present invention.
  • the block decoder 400 comprises means 410 for providing an intensity modifier set from an associated intensity table 500 based on the intensity codeword.
  • This provider 410 could be configured for fetching a first sub-set of modifier values from the intensity table 500 and determining a second sub-set of modifiers based on the first sub-set.
  • a color generator 420 generates a single color representation for all image elements in the image block based on the color codeword. This generator 420 preferably expands the 12-bit color of the codeword into a 24-bit (RGB) color.
  • An intensity modifier selector 430 is arranged for selecting one of the intensity modifier values from the modifier set provided by the means 410.
  • the modifier selector 430 is configured for selecting correct modifier values for the image elements in the encoded image block based on the sequence of intensity representations.
  • the expanded color from the color generator 420 and modifier value from modifier selector 430 are forwarded to an intensity modulator or modifier 440 that modifies the intensity of the color components of the expanded color with the modifier value.
  • the modifier 440 could use a weighted intensity modifier value, with different weights for the different color components.
  • the modifier 440 preferably clamps the components between a maximum and minimum threshold, e.g. between 0 and 255.
  • the units 410, 420, 430 and 440 of the block decoder 400 may be provided as software, hardware or a combination thereof.
  • the units 410, 420, 430, 440 and 500 may be implemented together in the block decoder 400. Alternatively, a distributed implementation is also possible with some of the units provided elsewhere in the image decoder.
  • Fig. 19 schematically illustrates a possible hardware implementation of a block decoder 400 according to the present invention.
  • the input to the block decoder 400 is an encoded block representation 700 comprising a 12-bit color codeword 710 (4 bits for each of the red, green and blue component), a 4-bit intensity codeword 720 and a 16-bit sequence of intensity 730.
  • the color codeword is provided to the color generator 420, which is realized by three bit extenders 422 to 426.
  • a first bit extender 422 receives the 4-bit red component
  • a second 424 and third 426 extender receives the 4-bit green and blue component, respectively.
  • the output from respective extender 422 to 426 is an 8-bit color component. This 8-bit component is simply obtained by multiplying the input component with 17, or multiplying the component with 16 and then adding the component.
  • a modifier selector 430 is implemented as a multiplexor 435.
  • a 3-bit address index is input to this multiplexor 435. Based on the address index, the multiplexor 435 selects which of the eight image elements to decode.
  • the 2-bit intensity representation associated with the selected image element is then forwarded to a table look-up 415.
  • This table look-up corresponds to the modifier set provider 410 and intensity table 500 of Fig. 18 .
  • the look-up 415 fetches the correct intensity modifier value from one of the modifier sets in the table.
  • This 9-bit signed (positive or negative) modifier value is then provided to an intensity modifier 440.
  • the modifier 440 comprises three adders 441 to 443 and three clampers 444 to 446.
  • the modifier value is input to respective adder 441 to 443.
  • a first adder 441 adds the intensity modifier value to the 8-bit red component from bit extender 422.
  • adder 442 and adder 443 adds the modifier value to the 8-bit green and blue component from bit extender 424 and 426, respectively.
  • the adders 441 to 443 can be replaced with other modifying elements, e.g. multipliers or XOR gates.
  • the outputs from the adders 441 to 443 are forwarded to clampers 444 to 446, which clamp the intensity modified color components between 0 and 255.
  • the output from the clampers 444 to 446 is the decompressed or decoded 24-bit color of the image element.
  • Fig. 20 schematically illustrates a possible hardware implementation of the bit extenders 422; 424; 426 of Fig. 19 .
  • These extenders receive a 4-bit (red, green or blue) color component and output an extended corresponding 8-bit color component.
  • the four most significant bits (MSBs) constitute the input 4-bit color component
  • the "fifth MSB” corresponds to the MSB of the input component
  • the "sixth MSB” corresponds to the "second MSB” of the input component
  • the remaining two least significant bits (LSBs) correspond to the two LSBs of the input component.
  • Fig. 21 schematically illustrates a possible hardware implementation of the table look-up 415 of Fig. 19 .
  • the three LSBs of the 4-bit input intensity codewords are input to two multiplexors 411 and 412 for selecting one 7-bit intensity modifier value from eight possible modifier values for each multiplexor 411 and 412. From these 16 intensity modifiers the remaining 48 values could be calculated if employing an intensity table according to Table 1.
  • the selected intensity modifier values from the multiplexors 411 and 412 are input to another multiplexor 413 that selects one of these values based on 1-bit input data (1 bit of the 2-bit intensity representation) from the multiplexor 435 in Fig. 19 .
  • the selected modifier value is then forwarded both to a multiplexor 416 and to negation means 414 that negates the modifier value. Also this negated value is forwarded to the multiplexor 416.
  • This multiplexor 416 selects either the positive 7-bit intensity modifier value or the negated value based on the remaining bit of the intensity representation from the multiplexor 435 in Fig. 19 .
  • the selected (8-bit) modifier value is then brought both to a multiplexor 418 and to a bit-shifter 417 that shifts the modifier value one bit to the left, resulting in a 9-bit intensity modifier (corresponds to a multiplication of the value, in base ten, by two).
  • the multiplexor 418 selects either the 8-bit modifier value or the 9-bit modifier value based on the MSB from the intensity codeword.
  • the result from the selection is the 9-bit intensity modifier value, out of the 64 possible modifier values, to use for a specific image element.
  • Fig. 22 schematically illustrates a possible hardware implementation of the clampers 444; 445; 446 of Fig. 19 .
  • the input to the clamper 444; 445; 446 is a 10-bit intensity-modified color component value.
  • the eight LSBs of this input value are brought a multiplexor 447.
  • the other input to the multiplexor is the maximum threshold value (255; 8 bits).
  • the multiplexor 447 selects either the 8-bit input value or the maximum threshold value based on the second MSB of the intensity modified color component. In other words, if this second MSB is equal to one, the multiplexor 447 outputs the threshold value, otherwise (the second MSB is equal to zero) the 8-bit input value is output to a second multiplexor 448.
  • This second multiplexor 448 compares the output from the first multiplexor 447 with the minimum threshold value (0; 8 bits) based on the MSB of the color component. If this MSB or sign bit is equal to one, the output from the first multiplexor 447 is negative and the minimum threshold value should be selected by the second multiplexor 448. However, if the sign bit is zero the output from the first multiplexor 447 should also be output from the second multiplexor 448.
  • the hardware solution for the block encoder 400 in Fig. 19 is very simple, it basically comprises only three additions, one negation and 12 multiplexors if the bit extenders 422; 424; 426, table lock-up 415 and clampers 444; 445; 446 are implemented according to Fig. 20 , Fig. 21 and Fig. 22 , respectively. This should be compared to decompression using the S3TC scheme [3], which requires up to 42 additions and two multiplexors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
  • Color Television Systems (AREA)
  • Image Processing (AREA)
EP04749156.8A 2003-12-19 2004-07-08 Image processing Active EP1697897B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PL13172504T PL2688042T3 (pl) 2003-12-19 2004-07-08 Przetwarzanie obrazu
EP13172504.6A EP2688042B1 (en) 2003-12-19 2004-07-08 Image processing

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE0303497A SE526226C2 (sv) 2003-12-19 2003-12-19 Bildbehandling
PCT/SE2004/001120 WO2005059836A1 (en) 2003-12-19 2004-07-08 Image processing

Related Child Applications (2)

Application Number Title Priority Date Filing Date
EP13172504.6A Division EP2688042B1 (en) 2003-12-19 2004-07-08 Image processing
EP13172504.6A Division-Into EP2688042B1 (en) 2003-12-19 2004-07-08 Image processing

Publications (2)

Publication Number Publication Date
EP1697897A1 EP1697897A1 (en) 2006-09-06
EP1697897B1 true EP1697897B1 (en) 2014-05-21

Family

ID=30768816

Family Applications (2)

Application Number Title Priority Date Filing Date
EP04749156.8A Active EP1697897B1 (en) 2003-12-19 2004-07-08 Image processing
EP13172504.6A Active EP2688042B1 (en) 2003-12-19 2004-07-08 Image processing

Family Applications After (1)

Application Number Title Priority Date Filing Date
EP13172504.6A Active EP2688042B1 (en) 2003-12-19 2004-07-08 Image processing

Country Status (15)

Country Link
US (1) US7636471B2 (zh)
EP (2) EP1697897B1 (zh)
JP (1) JP4444967B2 (zh)
CN (3) CN100550057C (zh)
AU (1) AU2004298705B2 (zh)
BR (1) BRPI0417576B1 (zh)
CA (1) CA2545414C (zh)
ES (1) ES2716560T3 (zh)
HU (1) HUE042288T2 (zh)
PL (1) PL2688042T3 (zh)
RU (1) RU2317587C1 (zh)
SE (1) SE526226C2 (zh)
TR (1) TR201901009T4 (zh)
WO (1) WO2005059836A1 (zh)
ZA (1) ZA200604184B (zh)

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SE526226C2 (sv) 2003-12-19 2005-08-02 Ericsson Telefon Ab L M Bildbehandling
SE0401850D0 (sv) * 2003-12-19 2004-07-08 Ericsson Telefon Ab L M Image processing
SE0401852D0 (sv) 2003-12-19 2004-07-08 Ericsson Telefon Ab L M Image processing
EP3023940B1 (en) 2005-05-27 2019-02-27 Telefonaktiebolaget LM Ericsson (publ) Weight based image processing
ATE425516T1 (de) * 2005-08-19 2009-03-15 Ericsson Telefon Ab L M Texturkomprimierung auf der basis zweier farbtöne mit modifizierter helligkeit
US7873212B2 (en) 2006-01-24 2011-01-18 Nokia Corporation Compression of images for computer graphics
US7787691B2 (en) * 2006-04-11 2010-08-31 Telefonaktiebolaget Lm Ericsson (Publ) High quality image processing
US8233003B2 (en) 2007-03-12 2012-07-31 Seiko Epson Corporation Image processing device, image processing method, and electronic instrument
JP5157419B2 (ja) * 2007-03-12 2013-03-06 セイコーエプソン株式会社 画像処理装置、画像処理方法及び電子機器
WO2008123822A1 (en) * 2007-04-04 2008-10-16 Telefonaktiebolaget Lm Ericsson (Publ) Vector-based image processing
KR101431185B1 (ko) * 2007-06-22 2014-08-27 삼성전자 주식회사 영상 향상 방법 및 장치, 이를 이용한 영상 처리 시스템
TWI366391B (en) * 2007-06-28 2012-06-11 Novatek Microelectronics Corp Method and circuit for correcting signals and image correcting method and circuit using the same
US8013862B2 (en) * 2007-11-16 2011-09-06 Microsoft Corporation Texture codec
US20100226568A1 (en) * 2009-03-09 2010-09-09 Smith Micro Software, Inc. Effective color modeling method for predictive image compression
CN102571270A (zh) * 2009-03-27 2012-07-11 华为技术有限公司 解码方法及装置
KR20100136890A (ko) * 2009-06-19 2010-12-29 삼성전자주식회사 컨텍스트 기반의 산술 부호화 장치 및 방법과 산술 복호화 장치 및 방법
JP2011066720A (ja) * 2009-09-17 2011-03-31 Canon Inc 画像処理装置、制御方法、及びプログラム
US8655063B2 (en) * 2009-11-24 2014-02-18 Telefonaktiebolaget L M Ericsson (Publ) Decoding system and method operable on encoded texture element blocks
US8659616B2 (en) * 2010-02-18 2014-02-25 Nvidia Corporation System, method, and computer program product for rendering pixels with at least one semi-transparent surface
CN102244779B (zh) * 2010-05-11 2014-07-30 联想(北京)有限公司 数据的发送和接收方法及设备、数据传输系统
WO2012050489A1 (en) * 2010-06-18 2012-04-19 Telefonaktiebolaget Lm Ericsson (Publ) Texture compression and decompression
CN103098466B (zh) 2010-09-13 2016-08-17 索尼电脑娱乐公司 图像处理装置和图像处理方法
KR101834934B1 (ko) * 2010-12-03 2018-03-08 코닌클리케 필립스 엔.브이. ３ｄ 이미지 데이터의 전송
CN103404138B (zh) * 2011-01-05 2017-01-18 皇家飞利浦电子股份有限公司 保留ppg相关信息的视频编码与解码设备及方法
BR112013017072A2 (pt) * 2011-01-05 2018-02-14 Koninl Philips Electronics Nv dispositivo de codificação de vídeo para codificar dados de vídeo, método de codificação de vídeo para codificar dados de vídeo, dispositivo de decodificação de vídeo para decodificar um fluxo de vídeo, método de decodificação de vídeo para decodificar um fluxo de vídeo codificado, sistema de codificação de vídeo para codificar e decodificar dados de vídeo e programa de computador.
GB2487717B (en) * 2011-01-25 2017-06-21 Advanced Risc Mach Ltd Image encoding method
EP2695140A1 (en) 2011-04-04 2014-02-12 Telefonaktiebolaget LM Ericsson (PUBL) A method and a processor for texture compression
WO2012146320A1 (en) * 2011-04-29 2012-11-01 Telefonaktiebolaget L M Ericsson (Publ) Encoder, decoder and methods thereof for texture compression
JP5903597B2 (ja) 2011-07-13 2016-04-13 パナソニックＩｐマネジメント株式会社 画像圧縮装置、画像伸長装置および画像処理装置
JP6003049B2 (ja) * 2011-11-30 2016-10-05 富士通株式会社 情報処理装置、画像送信方法及び画像送信プログラム
JP5915131B2 (ja) * 2011-12-08 2016-05-11 ソニー株式会社 画像処理装置、画像処理方法及びプログラム
US9106936B2 (en) * 2012-01-25 2015-08-11 Altera Corporation Raw format image data processing
KR102091137B1 (ko) * 2012-07-17 2020-03-20 삼성전자주식회사 영상 제공 시스템 및 방법
WO2014014238A1 (en) 2012-07-17 2014-01-23 Samsung Electronics Co., Ltd. System and method for providing image
US10147202B2 (en) * 2013-03-15 2018-12-04 Arm Limited Methods of and apparatus for encoding and decoding data
US9716871B2 (en) * 2014-09-30 2017-07-25 Apple Inc. YUV signal generation for high dynamic range video content
CN104899838B (zh) * 2015-05-29 2017-11-28 小米科技有限责任公司 调整图像的方法及装置
US10645418B2 (en) * 2018-04-13 2020-05-05 Google Llc Morphological anti-ringing filter for lossy image compression
CN113076050B (zh) * 2021-04-25 2022-05-20 深圳创维－Rgb电子有限公司 基于内存拷贝的人机交互方法、装置、电视机及存储介质

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6451888A (en) * 1987-08-24 1989-02-28 Sharp Kk Picture processor
CA1315392C (en) * 1988-11-18 1993-03-30 Taejeong Kim Side-match and overlap-match vector quantizers for images
US5204665A (en) * 1990-05-02 1993-04-20 Xerox Corporation Color editing with simple encoded images
DE4335143A1 (de) 1993-10-15 1995-04-20 Hell Ag Linotype Verfahren und Einrichtung zur Umsetzung von Farbwerten
RU2105351C1 (ru) * 1995-02-06 1998-02-20 Товарищество с ограниченной ответственностью "Информ Инжиниринг" Способ анализа изображения текстуры объекта
US5742892A (en) * 1995-04-18 1998-04-21 Sun Microsystems, Inc. Decoder for a software-implemented end-to-end scalable video delivery system
US5793371A (en) 1995-08-04 1998-08-11 Sun Microsystems, Inc. Method and apparatus for geometric compression of three-dimensional graphics data
US6404923B1 (en) * 1996-03-29 2002-06-11 Microsoft Corporation Table-based low-level image classification and compression system
US5748904A (en) * 1996-09-13 1998-05-05 Silicon Integrated Systems Corp. Method and system for segment encoded graphic data compression
US5926647A (en) 1996-10-11 1999-07-20 Divicom Inc. Processing system with dynamic alteration of a color look-up table
US5956431A (en) * 1997-10-02 1999-09-21 S3 Incorporated System and method for fixed-rate block-based image compression with inferred pixel values
US6658146B1 (en) * 1997-10-02 2003-12-02 S3 Graphics Co., Ltd. Fixed-rate block-based image compression with inferred pixel values
JPH11313219A (ja) 1998-01-20 1999-11-09 Fujitsu Ltd カラーデータ変換方法
WO2000002393A1 (en) 1998-07-03 2000-01-13 Hudson Co., Ltd. Image coding/decoding method and recorded medium on which program is recorded
US6636222B1 (en) 1999-11-09 2003-10-21 Broadcom Corporation Video and graphics system with an MPEG video decoder for concurrent multi-row decoding
WO2000060479A1 (en) * 1999-03-19 2000-10-12 Microsoft Corporation Methods and apparatus for generating and representing luminance intensity values
US7251360B2 (en) 2003-02-14 2007-07-31 Canon Kabushiki Kaisha Look-up table generation method, computer program, and imaging apparatus
SE0401850D0 (sv) 2003-12-19 2004-07-08 Ericsson Telefon Ab L M Image processing
SE526226C2 (sv) 2003-12-19 2005-08-02 Ericsson Telefon Ab L M Bildbehandling
SE0401852D0 (sv) 2003-12-19 2004-07-08 Ericsson Telefon Ab L M Image processing

Also Published As

Publication number Publication date
EP1697897A1 (en) 2006-09-06
CA2545414C (en) 2014-08-26
RU2317587C1 (ru) 2008-02-20
JP2007515124A (ja) 2007-06-07
CN100550057C (zh) 2009-10-14
US7636471B2 (en) 2009-12-22
BRPI0417576B1 (pt) 2017-04-18
TR201901009T4 (tr) 2019-02-21
US20070140554A1 (en) 2007-06-21
SE0303497L (sv) 2005-06-20
CN1898700A (zh) 2007-01-17
ZA200604184B (en) 2007-11-28
AU2004298705A1 (en) 2005-06-30
BRPI0417576A (pt) 2007-03-20
EP2688042B1 (en) 2019-01-09
HUE042288T2 (hu) 2019-06-28
ES2716560T3 (es) 2019-06-13
CN1898699A (zh) 2007-01-17
CN100498838C (zh) 2009-06-10
CA2545414A1 (en) 2005-06-30
CN100504926C (zh) 2009-06-24
CN1918602A (zh) 2007-02-21
EP2688042A1 (en) 2014-01-22
PL2688042T3 (pl) 2019-06-28
JP4444967B2 (ja) 2010-03-31
WO2005059836A1 (en) 2005-06-30
SE526226C2 (sv) 2005-08-02
AU2004298705B2 (en) 2009-08-20
SE0303497D0 (sv) 2003-12-19

Similar Documents

Publication Publication Date Title
EP1697897B1 (en) Image processing
RU2407223C2 (ru) Сжатие текстуры на основании двух оттенков с модифицированной яркостью
EP1697900B1 (en) Multi-mode image processing
EP1697898B1 (en) Alpha image processing
EP2005393B1 (en) High quality image processing
US9466125B2 (en) Weight based image processing
US7751630B2 (en) Multi-mode image processing

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20060719

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20110705

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20140109

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 19/60 20140101ALI20140416BHEP

Ipc: G06T 9/00 20060101AFI20140416BHEP

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 669932

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140615

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602004045158

Country of ref document: DE

Effective date: 20140703

REG Reference to a national code

Ref country code: CH

Ref legal event code: NV

Representative=s name: MARKS AND CLERK (LUXEMBOURG) LLP, CH

RAP2 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 669932

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140521

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140822

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140922

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602004045158

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140708

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

26N No opposition filed

Effective date: 20150224

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602004045158

Country of ref document: DE

Effective date: 20150224

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140708

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 13

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140521

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20040708

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 15

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230523

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20230726

Year of fee payment: 20

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230727

Year of fee payment: 20

Ref country code: FI

Payment date: 20230725

Year of fee payment: 20

Ref country code: CH

Payment date: 20230802

Year of fee payment: 20

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230725

Year of fee payment: 20

Ref country code: DE

Payment date: 20230727

Year of fee payment: 20