EP1779362B1 - Rapid image rendering on dual-modulator displays - Google Patents

Rapid image rendering on dual-modulator displays Download PDF

Info

Publication number
EP1779362B1
EP1779362B1 EP05748546.8A EP05748546A EP1779362B1 EP 1779362 B1 EP1779362 B1 EP 1779362B1 EP 05748546 A EP05748546 A EP 05748546A EP 1779362 B1 EP1779362 B1 EP 1779362B1
Authority
EP
European Patent Office
Prior art keywords
luminance pattern
effective luminance
point spread
spread function
light source
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP05748546.8A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP1779362A4 (en
EP1779362A1 (en
Inventor
Lorne A. Whitehead
Helge Seetzen
Wolfgang Heidrich
Gregory John Ward
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dolby Laboratories Licensing Corp
Original Assignee
Dolby Laboratories Licensing Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dolby Laboratories Licensing Corp filed Critical Dolby Laboratories Licensing Corp
Publication of EP1779362A1 publication Critical patent/EP1779362A1/en
Publication of EP1779362A4 publication Critical patent/EP1779362A4/en
Application granted granted Critical
Publication of EP1779362B1 publication Critical patent/EP1779362B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09F—DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS
    • G09F9/00—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements
    • G09F9/30—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements in which the desired character or characters are formed by combining individual elements
    • G09F9/33—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements in which the desired character or characters are formed by combining individual elements being semiconductor devices, e.g. diodes
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09F—DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS
    • G09F9/00—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements
    • G09F9/30—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements in which the desired character or characters are formed by combining individual elements
    • G09F9/35—Indicating arrangements for variable information in which the information is built-up on a support by selection or combination of individual elements in which the desired character or characters are formed by combining individual elements being liquid crystals
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • G09G3/22—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters using controlled light sources
    • G09G3/30—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters using controlled light sources using electroluminescent panels
    • G09G3/32—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters using controlled light sources using electroluminescent panels semiconductive, e.g. using light-emitting diodes [LED]
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • G09G3/34—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters by control of light from an independent source
    • G09G3/3406—Control of illumination source
    • G09G3/342—Control of illumination source using several illumination sources separately controlled corresponding to different display panel areas, e.g. along one dimension such as lines
    • G09G3/3426—Control of illumination source using several illumination sources separately controlled corresponding to different display panel areas, e.g. along one dimension such as lines the different display panel areas being distributed in two dimensions, e.g. matrix
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/06—Adjustment of display parameters
    • G09G2320/0626—Adjustment of display parameters for control of overall brightness
    • G09G2320/0646—Modulation of illumination source brightness and image signal correlated to each other
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • G09G3/34—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters by control of light from an independent source
    • G09G3/3406—Control of illumination source
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • G09G3/34—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters by control of light from an independent source
    • G09G3/36—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters by control of light from an independent source using liquid crystals
    • G09G3/3611—Control of matrices with row and column drivers

Definitions

  • This invention pertains to systems and methods for displaying images on displays of the type that have two modulators.
  • a first modulator produces a light pattern and a second modulator modulates the light pattern produced by the first modulator to yield an image.
  • the modulated light source layer is driven to produce a comparatively low-resolution representation of an image.
  • the low-resolution representation is modulated by the display layer to provide a higher resolution image which can be viewed by an observer.
  • the light source layer may comprise a matrix of actively modulated light sources, such as light emitting diodes (LEDs).
  • the display layer which is positioned and aligned in front of the light source layer, may be a liquid crystal display (LCD).
  • the two layers have different spatial resolutions (e.g. the light source layer's resolution may be about 0.1 % that of the display layer) then both software correction methods and psychological effects (such as veiling luminance) prevent the viewer from noticing the resolution mismatch.
  • the two layers have different spatial resolutions (e.g. the light source layer's resolution may be about 0.1 % that of the display layer) then both software correction methods and psychological effects (such as veiling luminance) prevent the viewer from noticing the resolution mismatch.
  • US 2003/0090455 A1 discloses a method for increasing a dynamic range of a backlit display and a corresponding backlit display. A luminance of a light source illuminating a displayed pixel is varied in response to an intensity value of this pixel.
  • LCD computer displays and televisions are commercially available. Such displays and televisions include circuitry for controlling the amount of light transmitted by individual pixels in an LCD panel.
  • the task of deriving driving from image data signals to control a light source layer and display layer can be computationally expensive. Deriving such signals can be executed by a processor of a computer's video/graphics card, or by some other appropriate processor integral to a computer, to the display itself or to a secondary device.
  • the task of deriving from image data signals to control a light source layer and display layer can be computationally expensive. Deriving such signals can be executed by a processor of a computer's video/graphics card, or by some other appropriate processor integral to a computer, to the display itself or to a secondary device. Performance limitations of the processor can undesirably limit the rate at which successive image frames can be displayed. For example, if the processor is not powerful enough to process incoming video data at the frame rate of the video data then an observer may detect small pauses between successive frames of a video image such as a movie. This can distract the observer and negatively affecting the observer's image viewing experience.
  • the invention may be applied in a wide range of applications wherein an image is displayed by producing a light pattern that is determined at least in part by image data, and modulating the light pattern to yield an image.
  • the light pattern may be produced by any suitable apparatus. Some examples include:
  • rendering image frames or a frame set for display on an LED/LCD layer display entails the following computational steps:
  • the point spread function of each LED in an LED layer is determined by the geometry of the LED.
  • a simple technique for determining an LED layer's total effective luminance pattern is to initially multiply each LED's point spread function (specifically, the point spread function of the light which is emitted by the LED and passes through all optical structures between the LED and LCD layers) by a selected LED driving value and by an appropriate scaling parameter to obtain the LED's effective luminance contribution, for that driving value, to each pixel on the LCD layer.
  • the luminance contributions of every LED in the LED layer can be determined and summed to obtain the total effective luminance pattern, on the LCD layer, that will be produced when the selected driving values are applied to the LED layer.
  • these multiplication and addition operations are very computationally expensive (i.e. time consuming), because the effective luminance pattern must be determined to the same spatial resolution as the LCD layer in order to facilitate the division operation of step 4 above.
  • the computational expense is especially great if the LED point spread function has a very wide "support.”
  • the "support" of an LED point spread function is the number of LCD pixels that are illuminated in a non-negligible amount by an LED.
  • the support can be specified in terms of a radius, measured in LCD layer pixels, at which the LED point spread function becomes so small that is perceptually irrelevant to an observer.
  • the support corresponds to a number of LCD pixels that are illuminated in a significant amount by each LED.
  • each LED has a point spread function having a support of 150 LCD pixels then each pixel in the center portion of the LCD layer will be illuminated by light from approximately 35 of the LEDs. Calculation of the effective luminance pattern for this example accordingly requires 35 operations for each pixel of the LCD layer, in order to account for the light contributed to each pixel by each relevant LED. Where the LCD layer has a high spatial resolution, this is very computationally expensive (i.e. time consuming).
  • the time required to determine the effective luminance pattern produced on the LCD can be reduced by computing the effective luminance pattern at a reduced spatial resolution that is lower than that of the high resolution image which is to appear on the LCD layer. This is feasible because the point spread functions of individual light sources are generally smoothly varying. Therefore, the effective luminance pattern will be relatively slowly varying at the resolution of the LCD. It is accordingly possible to compute the effective luminance pattern at a lower resolution and then to scale the effective luminance pattern up to a desired higher resolution, without introducing significant artifacts.
  • the scaling may be done using suitable linear, Gaussian or other interpolation techniques. Such spatial resolution reduction yields an approximately linear decrease in the computational cost of establishing the effective luminance pattern.
  • Many available interpolation methods that can be used to scale up an effective luminance pattern computed at a lower resolution are computationally inexpensive as compared to the computational cost of computing the effective luminance pattern at the resolution of the LCD or other second light modulator.
  • a 10-times resolution reduction in both the width and height directions yields an approximate 100-times reduction in computational cost. This is because the total number of pixels in the reduced resolution image is 100-times fewer than the total number of pixels in the high resolution image which is to appear on the LCD layer. Each pixel in the reduced resolution image still receives light from 35 LEDs, necessitating 35 computational operations per pixel-but those operations are applied to 100-times fewer pixels in comparison to a case in which the computations are performed separately for every pixel in the actual high resolution image which is to appear on the LCD layer.
  • the computational cost of image rendering can also be reduced by decomposing the point spread function of each light source (e.g. each LED) into several components (e.g. by performing a Gaussian decomposition) in such a way that the recombination of all of the components yields the original point spread function.
  • An effective luminance pattern can then be determined separately for each component. Once an effective luminance pattern has been determined for each component, those effective luminance patterns can be combined to produce a total effective luminance pattern. The combination may be made by summing, for example.
  • Computing the effective luminance patterns contributed by the components may be performed at the resolution of the LCD layer or at a reduced resolution, as described above.
  • a speed benefit is attained even if the effective luminance pattern for each component is computed at the resolution of the LCD layer since hardware components specially adapted to perform rapid computations based upon standard point spread functions (e.g. Gaussian point spread functions) are commercially available. Such hardware components are not normally commercially available for the typically non-standard point spread function of the actual LEDs in the display's LED layer-necessitating resort to considerably slower computational techniques using general purpose processors.
  • standard point spread functions e.g. Gaussian point spread functions
  • Figure 1 depicts (solid line) an example LED point spread function having a steep central portion 10 and a wide tail portion 12.
  • the actual point spread function can be decomposed into a narrow base Gaussian component 14A and a wide base Gaussian component 14B, as depicted.
  • the wide base Gaussian component 14B (dotted line) contributes relatively little image intensity, in comparison to narrow base Gaussian segment 14A (dashed line). Further, wide base Gaussian component 14B is more slowly varying than narrow base Gaussian component 14A . Accordingly, an effective luminance pattern for narrow base Gaussian component 14A may be determined at a moderately high spatial resolution while an effective luminance pattern for the wide base Gaussian component 14B can be computed at a significantly lower spatial resolution. This preserves a substantial portion of the image intensity information contained in narrow base Gaussian component 14A and is still relatively fast since the effective support of the narrow base Gaussian segment is small and thus few LCD pixels are covered by that component. By contrast, since wide base Gaussian component 14B contains relatively little image intensity information, that component can be processed relatively quickly at low resolution without substantially degrading the resolution of the total effective luminance pattern produced by combining the patterns derived for each component.
  • Image data is typically provided in 16-bit word form.
  • High-end (i.e. more expensive) graphic processors typically perform computations in the 16-bit domain.
  • Such processors may have dedicated 16-bit or floating point arithmetic units that can perform 16-bit operations quickly.
  • the need for a high-end processor capable of performing 16-bit operations quickly can be alleviated by computing the effective luminance pattern in the 8-bit domain. Such computations can be performed reasonably quickly by less expensive processors.
  • Each LED's point spread function is a two dimensional function of intensity versus distance relative to the center of the LED.
  • Such a point spread function may be characterized by a plurality of 16-bit data words. Where the point spread function is represented by a look up table, many 16-bit values are required to define the point spread function; for example, one value may be provided for every LCD pixel lying on or within a circle centered on the LED and having a radius corresponding to the support of the point spread function.
  • the 8-bit values are preferably extracted only after all necessary scaling and manipulation operations have been applied to the input 16-bit data.
  • Figure 2A depicts a 16-bit point spread function;
  • Figures 2B and 2C respectively depict the 8-bit high and low byte components of the Figure 2A 16-bit point spread function.
  • a 16-bit data word is capable of representing integer values from 2 0 -1 to 2 16 -1 (i.e. from 0 to 65535).
  • An 8-bit byte is capable of representing integer values from 2 0 -1 to 2 8 -1 (i.e. from 0 to 255).
  • the "support" (as previously defined) of a point spread function characterized by an 8-bit high byte component is much smaller (narrower) than the support of the point spread function as a whole. This is because the 8-bit high byte component reaches the lowest value (zero) of its 255 possible values, when the 16-bit data word characterizing the point spread function as a whole reaches the value 255 out of its range of 65535 possible values.
  • the remaining 255 values are provided by the low byte component with the high byte component's value equal to zero.
  • the effective luminance pattern corresponding to the narrow base 8-bit high byte component can accordingly be rapidly determined, without substantial loss of image intensity information.
  • the resolution reduction and/or other techniques described above may be used to further speed up the determination of the effective luminance pattern for the 8-bit high byte component.
  • the support of a point spread function characterized by an 8-bit low byte component is comparatively wide. Specifically, although the 8-bit low byte component has only 255 possible values, those values decrease from 255 to 0 (out of 65535 values for the point spread function as a whole) and those 255 values correspond to the 255 lowest intensity levels (i.e. levels at which the value of the high byte component is equal to zero). Those 255 levels represent the valued of the point spread function in its peripheral parts.
  • the low byte component can be separated into two regions.
  • a central region lying within the boundary on which the point spread function characterized by the high byte component reaches zero.
  • the low-byte component typically varies in an irregular saw-tooth pattern (as depicted in Figure 3 ) if the original 16-bit point spread function is reasonably smooth. This is because, in the central region, the portion of the point spread function characterized by the low byte component augments the portion of the point spread function characterized by the high byte component.
  • the value of the high byte component's contribution remains at 39, while the value of the low byte component's contribution smoothly decreases from 255 to 0-the point at which the original 16-bit point spread function has the value 9984 (i.e. 39*256+0).
  • the value of the high byte component's contribution to the point spread function then changes smoothly from 39 to 38, but that change is accompanied by an abrupt change (from 0 to 255) in the value of the low byte component's contribution to the point spread function.
  • the contributions from the low-byte component of the point spread function can be processed differently in these two regions (i.e. the regions inside and outside the radius R ) to avoid unwanted artifacts.
  • the effective luminance pattern for that region is preferably determined using the same relatively high resolution used to determine the effective luminance pattern for the high byte component's contribution to the point spread function, as previously described.
  • the effective luminance pattern for the region outside the radius R can be determined using a much lower resolution, without substantial loss of image intensity information.
  • the results are individually up-sampled to match the resolution of the LCD layer, then recombined with appropriate scaling factors being applied.
  • Recombination typically involves summation of the values for the two low byte component regions and the value for the high byte component, after the value for the high byte component has been multiplied by 256.
  • an effective luminance pattern value is determined using a resolution lower than the resolution of the LCD layer, it is necessary to up-sample that value to match the resolution of the LCD layer.
  • Interpolation techniques for up-sampling low resolution images into high resolution images are well known, with both linear and Gaussian based techniques being common. Although such prior art techniques can be used in conjunction with the above described techniques, accuracy, or speed, or both may be improved by utilizing an interpolation technique which is optimized for a particular display configuration. Optimization facilities higher resolution image compression, minimizes introduction of unwanted interpolation artifacts, and reduces the image rendering time. In extreme cases, an interpolation technique can be used to reduce the resolution of the effective luminance pattern resolution to match the resolution of the LED layer.
  • Prior art interpolation techniques are often restricted to use with specific pre-interpolation data, or to use with specific interpolation functions.
  • the interpolation techniques used to match the resolution of the effective luminance pattern to that of the LCD display do not need to satisfy such restrictions, provided convolution of the pre-interpolation data with the selected interpolation function will yield an effective luminance pattern having adequate similarity to the actual effective luminance pattern.
  • the required degree of similarity depends on the display application. Different applications require different degrees of similarity-in some applications relatively small deviations may unacceptably distract an observer, whereas larger deviations may be tolerable in other applications (such as applications involving television or computer game images in which relatively large deviations nonetheless yield images of quality acceptable to most observers). Consequently, it is not necessary to apply the interpolation technique directly to the actual LED driving values or to the actual LED point spread function.
  • Figure 5 depicts the result obtained by using an iteratively-derived interpolation technique to reduce the resolution of the effective luminance pattern to match the resolution of the LED layer.
  • the pixel values at the LED layer's resolution are not the LED driving values-they are the luminance values of the effective luminance pattern before interpolation.
  • the interpolation function can be determined using standard iteration methods and a random starting condition. As seen in Figure 5 , convolution of the iteratively-derived interpolation function with the effective luminance pattern values yields results which are reasonably close to the actual effective luminance pattern.
  • interpolation techniques There need not be any correlation between the interpolation function and the LEDs' point spread function, the LED driving values, or any other characteristic of the display, provided the selected interpolation function and the input parameters selected for use with that function yields a result reasonably close to the actual effective luminance pattern.
  • Figures 6 to show some example embodiments of the invention.
  • Figure 6 shows a display 30 comprising a modulated light source layer 32 and a display layer 34.
  • Light source layer 32 may comprise, for example:
  • Display layer 34 comprises a light modulator that further spatially modulates the intensity of light incident on display layer 34 from light source layer 32 .
  • Display layer 34 may comprise an LCD panel or other transmission-type light modulator, for example.
  • Display layer 34 typically has a resolution higher than a resolution of light source layer 32.
  • Optical structures 36 suitable for carrying light from light source layer 32 to display layer 34 may be provided between light source layer 32 and display layer 34.
  • Optical structures 36 may comprise elements such as open space, light diffusers, collimators, and the like.
  • a controller 40 comprising a data processor 42 and suitable interface electronics 44A for controlling light source layer 32 and 44B for controlling display layer 34 receives image data 46 specifying images to be displayed on display 30. Controller 40 drives the light emitters (e.g. LEDs 33) of light source layer 34 and the pixels 35 of display layer 34 to produce the desired image for viewing by a person or persons.
  • a program store 46 accessible to processor 42 contains software instructions that, when executed by processor 42 cause processor 42 to execute a method as described herein.
  • Controller 40 may comprise a suitably programmed computer having appropriate software/hardware interfaces for controlling light source layer 32 and display layer 34 to display an image specified by image data 48 .
  • Figure 7 shows a method 50 for displaying image data on a display of the general type shown in Figure 6 .
  • Method 50 begins by receiving image data 48 at block 52.
  • first driving signals for light source layer 32 are derived from image data 48. Suitable known methods may be applied to obtain the first driving signals in block 54.
  • method 50 computes an effective luminance pattern.
  • the effective luminance pattern may be computed from the first driving signals and known point spread functions for the light sources of light source layer 32.
  • Block 56 computes the effective luminance pattern at a resolution that is lower than a resolution of display layer 34.
  • block 56 may compute the effective luminance pattern at a resolution that is a factor of 4 or more smaller in each dimension (in some embodiments a factor in the range of 4 to 16 smaller in each dimension) than the resolution of display layer 34.
  • the effective luminance pattern computed in block 58 is upsampled to the resolution of display layer 34. This may be done through the use of any suitable interpolation technique for example.
  • second driving signals for the display layer are determined from the upsampled effective luminance pattern and the image data. The second driving signals may also take into account known characteristics of the display layer and any desired image corrections, colour corrections or the like.
  • the first driving signal obtained in block 54 is applied to the light source layer and the second driving signals of block 62 are applied to the display layer to display an image for viewing.
  • Figure 8 shows a method 70 for computing an effective luminance pattern.
  • Method 70 may be applied within block 56 of method 50 or may be used in other contexts.
  • Method 70 begins by computing an ELP for each component of the point spread function for the light sources of light source layer 32 (blocks 72A , 72B and 72C - collectively blocks 72).
  • Blocks 72 may be performed in any sequence or may be performed in parallel with one another.
  • Figure 8 shows three PSF components 73A, 73B and 73C and three corresponding blocks 72. The method could be practised with two or more PSF components 73.
  • Each of blocks 72 may comprise, for each light source of light source layer 32, multiplying values that define a component of the point spread function by a value representing the intensity of the light source.
  • the effective luminance patters determined in blocks 72 are combined, for example by summing, to yield an overall estimate of the effective luminance pattern that would be produced by applying the first driving signals to light source layer 32.
  • Figure 9 illustrates a method 80 that may be applied for computing effective luminance patterns.
  • Method 80 may be applied to:
  • Method 80 begins in block 82 with data characterizing a point spread function (or a PSF component) for a light source of light source layer 32 and data indicative of how intensely the light source will operate under the control of the first driving signals. Method 80 combines these values (e.g. by multiplying them together) to obtain a set of values characterizing the contribution of the light source to the effective luminance pattern at various spatial locations.
  • a point spread function or a PSF component
  • Block 84 obtains high-order and low-order components of the resulting values.
  • the resulting values are 16-bit words
  • the high-order component is an 8-bit byte
  • the low-order component is an 8-bit byte.
  • Contributions to the ELP are determined separately for the high-order and low-order components in blocks 86 and 88.
  • the area of support for which values are included in the high-order contribution of 86 is typically significantly smaller than the area of support for which values are included in the low-order contribution of block 88.
  • Block 88 typically computes the low-order contribution for points located within the area of support of the high-order contribution (block 90) separately than for points located outside of the area of support of the high-order contribution (block 92).
  • Blocks 86, 90 and 92 may be performed in any order or simultaneously.
  • blocks 86, 90 and 92 the contributions from blocks 86, 90 and 92 are combined to yield an overall ELP.
  • the computations in blocks 86 90 and 92 may be performed primarily or entirely in the 8-bit domain (i.e. using 8-bit operations on 8-bit operands) in the case that the high-order and low-order components are 8-bit bytes or smaller.
  • Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention.
  • processors in a computer or other display controller may implement the methods of Figures 7 , 8 or 9 by executing software instructions in a program memory accessible to the processors.
  • the invention may also be provided in the form of a program product.
  • the program product may comprise any medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention.
  • Program products according to the invention may be in any of a wide variety of forms.
  • the program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like or transmission-type media such as digital or analog communication links.
  • the computer-readable signals on the program product may optionally be compressed or encrypted.
  • a component e.g. a member, part, assembly, device, processor, controller, collimator, circuit, etc.
  • reference to that component should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.
EP05748546.8A 2004-07-27 2005-05-27 Rapid image rendering on dual-modulator displays Active EP1779362B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US59182904P 2004-07-27 2004-07-27
PCT/CA2005/000807 WO2006010244A1 (en) 2004-07-27 2005-05-27 Rapid image rendering on dual-modulator displays

Publications (3)

Publication Number Publication Date
EP1779362A1 EP1779362A1 (en) 2007-05-02
EP1779362A4 EP1779362A4 (en) 2009-08-12
EP1779362B1 true EP1779362B1 (en) 2016-04-20

Family

ID=35785864

Family Applications (1)

Application Number Title Priority Date Filing Date
EP05748546.8A Active EP1779362B1 (en) 2004-07-27 2005-05-27 Rapid image rendering on dual-modulator displays

Country Status (9)

Country Link
EP (1) EP1779362B1 (ja)
JP (3) JP5419352B2 (ja)
KR (1) KR101121131B1 (ja)
CN (2) CN100507988C (ja)
CA (4) CA3043550C (ja)
DK (1) DK1779362T3 (ja)
ES (1) ES2575929T3 (ja)
HK (1) HK1108756A1 (ja)
WO (1) WO2006010244A1 (ja)

Families Citing this family (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2267520B1 (en) 2001-02-27 2018-07-25 Dolby Laboratories Licensing Corporation A method and device for displaying an image
ES2675880T3 (es) 2002-03-13 2018-07-13 Dolby Laboratories Licensing Corporation Compensación de falla de elemento emisor de luz en un monitor
US8687271B2 (en) 2002-03-13 2014-04-01 Dolby Laboratories Licensing Corporation N-modulation displays and related methods
JP2006330400A (ja) * 2005-05-26 2006-12-07 Sony Corp 透過型カラー液晶表示装置
WO2008094153A1 (en) 2007-01-31 2008-08-07 Dolby Laboratories Licensing Corporation Multiple modulator displays and related methods
EP2860721B1 (en) 2007-02-01 2021-09-01 Dolby Laboratories Licensing Corporation Calibration of displays having spatially-variable backlight
US8233738B2 (en) 2007-07-30 2012-07-31 Dolby Laboratories Licensing Corporation Enhancing dynamic ranges of images
US8135230B2 (en) 2007-07-30 2012-03-13 Dolby Laboratories Licensing Corporation Enhancing dynamic ranges of images
KR101320018B1 (ko) * 2007-12-04 2013-10-18 삼성디스플레이 주식회사 광원 모듈과 이의 구동 방법 및 이를 포함하는 표시 장치
US20090322800A1 (en) 2008-06-25 2009-12-31 Dolby Laboratories Licensing Corporation Method and apparatus in various embodiments for hdr implementation in display devices
KR101256806B1 (ko) * 2008-09-30 2013-04-22 돌비 레버러토리즈 라이쎈싱 코오포레이션 고 휘도 및 고 동적 범위 디스플레이들에 대한 이미지 프로세싱에서 적응형 감마를 적용하는 시스템들 및 방법들
KR101225574B1 (ko) 2008-10-14 2013-01-25 돌비 레버러토리즈 라이쎈싱 코오포레이션 하이 다이나믹 레인지 이미지를 위한 광의 공간 변조를 결정하기 위한 감소된 해상도에서 백라이트 시뮬레이션
WO2010085505A1 (en) 2009-01-21 2010-07-29 Dolby Laboratories Licensing Corporation Apparatus and methods for color displays
US20100214282A1 (en) 2009-02-24 2010-08-26 Dolby Laboratories Licensing Corporation Apparatus for providing light source modulation in dual modulator displays
CN105842916A (zh) 2009-04-15 2016-08-10 杜比实验室特许公司 显示器、显示方法及生成光的方法
KR101612455B1 (ko) * 2009-04-16 2016-04-15 삼성디스플레이 주식회사 화소 데이터의 보정 방법 및 이를 수행하기 위한 표시 장치
US9692946B2 (en) 2009-06-29 2017-06-27 Dolby Laboratories Licensing Corporation System and method for backlight and LCD adjustment
WO2011005792A1 (en) 2009-07-07 2011-01-13 Dolby Laboratories Licensing Corporation Edge-lit local dimming displays, display components and related methods
US8890910B2 (en) 2009-07-22 2014-11-18 Dolby Laboratories Licensing Corporation Image display based on multiple brightness indicators
WO2011011446A1 (en) 2009-07-22 2011-01-27 Dolby Laboratories Licensing Corporation Control of array of two-dimensional imaging elements in light modulating displays
WO2011011250A1 (en) 2009-07-24 2011-01-27 Dolby Laboratories Licensing Corporation Image control for displays
EP2477182A4 (en) * 2009-09-09 2013-05-08 Sharp Kk DISPLAY DEVICE
US9341887B2 (en) 2009-09-11 2016-05-17 Dolby Laboratories Licensing Corporation Displays with a backlight incorporating reflecting layer
WO2011056430A1 (en) 2009-10-28 2011-05-12 Dolby Laboratories Licensing Corporation Stereoscopic dual modulator display device using full color anaglyph
KR101267304B1 (ko) 2010-02-22 2013-05-27 돌비 레버러토리즈 라이쎈싱 코오포레이션 듀얼 변조 디스플레이들에서의 전력 소모를 감소시키기 위한 방법들 및 시스템들
US9135864B2 (en) 2010-05-14 2015-09-15 Dolby Laboratories Licensing Corporation Systems and methods for accurately representing high contrast imagery on high dynamic range display systems
WO2012122104A2 (en) 2011-03-09 2012-09-13 Dolby Laboratories Licensing Corporation High contrast grayscale and color displays
US9576555B2 (en) 2010-06-21 2017-02-21 Dolby Laboratories Licensing Corporation Displaying images on local-dimming displays
US8717278B2 (en) 2010-08-31 2014-05-06 Dolby Laboratories Licensing Corporation Method and apparatus for adjusting drive values for dual modulation displays
US20120115016A1 (en) * 2010-11-04 2012-05-10 Myung-Chul Kim Battery module
WO2012078262A1 (en) 2010-12-06 2012-06-14 Dolby Laboratories Licensing Corporation Methods and apparatus for image adjustment for displays having 2d and 3d display modes
US9224320B2 (en) 2011-03-09 2015-12-29 Dolby Laboratories Licensing Corporation Projection display providing additional modulation and related methods
EP2766894A1 (en) 2011-10-13 2014-08-20 Dolby Laboratories Licensing Corporation Methods and apparatus for backlighting dual modulation display devices
US9747866B2 (en) 2011-11-22 2017-08-29 Dolby Laboratories Licensing Corporation Optimizing light output profile for dual-modulation display performance
EP2862162B1 (en) 2012-06-15 2020-03-18 Dolby Laboratories Licensing Corporation Systems and methods for controlling dual modulation displays
KR102118309B1 (ko) 2012-09-19 2020-06-03 돌비 레버러토리즈 라이쎈싱 코오포레이션 양자점/리모트 인광 디스플레이 시스템 개선
BR112015020571B1 (pt) 2013-03-08 2022-04-12 Dolby Laboratories Licensing Corporation Método para acionamento de um monitor de escurecimento local, meio de armazenamento não transitório legível por computador e aparelho
EP4012695A1 (en) 2013-05-07 2022-06-15 Dolby Laboratories Licensing Corp. Multi-half-tone imaging and dual modulation projection/dual modulation laser projection
JP6237020B2 (ja) * 2013-09-13 2017-11-29 セイコーエプソン株式会社 画像表示装置、および画像表示装置の制御方法
JP5945559B2 (ja) * 2014-02-26 2016-07-05 ドルビー ラボラトリーズ ライセンシング コーポレイション 複数の変調器ディスプレイおよび関連する方法
EP3123240A2 (en) 2014-03-26 2017-02-01 Dolby Laboratories Licensing Corp. Global light compensation in a variety of displays
JP6236188B2 (ja) 2014-08-21 2017-11-22 ドルビー ラボラトリーズ ライセンシング コーポレイション 光変換を伴うデュアルモジュレーションの技術

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6275878A (ja) * 1985-09-30 1987-04-07 Toshiba Corp 画像処理装置
GB9123210D0 (en) * 1991-11-01 1991-12-18 Marconi Gec Ltd Filter
EP0935145A1 (de) * 1998-02-04 1999-08-11 IMS Industrial Micro System AG Optische Signal- und Anzeigevorrichtung
US6559826B1 (en) * 1998-11-06 2003-05-06 Silicon Graphics, Inc. Method for modeling and updating a colorimetric reference profile for a flat panel display
JP3766231B2 (ja) * 1999-05-10 2006-04-12 Ｎｅｃビューテクノロジー株式会社 液晶表示装置
JP4355977B2 (ja) * 1999-11-12 2009-11-04 ソニー株式会社 映像表示装置および映像表示装置における照明制御方法
CN1321912A (zh) * 2000-05-02 2001-11-14 邱励楠 场顺序式彩色液晶显示器件
JP2001332394A (ja) * 2000-05-23 2001-11-30 Sharp Corp 光変調情報表示装置および照明制御装置
JP2002014649A (ja) * 2000-06-28 2002-01-18 Matsushita Electric Ind Co Ltd 画像表示装置
JP2002072980A (ja) * 2000-08-31 2002-03-12 Nec Corp カラー映像表示方法および装置
JP3971892B2 (ja) * 2000-09-08 2007-09-05 株式会社日立製作所 液晶表示装置
JP3523170B2 (ja) * 2000-09-21 2004-04-26 株式会社東芝 表示装置
EP2267520B1 (en) * 2001-02-27 2018-07-25 Dolby Laboratories Licensing Corporation A method and device for displaying an image
JP3898012B2 (ja) * 2001-09-06 2007-03-28 シャープ株式会社 表示装置
US7064740B2 (en) 2001-11-09 2006-06-20 Sharp Laboratories Of America, Inc. Backlit display with improved dynamic range
CN2511994Y (zh) * 2001-12-28 2002-09-18 邱虹云 长寿命高亮度大屏幕家用液晶投影电视
ES2675880T3 (es) * 2002-03-13 2018-07-13 Dolby Laboratories Licensing Corporation Compensación de falla de elemento emisor de luz en un monitor
JP2005309338A (ja) * 2004-04-26 2005-11-04 Mitsubishi Electric Corp 画像表示装置および画像表示方法

Also Published As

Publication number Publication date
CA2992935A1 (en) 2006-02-02
WO2006010244A1 (en) 2006-02-02
CA2572968C (en) 2014-04-08
CA2992935C (en) 2019-07-02
KR20070049143A (ko) 2007-05-10
CA2840548A1 (en) 2006-02-02
CN101266758A (zh) 2008-09-17
JP5419352B2 (ja) 2014-02-19
CA3043550C (en) 2020-09-15
EP1779362A4 (en) 2009-08-12
CN100507988C (zh) 2009-07-01
CA2840548C (en) 2018-03-13
JP2015118384A (ja) 2015-06-25
CA2572968A1 (en) 2006-02-02
CN101266758B (zh) 2010-11-17
JP6163505B2 (ja) 2017-07-12
JP5973403B2 (ja) 2016-08-23
JP2014056240A (ja) 2014-03-27
ES2575929T3 (es) 2016-07-04
HK1108756A1 (en) 2008-05-16
CA3043550A1 (en) 2006-02-02
CN101010712A (zh) 2007-08-01
EP1779362A1 (en) 2007-05-02
JP2008507735A (ja) 2008-03-13
KR101121131B1 (ko) 2012-03-20
DK1779362T3 (en) 2016-06-27

Similar Documents

Publication Publication Date Title
US11501724B2 (en) Rapid estimation of effective illuminance patterns for projected light fields
EP1779362B1 (en) Rapid image rendering on dual-modulator displays
EP2183723B1 (en) Enhancing dynamic ranges of images
US9390660B2 (en) Image control for displays
US20140168249A1 (en) Enhancing dynamic ranges of images
EP2126787B1 (en) Multiple modulator displays and related methods

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20070208

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR

DAX Request for extension of the european patent (deleted)
RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: DOLBY LABORATORIES LICENSING CORP.

A4 Supplementary search report drawn up and despatched

Effective date: 20090714

17Q First examination report despatched

Effective date: 20091111

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602005049070

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G09G0003220000

Ipc: G09G0003340000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G09G 3/36 20060101ALI20151022BHEP

Ipc: G09G 3/34 20060101AFI20151022BHEP

INTG Intention to grant announced

Effective date: 20151120

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU MC NL PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 793200

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160515

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 12

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602005049070

Country of ref document: DE

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20160623

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2575929

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20160704

REG Reference to a national code

Ref country code: SE

Ref legal event code: TRGR

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

REG Reference to a national code

Ref country code: GR

Ref legal event code: EP

Ref document number: 20160401275

Country of ref document: GR

Effective date: 20160729

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160822

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602005049070

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20170123

REG Reference to a national code

Ref country code: FR

Ref legal event code: RU

Effective date: 20170329

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 13

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160527

REG Reference to a national code

Ref country code: FR

Ref legal event code: D7

Effective date: 20170724

REG Reference to a national code

Ref country code: AT

Ref legal event code: UEP

Ref document number: 793200

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160420

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 14

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20050527

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160527

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160420

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230512

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20230419

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20230420

Year of fee payment: 19

Ref country code: FR

Payment date: 20230420

Year of fee payment: 19

Ref country code: ES

Payment date: 20230601

Year of fee payment: 19

Ref country code: DK

Payment date: 20230419

Year of fee payment: 19

Ref country code: DE

Payment date: 20230419

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: SE

Payment date: 20230419

Year of fee payment: 19

Ref country code: GR

Payment date: 20230420

Year of fee payment: 19

Ref country code: FI

Payment date: 20230419

Year of fee payment: 19

Ref country code: AT

Payment date: 20230420

Year of fee payment: 19

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230420

Year of fee payment: 19