EP3198625A1 - Dispositif d'acquisition d'images bimode a photocathode - Google Patents
Dispositif d'acquisition d'images bimode a photocathodeInfo
- Publication number
- EP3198625A1 EP3198625A1 EP15766546.4A EP15766546A EP3198625A1 EP 3198625 A1 EP3198625 A1 EP 3198625A1 EP 15766546 A EP15766546 A EP 15766546A EP 3198625 A1 EP3198625 A1 EP 3198625A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- zone
- sensor
- filters
- pixels
- matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004907 flux Effects 0.000 claims abstract description 87
- 239000011159 matrix material Substances 0.000 claims abstract description 85
- 238000005286 illumination Methods 0.000 claims abstract description 52
- 238000012545 processing Methods 0.000 claims abstract description 42
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 7
- OAICVXFJPJFONN-UHFFFAOYSA-N Phosphorus Chemical compound [P] OAICVXFJPJFONN-UHFFFAOYSA-N 0.000 claims description 21
- 239000013307 optical fiber Substances 0.000 claims description 15
- 238000001429 visible spectrum Methods 0.000 claims description 15
- 238000002329 infrared spectrum Methods 0.000 claims description 7
- 230000003287 optical effect Effects 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 7
- 230000000295 complement effect Effects 0.000 claims description 6
- 230000000737 periodic effect Effects 0.000 claims description 6
- 238000001228 spectrum Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 4
- 238000000034 method Methods 0.000 claims description 3
- 230000003595 spectral effect Effects 0.000 description 7
- 239000003086 colorant Substances 0.000 description 5
- 230000005686 electrostatic field Effects 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 5
- 239000010410 layer Substances 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 230000004297 night vision Effects 0.000 description 3
- 230000007480 spreading Effects 0.000 description 3
- 239000000758 substrate Substances 0.000 description 3
- 238000003786 synthesis reaction Methods 0.000 description 3
- 230000007704 transition Effects 0.000 description 3
- 238000002211 ultraviolet spectrum Methods 0.000 description 3
- JBRZTFJDHDCESZ-UHFFFAOYSA-N AsGa Chemical compound [As]#[Ga] JBRZTFJDHDCESZ-UHFFFAOYSA-N 0.000 description 2
- 229910001218 Gallium arsenide Inorganic materials 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 2
- 239000000654 additive Substances 0.000 description 2
- 230000000996 additive effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000002161 passivation Methods 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 101000694017 Homo sapiens Sodium channel protein type 5 subunit alpha Proteins 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 244000145845 chattering Species 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003707 image sharpening Methods 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 239000002861 polymer material Substances 0.000 description 1
- 239000011241 protective layer Substances 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J31/00—Cathode ray tubes; Electron beam tubes
- H01J31/08—Cathode ray tubes; Electron beam tubes having a screen on or from which an image or pattern is formed, picked up, converted, or stored
- H01J31/50—Image-conversion or image-amplification tubes, i.e. having optical, X-ray, or analogous input, and optical output
- H01J31/56—Image-conversion or image-amplification tubes, i.e. having optical, X-ray, or analogous input, and optical output for converting or amplifying images in two or more colours
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J31/00—Cathode ray tubes; Electron beam tubes
- H01J31/08—Cathode ray tubes; Electron beam tubes having a screen on or from which an image or pattern is formed, picked up, converted, or stored
- H01J31/50—Image-conversion or image-amplification tubes, i.e. having optical, X-ray, or analogous input, and optical output
- H01J31/508—Multistage converters
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01J—ELECTRIC DISCHARGE TUBES OR DISCHARGE LAMPS
- H01J2231/00—Cathode ray tubes or electron beam tubes
- H01J2231/50—Imaging and conversion tubes
- H01J2231/50005—Imaging and conversion tubes characterised by form of illumination
- H01J2231/5001—Photons
- H01J2231/50015—Light
- H01J2231/50026—Infrared
Definitions
- the present invention relates to the field of night vision image acquisition devices, comprising a photocathode adapted to convert a stream of photons into a stream of electrons.
- the field of the invention is more particularly that of such devices, using matrix color filters.
- Such a device is for example an image intensifier tube, comprising a photocathode, adapted to convert an incident flux of photons into an initial flow of electrons.
- This initial flow of electrons propagates inside the intensifier tube, where it is accelerated by a first electrostatic field towards multiplication means.
- These multiplying means receive said initial flow of electrons, and in response provide a secondary electron flow.
- Each initial electron incident on an input side of the multiplication means causes the emission of several secondary electrons on the side of the output face of these same means.
- the secondary electron flux is accelerated by a third electrostatic field in the direction of a phosphor screen, which converts the secondary electron flux into a photon flux.
- the photon flux provided by the phosphor screen corresponds to the flux of photons incident on the photocathode, but in more intense.
- each photon of the photon flux incident on the photocathode corresponds to several photons of the photon flux supplied by the phosphor screen.
- the photocathode and the multiplying means are placed in a vacuum tube having an entrance window to let the incident photon flux enter the photocathode.
- the vacuum tube can be closed by the phosphor screen.
- the photon flux provided by the phosphor screen corresponds to a monochrome image.
- GB 2 302 444 proposes an image intensifier tube for rendering a poly-chromatic image.
- a first primary color filter array is disposed upstream of the photocathode to filter incident photon flux before it reaches the photocathode.
- a primary color filter is a spectral filter, which does not transmit a portion of the visible spectrum complementary to this primary color.
- a primary color filter is a spectral filter that transmits part of the visible spectrum corresponding to this primary color, and possibly part of the infrared spectrum, and even part of the near-UV spectrum (200 to 400 nm) or even UV (10 to 200 nm).
- the first primary color filter matrix consists of red, green, and blue filters that draw primary color pixels on the primary color filter.
- photocathode a photon flux incident on a given pixel of the photocathode corresponds to a given primary color.
- the electron flow supplied in response by the photocathode does not directly contain chromatic information, but corresponds to this given primary color.
- the photon flux supplied by the phosphor screen corresponds to a white light, a combination of several wavelengths corresponding in particular to red, green and blue.
- This stream is filtered by a second matrix of primary color filters.
- This second matrix draws pixels of primary color on the phosphor screen.
- a flux of photons emitted by a given pixel of the phosphor screen is filtered by a primary color filter.
- a flux of photons corresponding to a given primary color is obtained.
- the second matrix is identical to and aligned with the first matrix.
- the pixels of the phosphor screen are therefore aligned with the pixels of the photocathode.
- the image supplied at the output of the second matrix is thus composed of pixels of three primary colors, corresponding to an intensified image of the pixelated image at the output of the first matrix.
- this intensifier tube has high energy losses, detrimental in a field characterized by the need for a strong intensification of a photon flux.
- An object of the present invention is to provide an image acquisition device for acquiring color images while minimizing the damage caused by energy losses.
- an image acquisition device comprising:
- a photocathode adapted to convert an incident flux of photons into a stream of electrons
- a sensor consisting of a matrix of elements, called pixels
- the device comprises a matrix of elementary filters, each associated with at least one pixel of the sensor, said matrix being arranged upstream of the photocathode, so that an initial photon flux passes through said matrix before reaching the photocathode;
- the matrix comprises primary color filters, a primary color filter not transmitting a part of the visible spectrum complementary to said primary color, and filters transmitting the entire visible spectrum, called panchromatic filters; and
- the treatment means are adapted to:
- the photocathode is disposed inside a vacuum chamber, and the matrix of elementary filters is located on an inlet window of said vacuum chamber.
- the photocathode is placed inside a closed vacuum chamber by a bundle of optical fibers, and each elementary filter of the matrix of elementary filters is deposited on an end of an optical fiber of said bundle.
- the sensor may be a photosensitive sensor
- the processing means may be adapted to calculate a magnitude representative of an average surface flux of photons
- the device may furthermore comprise:
- multiplication means adapted to receive the electron flow emitted by the photocathode, and to provide in response a secondary electron flow
- a phosphor screen adapted to receive the secondary electron flux and to provide in response a photon flux, said useful photon flux, the sensor being arranged to receive said useful photon flux.
- the senor may be an electron-sensitive sensor adapted to receive the electron flux emitted by the photocathode, and the processing means may be adapted to calculate a magnitude representative of an average electron surface flux.
- the panchromatic filters represent 75% of the elementary filters.
- R, G, B represent red, green, blue primary color filters
- W represents a panchromatic filter, the pattern being defined at a permutation near R, G, B.
- the matrix of elementary filters can be generated by the two-dimensional periodic repetition of the following pattern:
- Ye, Ma, Cy respectively represent primary yellow, magenta and cyan primary color filters
- W represents a panchromatic filter, the pattern being defined at a permutation near Ye, Ma, Cy.
- the treatment means are adapted to:
- the processing means are advantageously adapted to combining a monochrome image and the color image of said zone, the monochrome image of said zone being obtained from the panchromatic pixels of this zone. zoned.
- the treatment means are adapted to:
- the matrix of elementary filters may furthermore comprise infrared filters that do not transmit the visible part of the spectrum, with each infrared filter being associated with at least one pixel of the so-called infrared pixel sensor.
- the processing means are advantageously adapted to:
- the processing means are advantageously adapted to:
- the matrix of elementary filters may consist of an image projected by a projection optical system.
- the invention also relates to a method for forming an image, implemented in a device comprising a photocathode adapted to convert an incident photon flux into an electron flux, and a sensor, the method comprising the following steps: an initial flux of photons, to provide said incident flux of photons, this filtering implementing a matrix of elementary filters comprising primary color filters, a primary color filter not transmitting a part of the visible spectrum complementary to said color primary, and filters transmitting the entire visible spectrum, called panchromatic filters;
- a quantity, called useful quantity to determine if at least one zone of the sensor is in conditions of low or high illumination, the useful quantity being representative of an average surface flux of photons or electrons detected on a set of so-called panchromatic pixels of the sensor, each panchromatic pixel being associated with a panchromatic filter; only if said zone is in conditions of high illumination, forming a color image of said zone from the pixels of this zone associated with primary color filters.
- FIG. 1 schematically illustrates the principle of a device according to the invention
- FIG. 2 schematically illustrates a first embodiment of a processing implemented by the processing means according to the invention
- FIGS. 3A and 3B schematically illustrate two variants of a first embodiment of a matrix of elementary filters according to the invention
- Figure 4 schematically illustrates a first embodiment of a device according to the invention
- FIGS. 5A and 5B schematically illustrate two variants of a second embodiment of a device according to the invention
- FIG. 6 schematically illustrates a second embodiment of a matrix of elementary filters according to the invention.
- FIG. 7 schematically illustrates a second embodiment of a processing implemented by the processing means according to the invention.
- FIG. 1 schematically illustrates the principle of an image acquisition device 100 according to the invention.
- the device 100 comprises a photocathode 120, operating as described in the introduction, and a matrix 110 of elementary filters 111 located upstream of the photocathode.
- a GaAs photocathode is used. (gallium arsenide). Any other type of photocathode, in particular sensitive photocathodes, can be used in a widest wavelength spectrum, including the visible (about 400 to 800 nm), and possibly the near infra-red or even the same. infra-red, and / or the near UV (ultraviolet), or even the UV.
- Each elementary filter 111 filters the incident light at a location of the photocathode 120. Each elementary filter 111 thus defines a pixel on the photocathode 120.
- the elementary filters 111 are transmission filters of at least two different categories: primary color filters, and transparent (or panchromatic) filters.
- a primary color elementary filter is defined in the introduction.
- the elementary filters of the matrix 110 include three types of primary color filters, i.e. filters of three primary colors. This allows an additive or subtractive synthesis of all the colors of the visible spectrum.
- each type of primary color filter transmits only a portion of the visible spectrum, i.e., a band of the 400-700 nm wavelength range, and the different types of primary color pixels. cover all this gap.
- each primary color filter can transmit a portion of the near infra-red or even infra-red spectrum and / or a portion of the near UV or UV spectrum.
- the color filters can be red, green, blue filters, in the case of an additive synthesis, or yellow, magenta, cyan filters, in the case of a subtractive synthesis.
- Other sets of primary colors may be contemplated by those skilled in the art without departing from the scope of the present invention.
- panchromatic elementary filters let pass the whole visible spectrum. Where appropriate, they may also transmit at least a portion of the near-infrared and even infrared spectrum and / or at least a portion of the near UV and even UV spectrum.
- Elementary filters panchromatic can be transparent elements in the visible, or openings (or savings) in the matrix 110. In this second case, the pixels of the photocathode located under these panchromatic elementary filters receive unfiltered light.
- the different types of primary color filters, and the panchromatic filters, are scattered on the elementary filter matrix.
- the elementary filters are advantageously arranged in the form of a periodic repeating pattern in two distinct, generally orthogonal directions in the plane of the photocathode 120.
- Each pattern preferably comprises at least one primary color filter of each type. , and panchromatic filters.
- elementary filters of square shape have been illustrated, these may have any other geometrical shape, for example a hexagon, a disk, or a surface defined according to constraints relating to the transfer function of the device 100 according to the invention.
- the matrix of elementary filters according to the invention can be real, or virtual.
- the matrix of elementary filters is said to be real when it comprises elementary filters having a certain thickness, for example elementary filters made of polymer material or interferential filters.
- the matrix of elementary filters is called virtual when it consists of an image of a second matrix of elementary filters, projected upstream of the photocathode.
- the second matrix of elementary filters consists of a real matrix of elementary filters. It is located in the object plane of a projection optical system. The image formed in the image plane of this projection optical system corresponds to said matrix of virtual elementary filters.
- the example of a real elementary filter matrix has been developed.
- Many variants can be envisaged, by replacing the matrix of real elementary filters by a matrix of virtual elementary filters.
- the device according to the invention will then comprise the second matrix of elementary filters and the projection optical system, as mentioned above.
- the proportion of panchromatic elementary filters in the matrix 110 is greater than or equal to 50%.
- the proportion of panchromatic elementary filters is equal to 75%.
- the elementary filters of primary color can be divided in equal proportions.
- the elementary filters of primary color are distributed in unequal proportions.
- the proportion of a first type of primary color filter does not exceed twice the proportion of the other types of primary color filters.
- the proportion of panchromatic elementary filters is equal to 75%
- the proportion of filters of a first primary color is equal to 12.5%
- the proportion of filters of a second and a third primary color is equal to at 6.25% and 6.25%.
- Matrix 120 receives an initial photon flux.
- initial elementary fluxes of photons 101 each associated with an elementary filter 111, are represented.
- the initial elementary fluxes of photons 101 together form a poly-chromatic image, and may comprise photons located in the visible spectrum, which are close to one another. infrared and even infrared.
- An elementary filter 111 transmits a filtered elementary flux 102, the filtered elementary streams together forming a flux of photons incident on the photocathode.
- the photocathode 120 emits a stream of electrons.
- Each filtered elementary flux 102 corresponds to an elementary electron flux 103.
- An elementary electron flux 103 is all the more important that the corresponding filtered elementary flux 102 comprises photons.
- the elementary electron fluxes 103 do not directly convey chromatic information, but depend directly on a number of photons transmitted by a corresponding elementary filter 111.
- the elementary electron fluxes 103 together form a stream of electrons emitted by the photocathode 120.
- the device 100 further comprises a digital sensor 130.
- the sensor 130 can directly receive the stream of electrons emitted by the photocathode 120.
- this stream of electrons emitted by the photocathode 120 can be converted into a photon flux so that the sensor 130 finally receives a photon flux.
- FIG. 1 being a simple illustration of principle, the sensor 130 is shown directly after the photocathode 120.
- the sensor 130 may be a photon-sensitive or electron-sensitive sensor, and other elements may be interposed between the photocathode 120 and the sensor 130.
- the sensor is sensitive to electrons as emitted by the photocathode, or to photons obtained from these electrons.
- the senor is sensitive:
- the sensor is formed by a matrix of elements, said pixels 131, sensitive to photons or electrons.
- Each elementary filter 111 is associated with at least one pixel 131 of the sensor.
- each elementary filter 111 is aligned with at least one pixel 131 of the sensor, so that a major part of a stream of electrons or photons, resulting from the photons transmitted by this elementary filter 111, reaches this at least one pixel 131.
- each elementary filter 111 is associated with exactly one pixel 131 of the sensor.
- the surface of an elementary filter 111 corresponds to the surface of a pixel 131 of the sensor or to a surface corresponding to the juxtaposition of an integer number of pixels 131 of the sensor.
- panchromatic pixel can be called a pixel of the sensor associated with a panchromatic elementary filter, and "primary color pixel” a pixel of the sensor associated with an elementary filter. of primary color.
- the panchromatic pixels detect electrons or photons associated with the spectral band transmitted by the panchromatic filters.
- Each type of primary color pixel detects electrons or photons associated with the spectral band transmitted by the corresponding primary color filter type.
- the sensor 130 is connected to processing means 140, that is to say calculation means including a processor or a microprocessor.
- the processing means 140 receive as input electrical signals supplied by the sensor 130, and corresponding, for each pixel 131, to the stream of photons received and detected by this pixel when the sensor is sensitive to photons, or to the electron flow received. and detected by this pixel when the sensor is sensitive to electrons.
- the processing means 140 output an image, corresponding to the initial flux of incident photons on the matrix of elementary filters, this flux having been intensified.
- the processing means 140 are adapted to assign, to each pixel of the sensor, information on a type of elementary filter associated with this sensor. For this purpose, they store information making it possible to connect each pixel of the sensor and a type of elementary filter. This information can be in the form of a deconvolution matrix. Thus, the spectral information that is lost during the passage through the photocathode, is restored by the processing means 140.
- the processing means 140 are adapted to implement a treatment, as illustrated in FIG. 2.
- the processing means realize a monochrome image by interpolation of all the panchromatic pixels of the sensor. This image is called “monochrome image of the sensor”. They then implement a segmentation of the sensor in several zones, each zone being homogeneous in terms of photon or electron flux detected by the corresponding panchromatic pixels.
- the processing means then implement the following steps.
- a magnitude F representative of an average surface flux of photons or electrons received and detected by the panchromatic pixels of an area of the sensor, respectively sensitive to photons or electrons.
- the useful magnitude may be equal to the average surface flux of photons or electrons. If the sensor 130 is sensitive to photons, the useful magnitude may be an average luminance on the panchromatic pixels of the sensor area. Thus, the useful magnitude may be an average surface flux of photons or electrons detected on a set of so-called panchromatic pixels of the sensor.
- the useful quantity provides a measurement of the illumination on said sensor zone.
- Conditions of high illumination are associated for example with a higher illuminance at a first threshold between 450 and 550 ⁇ -ux.
- Conditions of low illumination are associated for example with a light illumination less than a second threshold between 400 and 550 ⁇ -ux, the first and the second threshold being equal. If the first and second thresholds are not equal, the first threshold is strictly greater than the second threshold.
- the useful magnitude F is compared with a threshold value F th . If the useful magnitude F is greater than the threshold value F th , the sensor zone is in conditions of high illumination. If the useful magnitude F is smaller than the threshold value F th , the sensor zone is in low illumination conditions.
- Steps 280 and 281 together form a step to determine whether the sensor zone 130 is in low or high light conditions.
- a strong illumination corresponds for example to the acquisition of an image of a night scene, illuminated by the moon (night level 1 to 3).
- a low illumination corresponds for example to the acquisition of an image of a night scene, not illuminated by the moon (night level 4 to 5, ie a luminous illumination of less than 500 ⁇ -ux).
- step 282A If the area is under high illumination conditions, a color image of this area is formed using the primary color pixels of this area (step 282A). It is said that the device operates in high illumination mode.
- an image is formed of each primary color, and the images of each primary color are combined with each other.
- An image of a primary color is formed by interpolating the pixels of this area associated with said primary color. The interpolation makes it possible to compensate for the small proportion of pixels of the sensor of a given primary color.
- the interpolation of the pixels of a primary color is to use the values taken by these pixels to estimate the values that would be taken by the neighboring pixels if these were also pixels of this primary color.
- Primary color images can be optionally processed to sharpen image sharpening. For example, one can obtain a monochrome image of the area by interpolating the panchromatic pixels of this area, and combine this monochrome image, where appropriate after high-pass filtering, with each primary color image of the same area. As the proportion of panchromatic pixels in the matrix is higher than that of the primary color pixels, the resolution of the primary color images is thus improved.
- a monochrome image of said zone is formed from the panchromatic pixels of this zone.
- a monochrome image is formed using the panchromatic pixels of this area (step 282B), and without using the primary color pixels of this area.
- the monochrome image can be obtained by interpolation of the panchromatic pixels of this zone.
- the device is said to operate in low illumination mode.
- the color or monochrome images of the different areas of the sensor are combined to obtain an image of the entire sensor.
- the image of the entire sensor can be displayed, or stored in memory for further processing.
- a color image of each zone of strong is formed illumination, then, in the monochrome image of the sensor used for segmentation, the areas corresponding to these zones of high illumination are replaced by the color images of these zones.
- a linear combination of the monochrome image of the sensor and these color images is performed.
- the color image and the monochrome image are superimposed.
- the zones of the sensor are treated separately. Alternatively, it is determined whether the entire sensor is in conditions of low or high illumination, and is treated in the same way the entire sensor. In this case, there is no segmentation of the monochrome image of the sensor, or combination of the images obtained. Steps 280, 281 and 282A or 282B are implemented over the entire surface of the sensor. In other words, the sensor zone as mentioned above corresponds to the entire sensor.
- the processing means 140 receive as input signals from the sensor, store information for associating each pixel of the sensor with a type of elementary filter, and output a color image, or a monochrome image or a combination of a color image and a monochrome image.
- the invention thus provides an image acquisition device for acquiring a color image of an area of the sensor, when the illumination of the scene detected on this area allows it.
- the device provides an image of the area obtained from the panchromatic elementary filters, thus with a minimal energy loss.
- the device automatically selects one or the other mode of operation.
- switching from one mode to another operates with hysteresis so as to avoid any switching noise (chattering).
- a first threshold for the useful magnitude is provided for the transition from the high illumination mode to the low illumination mode and a second threshold for the useful magnitude is provided for the inverse transition, the first threshold being chosen lower than the second threshold.
- the switching from one mode to the other is done progressively through a transition phase.
- the image acquisition device operates in low illumination mode when the useful magnitude is less than a first threshold and in high illumination mode when it is greater than a second threshold, the second threshold being chosen greater than the first threshold.
- the image acquisition device performs a linear combination of the image obtained by the treatment in high illumination mode and that obtained by the low light mode treatment, the weighting coefficients being given by the deviations of the useful magnitude with the first and second thresholds respectively.
- each elementary filter 111 is aligned with at least one pixel 131 of the sensor, so that each pixel of the sensor associated with an elementary filter receives only photons or electrons corresponding to this elementary filter.
- This disadvantage can be countered by an initial calibration step making it possible to compensate for the misalignment between an elementary filter and a sensor pixel.
- This calibration aims to compensate for the slight degradation due to the transfer function of the optical elements of the device according to the invention (photocathode and, if appropriate, multiplication means and phosphor screen).
- the geometric shape of the filters composing the matrix of elementary filters is calibrated so as to compensate for the effect of said spatial spread.
- the image of an elementary filter is then superimposed perfectly on one or more pixels of the sensor.
- Interstices between adjacent elementary filters are advantageously opaque, in order to block any radiation likely to reach the photocathode without having passed through an elementary filter.
- FIGS. 3A and 3B schematically illustrate two variants of a first embodiment of a matrix 110 of elementary filters according to the invention.
- the primary color elementary filters are red (R), green (G) or blue (B) filters.
- the matrix has 75% panchromatic filters (W).
- the matrix 110 is generated by a two-dimensional periodic repetition 4x4 basic pattern
- Variations of this matrix can be obtained by permutating the R, G, B filters in the pattern (1).
- the green pixels are twice as many as the red pixels, respectively blue.
- This imbalance can be corrected by weighting coefficients adapted when combining three primary color images to form a color image.
- the matrix of FIG. 3B corresponds to the matrix of FIG. 3A, in which the primary color elementary filters R, G, B are replaced respectively by elementary primary color filters yellow (Ye), magentas (Ma), cyans ( Cy). Again, the filters Ye, Ma, Cy can be switched.
- panchromatic filters representing 50% of the elementary filters is as follows:
- FIG. 4 schematically illustrates a first embodiment of a device 400 according to the invention.
- FIG. 4 will only be described for its differences with respect to FIG. 1.
- the use of a calibration step such as that detailed above, is particularly advantageous in this embodiment.
- the device 400 is based on the so-called intensified CMOS or intensified CCD (ICMOS or ICCD, for "Intensified! CMOS” or “Intensified! CCD”) technology.
- the photocathode 420 is disposed inside a vacuum tube 450 of the vacuum tube type of an image intensifier tube according to the prior art and as described in the introduction.
- a vacuum tube designates a vacuum chamber having more particularly a tube shape.
- the vacuum tube 450 has an inlet window 451, transparent in particular in the visible, and optionally in the near infrared or even the infrared.
- the input window allows to let enter, inside the vacuum tube, the flux of photons incident on the photocathode.
- the entrance window is in particular glass.
- the input window is preferably a simple plate.
- the matrix of elementary filters 410 is glued on one face of the inlet window 451, preferably on the inside of the vacuum tube.
- the photocathode is pressed against the matrix of elementary filters 410.
- a metal layer (not shown) may be deposited on the input window, around the matrix of elementary filters 410, to form an electrical contact point for the application. an electrostatic field.
- the phosphor screen emits a stream of photons, called useful flux, which is received by the sensor 430.
- the sensor 430 is photosensitive. This is in particular a CCD (Charge-Coupled Device) sensor, or a CMOS sensor (Complementary Metal Oxide Semiconductor).
- CCD Charge-Coupled Device
- CMOS Complementary Metal Oxide Semiconductor
- FIG. 4 the sensor 430 is represented inside the vacuum tube, the tube being traversed by electrical connections between the sensor 430 and the processing means 440.
- the processing means 440 operate as described with reference to FIG. 2, the useful quantity being representative of the surface flux of photons detected by the panchromatic pixels of the sensor 430.
- the sensor 430 may be in direct contact with the phosphor screen, to limit any spatial spread of the photon beam emitted by the phosphor screen.
- the sensor 430 may be inside the vacuum tube, or outside and against an outlet face of the vacuum tube, formed by the phosphor screen.
- the sensor 430 may be offset outside the vacuum tube 450.
- a bundle of optical fibers can connect the phosphor screen and the pixels of the sensor 430, the bundle of optical fibers forming an exit window of the vacuum tube.
- Such an optical fiber bundle is particularly suitable in the case where the surface of the sensor 430 is smaller than the inside diameter of the vacuum tube.
- each fiber has a diameter on the phosphor screen side greater than its diameter on the sensor side.
- the bundle of optical fibers is said to thin, and performs a reduction of the image provided by the phosphor screen.
- FIGS. 5A and 5B schematically illustrate two variants of a second embodiment of a device 500 according to the invention.
- the device 500 is based on the electro-bombarded CMOS technology, or EBCMOS for the English "Electron Bombarded CMOS”.
- the photocathode 520 is disposed inside a vacuum tube 550.
- the vacuum tube 550 has an inlet window 551, transparent in particular in the visible, and if necessary in the near infrared or even the infrared.
- the elementary filter matrix 510 is adhered to one face of the input window 551, preferably on the inside of the vacuum tube.
- the sensor 530 is disposed inside the vacuum tube 550, and directly receives the stream of electrons emitted by the photocathode.
- the photocathode 520 and the sensor 530 are within a few millimeters of each other, and subjected to a potential difference to create an electrostatic field in the interstice between them. This electrostatic field accelerates the electrons emitted by the photocathode 520 towards the sensor 530.
- the sensor 530 is sensitive to electrons. This is typically a sensor
- CMOS complementary metal-oxide-semiconductor
- CMOS complementary metal-oxide-semiconductor
- the electron-sensitive sensor is illuminated on the back side ("back side illuminated”).
- back side illuminated CMOS sensor whose substrate is thinned and passivated (in English, "back-thinned”).
- the sensor may include a passivation layer, forming an outer layer on the side of the photocathode.
- the passivation layer is deposited on the thinned substrate.
- the substrate receives detection diodes, each associated with a pixel of the sensor.
- the electron-sensitive sensor is illuminated on the front face.
- CMOS sensor whose front face is treated so as to remove the protective layers covering the diodes.
- the front face of a standard CMOS sensor is thus made sensitive to electrons.
- the processing means 540 operate as described with reference to FIG. 2, the useful quantity being representative of the surface flux of electrons detected by the panchromatic pixels of the sensor 530.
- FIG. 5B illustrates a variant of the device 500 of FIG. 5A, in which the vacuum tube 550 is closed by a bundle 552 of optical fibers receiving the matrix of elementary filters.
- the beam 552 of optical fibers is traversed by photons coming from the scene to be imaged.
- a first end of the fiber optic bundle 552 closes the vacuum tube.
- a second end of the beam 552 of optical fibers is in front of the scene to be imaged.
- the vacuum tube 550 no longer has the input window 551, which is replaced by the optical fiber bundle which allows the vacuum tube of the scene to be imaged to be displaced.
- Each elementary filter of the matrix 510 is associated with an optical fiber of the beam 552.
- each elementary filter is directly attached to an optical fiber end, advantageously on the opposite side to the vacuum tube.
- the matrix of elementary filters 510 is outside the vacuum tube, which simplifies its assembly.
- each elementary filter is directly attached to an end of optical fiber, the side of the vacuum tube.
- a variant of the device described with reference to FIG. 4 can be made in the same way.
- FIG. 6 schematically illustrates a second embodiment of a matrix of elementary filters according to the invention.
- the matrix of elementary filters of FIG. 6 differs from the previously described matrices in that it comprises infrared (IR) filters, not transmitting the visible part of the spectrum and allowing the near infrared to pass.
- Infrared filters let the wavelengths pass in the near infrared, or even in the infrared (wavelengths greater than 700 nm).
- Infrared filters transmit in particular the spectral band between 700 and 900 nm, or even between 700 and 1100 nm, and even between 700 and 1700 nm.
- the filter matrix of FIG. 6 differs from the matrix of FIG. 3A in that in the elementary pattern, one of the two green pixels (G) is replaced by an infrared (IR) pixel.
- Different variants of the matrix of FIG. 6 can be formed in the same way, for example from the matrix of FIG. 3B and replacing by an infrared pixel one of the two magenta pixels of the elementary pattern.
- FIG. 7 schematically illustrates a processing implemented by the processing means according to the invention, when the matrix of elementary filters comprises infrared pixels.
- Steps 780, 781 and 782B respectively correspond to steps 280, 281 and 282B as described with reference to FIG.
- the processing means measure a quantity, called secondary quantity, representative of the average surface flux of photons or electrons F [R detected by the infrared pixels of this zone (step 783).
- this average surface flux is an average surface flux of photons if the sensor is photosensitive, or an average surface flux of electrons if the sensor is sensitive to electrons.
- the processing means then make a comparison between this secondary quantity and an infrared threshold F [R th (step 784).
- a false color image of the zone is constructed, ie an image in which a given color is attributed to the infrared pixels of this zone.
- the false color image can be constructed by interpolating the infrared pixels of the considered area.
- the false color image is therefore a monochrome image, of a different color from the monochrome image associated with the panchromatic pixels. Then, we superimpose this image in false color to the monochrome image obtained using the panchromatic pixels of the same area of the sensor.
- step 782C These steps of constructing a false color image and superposition with the monochrome image together form a step 782C.
- zone infrared pixels belonging to this zone have an intensity greater than a predetermined infrared threshold and, if so, it is superimposed on the monochrome image of this zone.
- a single secondary quantity for the same zone is not calculated, but a secondary quantity per infrared pixel of the zone is calculated separately. Only the infrared pixels, for which the corresponding secondary quantity is greater than the infrared threshold, are superimposed on the monochrome image obtained from the panchromatic pixels. Thus, if a sensor zone has a high intensity in the infrared range, it will be easily identifiable in the resulting image.
- sub-zones of said sensor zone are identified, detecting an average surface flux of pixels or electrons homogeneous in the infrared spectrum, and each sub-zone is then treated separately as detailed above.
- the comparison with the infrared threshold is done by homogeneous sub-areas of the sensor.
- a false color image is obtained by interpolation of the infrared pixels of said sub-zone. These false color images are then superimposed on the corresponding locations on the monochrome image of the sensor area.
- a segmentation is performed on the basis of an image made by interpolation of the infrared pixels.
- sub-zones of this zone are identified, having a uniform intensity in the infrared spectrum, and it is determined, for each sub-zone thus identified, whether the The average of the infrared intensity in this sub-area is greater than a predetermined infrared threshold, and if so, this sub-area is represented by a false-color image based on the infrared pixels of that sub-area. false-color image of said sub-area then being superimposed with the monochrome image of the area to which it belongs.
- the infrared pixels of the sensor can also be used to improve a signal-to-noise ratio on a final color image. For this, when a zone of the sensor is in conditions of high illumination, an infrared image of this zone is produced by interpolation of the infrared pixels of the sensor. This infrared image is then subtracted from the color image of this zone, obtained as detailed with reference to FIG. 2. The subtraction of the infrared image makes it possible to improve the signal-to-noise ratio. To avoid saturation problems, a weighted infrared image can be subtracted from each of the primary color images. The weighting coefficients assigned to the infrared image may be identical or different, for each primary color image. Fragmented primary color images are obtained which are combined to form a denuded color image. Thus, the processing means are adapted to implement the following steps:
Landscapes
- Color Television Image Signal Generators (AREA)
- Image-Pickup Tubes, Image-Amplification Tubes, And Storage Tubes (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR1458903A FR3026223B1 (fr) | 2014-09-22 | 2014-09-22 | Dispositif d'acquisition d'images bimode a photocathode. |
PCT/EP2015/071789 WO2016046235A1 (fr) | 2014-09-22 | 2015-09-22 | Dispositif d'acquisition d'images bimode a photocathode |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3198625A1 true EP3198625A1 (fr) | 2017-08-02 |
EP3198625B1 EP3198625B1 (fr) | 2018-12-12 |
Family
ID=52474002
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15766546.4A Active EP3198625B1 (fr) | 2014-09-22 | 2015-09-22 | Dispositif d'acquisition d'images bimode a photocathode |
Country Status (9)
Country | Link |
---|---|
US (1) | US9972471B2 (fr) |
EP (1) | EP3198625B1 (fr) |
JP (1) | JP6564025B2 (fr) |
CN (1) | CN106716592B (fr) |
CA (1) | CA2961118C (fr) |
FR (1) | FR3026223B1 (fr) |
IL (1) | IL251222B (fr) |
SG (1) | SG11201702126UA (fr) |
WO (1) | WO2016046235A1 (fr) |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
FR3045263B1 (fr) * | 2015-12-11 | 2017-12-08 | Thales Sa | Systeme et procede d'acquisition d'images visibles et dans le proche infrarouge au moyen d'un capteur matriciel unique |
JP2017112401A (ja) * | 2015-12-14 | 2017-06-22 | ソニー株式会社 | 撮像素子、画像処理装置および方法、並びにプログラム |
US10197441B1 (en) * | 2018-01-30 | 2019-02-05 | Applied Materials Israel Ltd. | Light detector and a method for detecting light |
US11268849B2 (en) | 2019-04-22 | 2022-03-08 | Applied Materials Israel Ltd. | Sensing unit having photon to electron converter and a method |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH03112041A (ja) * | 1989-09-27 | 1991-05-13 | Hamamatsu Photonics Kk | カラーイメージ管 |
US5233183A (en) * | 1991-07-26 | 1993-08-03 | Itt Corporation | Color image intensifier device and method for producing same |
GB2273812B (en) * | 1992-12-24 | 1997-01-08 | Motorola Inc | Image enhancement device |
AU7687994A (en) * | 1993-08-20 | 1995-03-21 | Intevac, Inc. | Life extender and bright light protection for cctv camera system with image intensifier |
IL114181A (en) * | 1995-06-15 | 1999-07-14 | Orlil Ltd | Color image intensifier device and a method for producing the same |
CA2234950A1 (fr) * | 1995-10-31 | 1997-05-09 | Benjamin T. Gravely | Systeme d'imagerie |
KR100214885B1 (ko) * | 1996-02-29 | 1999-08-02 | 윤덕용 | 발광소자 및 전자 증배기를 이용한 평판 표시기 |
US5914749A (en) * | 1998-03-31 | 1999-06-22 | Intel Corporation | Magenta-white-yellow (MWY) color system for digital image sensor applications |
US6456793B1 (en) * | 2000-08-03 | 2002-09-24 | Eastman Kodak Company | Method and apparatus for a color scannerless range imaging system |
US20030147002A1 (en) * | 2002-02-06 | 2003-08-07 | Eastman Kodak Company | Method and apparatus for a color sequential scannerless range imaging system |
US6861638B2 (en) * | 2002-08-20 | 2005-03-01 | Northrop Grumman Corporation | Method and system for generating an image having multiple hues |
JP4311988B2 (ja) * | 2003-06-12 | 2009-08-12 | アキュートロジック株式会社 | 固体撮像素子用カラーフィルタおよびこれを用いたカラー撮像装置 |
US7123298B2 (en) * | 2003-12-18 | 2006-10-17 | Avago Technologies Sensor Ip Pte. Ltd. | Color image sensor with imaging elements imaging on respective regions of sensor elements |
JP4678172B2 (ja) * | 2004-11-22 | 2011-04-27 | 株式会社豊田中央研究所 | 撮像装置 |
WO2006138599A2 (fr) * | 2005-06-17 | 2006-12-28 | Infocus Corporation | Synchronisation d'un element de production d'image et d'un modulateur de couleur claire |
CN1971927B (zh) * | 2005-07-21 | 2012-07-18 | 索尼株式会社 | 物理信息获取方法、物理信息获取装置和半导体器件 |
US8139130B2 (en) * | 2005-07-28 | 2012-03-20 | Omnivision Technologies, Inc. | Image sensor with improved light sensitivity |
US7688368B2 (en) * | 2006-01-27 | 2010-03-30 | Eastman Kodak Company | Image sensor with improved light sensitivity |
KR20070115243A (ko) * | 2006-06-01 | 2007-12-05 | 삼성전자주식회사 | 이미지 촬상 장치, 및 그 동작 방법 |
CN102317951B (zh) * | 2009-02-11 | 2015-01-21 | 数据逻辑扫描公司 | 利用彩色成像器进行高分辨率光学代码成像的系统和方法 |
CN202696807U (zh) * | 2012-07-20 | 2013-01-23 | 合肥汉翔电子科技有限公司 | 一种微滤镜彩色微光成像机构 |
JP5981820B2 (ja) * | 2012-09-25 | 2016-08-31 | 浜松ホトニクス株式会社 | マイクロチャンネルプレート、マイクロチャンネルプレートの製造方法、及びイメージインテンシファイア |
FR3004882B1 (fr) * | 2013-04-17 | 2015-05-15 | Photonis France | Dispositif d'acquisition d'images bimode |
US9503623B2 (en) * | 2014-06-03 | 2016-11-22 | Applied Minds, Llc | Color night vision cameras, systems, and methods thereof |
-
2014
- 2014-09-22 FR FR1458903A patent/FR3026223B1/fr not_active Expired - Fee Related
-
2015
- 2015-09-22 EP EP15766546.4A patent/EP3198625B1/fr active Active
- 2015-09-22 CA CA2961118A patent/CA2961118C/fr active Active
- 2015-09-22 SG SG11201702126UA patent/SG11201702126UA/en unknown
- 2015-09-22 WO PCT/EP2015/071789 patent/WO2016046235A1/fr active Application Filing
- 2015-09-22 JP JP2017515796A patent/JP6564025B2/ja active Active
- 2015-09-22 US US15/512,253 patent/US9972471B2/en active Active
- 2015-09-22 CN CN201580050815.3A patent/CN106716592B/zh active Active
-
2017
- 2017-03-16 IL IL251222A patent/IL251222B/en active IP Right Grant
Also Published As
Publication number | Publication date |
---|---|
IL251222A0 (en) | 2017-05-29 |
CA2961118C (fr) | 2023-03-21 |
CN106716592B (zh) | 2019-03-05 |
CN106716592A (zh) | 2017-05-24 |
JP6564025B2 (ja) | 2019-08-21 |
IL251222B (en) | 2020-11-30 |
US9972471B2 (en) | 2018-05-15 |
CA2961118A1 (fr) | 2016-03-31 |
FR3026223B1 (fr) | 2016-12-23 |
EP3198625B1 (fr) | 2018-12-12 |
WO2016046235A1 (fr) | 2016-03-31 |
JP2017533544A (ja) | 2017-11-09 |
SG11201702126UA (en) | 2017-04-27 |
US20170287667A1 (en) | 2017-10-05 |
FR3026223A1 (fr) | 2016-03-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11031423B2 (en) | Imaging element and camera system | |
CA2909554C (fr) | Dispositif d'acquisition d'images bimode | |
EP3198625B1 (fr) | Dispositif d'acquisition d'images bimode a photocathode | |
US9497370B2 (en) | Array camera architecture implementing quantum dot color filters | |
JP2011199798A (ja) | 物理情報取得装置、固体撮像装置、物理情報取得方法 | |
EP3387824B1 (fr) | Système et procédé d'acquisition d'images visibles et dans le proche infrarouge au moyen d'un capteur matriciel unique | |
EP2257778A1 (fr) | Dispositif et procédé de mesure spatio-colorimétrique d'un objet tridimensionnel | |
EP3155660B1 (fr) | Capteur matriciel bispectral et son procédé de fabrication | |
US20190058837A1 (en) | System for capturing scene and nir relighting effects in movie postproduction transmission | |
FR3071342B1 (fr) | Capteur d'image a matrice de bayer | |
FR3071788A1 (fr) | Systeme d'observation de conducteur et son procede de saisie par le systeme et le procede de fabrication du systeme | |
US11696043B2 (en) | White balance compensation using a spectral sensor system | |
US20200007853A1 (en) | Imaging systems with depth detection | |
JP2019165447A (ja) | 固体撮像装置及び撮像システム | |
FR3026227A1 (fr) | Dispositif d'acquisition d'images 3d | |
EP1432044A1 (fr) | Photodiode à filtre en polysilicium intégré | |
FR2968777A1 (fr) | Systeme d'acquisition d'images en visibilite reduite eventuellement integrable dans un systeme de visualisation de casque | |
FR3107124A1 (fr) | Appareil pour l’observation d’un astre | |
EP3884656A1 (fr) | Appareil et procédé pour observer une scène comportant une cible | |
WO2022112688A1 (fr) | Imageur multispectral à domaine spectral élargi | |
FR2516705A1 (fr) | Structure de detection photoelectrique | |
FR2960065A1 (fr) | Dispositif combine d'observation et d'emission, et procede de visualisation d'une direction d'emission d'un tel dispositif |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20170324 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: LETEXIER, DAMIEN Inventor name: ROBERT, FRANCK |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20180703 |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: LETEXIER, DAMIEN Inventor name: ROBERT, FRANCK |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D Free format text: NOT ENGLISH |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1077094 Country of ref document: AT Kind code of ref document: T Effective date: 20181215 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015021427 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D Free format text: LANGUAGE OF EP DOCUMENT: FRENCH |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: FP |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190312 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190312 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1077094 Country of ref document: AT Kind code of ref document: T Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190313 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190412 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20190412 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015021427 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
26N | No opposition filed |
Effective date: 20190913 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190922 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190930 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190930 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190922 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20190930 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190930 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20150922 Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20181212 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230526 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20230927 Year of fee payment: 9 Ref country code: GB Payment date: 20230928 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230927 Year of fee payment: 9 Ref country code: DE Payment date: 20230927 Year of fee payment: 9 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602015021427 Country of ref document: DE Representative=s name: LAURENT CHARRAS, FR |