EP2911123B1 - A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation - Google Patents
A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation Download PDFInfo
- Publication number
- EP2911123B1 EP2911123B1 EP15155865.7A EP15155865A EP2911123B1 EP 2911123 B1 EP2911123 B1 EP 2911123B1 EP 15155865 A EP15155865 A EP 15155865A EP 2911123 B1 EP2911123 B1 EP 2911123B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- note
- unfit
- notes
- classification
- fit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims description 34
- 238000004458 analytical method Methods 0.000 claims description 39
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000000007 visual effect Effects 0.000 claims description 14
- 238000001228 spectrum Methods 0.000 claims description 13
- 238000012512 characterization method Methods 0.000 claims description 6
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 2
- 241000282414 Homo sapiens Species 0.000 description 5
- 230000008447 perception Effects 0.000 description 3
- 239000002390 adhesive tape Substances 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000003623 enhancer Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 238000012732 spatial analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
- G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
- G07D7/181—Testing mechanical properties or condition, e.g. wear or tear
- G07D7/187—Detecting defacement or contamination, e.g. dirt
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
- G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
- G07D7/06—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
- G07D7/12—Visible light, infrared or ultraviolet radiation
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
- G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
- G07D7/06—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
- G07D7/12—Visible light, infrared or ultraviolet radiation
- G07D7/1205—Testing spectral properties
Definitions
- the present invention consists of a method and device for characterising the state of use of banknotes and their classification as fit or unfit for circulation, the object of which is to avoid errors in said classification, thus preventing unfit notes from being misclassified as fit and being put back into circulation when they should have been destroyed, and avoiding fit notes from being misclassified as unfit and being prematurely destroyed as a result.
- One of the jobs carried out by the National Central Banks in relation to cash is to classify the notes that return from circulation by the state of use by separating the notes fit to remain in circulation from those that are not.
- fit notes are prepared to be put back into circulation and unfit notes are destroyed.
- This classification by state of use is carried out via the detection and analysis of different features on the notes, such as, for example, the degree of soiling of the notes; the existence of holes and determining their size; the existence of folded corners and/or that they are not there and determining their size; the existence of tears and determining their size; the existence of adhesive tapes and determining their size; the existence of stains and determining their size; etc.
- Document US 5175775 A may also be acknowledged, this document being related to a method of optical pattern recognition which can be applied to a joint transform correlator operative to effect correlation processing between a plurality of reference images and at least a single object image to produce correlation peaks corresponding to the respective reference images.
- Document EP 0691632 A1 refers to a method and an apparatus for detecting counterfeit bank-notes, the apparatus including an electronic camera which forms a digital representation of a small area of the bank-note.
- the digital image is stored in an image memory.
- the Fourier transform of the digital image is then computed and analyzed to determine if it contains high spatial frequency components, and if so, the note is identified as counterfeit.
- Document WO 2008/014090 A2 discloses techniques for authenticating security documents having security images that incorporate multiple security features. The techniques may be particularly useful in validating a security document having a security image composed of one or more "virtual" retroreflective images formed over a background of a repeating retroreflective image.
- document EP 1011079 A1 may also be acknowledged, this document being referred to an apparatus for determining the soil degree of printed matter.
- the invention proposes a new method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation.
- the main objective of the invention is to characterise and detect the soiling of notes, although it also enables other features, such as stains and writing or "graffiti" for example, to be detected.
- the main novelty of the invention lies in that after the analysis in the spatial domain, an analysis in the frequency domain of the captured image is carried out, to which end the mathematical transformation known as Fast Fourier Transform (FFT) is applied to said captured image in at least one of the R, G and B channels, such that a new image, referred to as "power spectrum", is obtained. Furthermore, in order to carry out the analysis in the frequency domain, one or more of the aforementioned statistics is calculated on the pixels of the captured image of the power spectrum.
- FFT Fast Fourier Transform
- Carrying out the analysis of the images in the frequency domain provides information that is complementary to the information obtained via traditional analysis in the spatial domain.
- the method of the invention also comprises visually and individually assessing beforehand a set of used notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification is carried out by a group of experts paying particular attention to the region of the captured image of the note to be analysed.
- the individual images of the set of used notes that were previously assessed visually are captured, in order to then analyse these images in all the RGB channels, both in the spatial and frequency domain, and obtain all the aforementioned statistics.
- the captured images must correspond with the captured image of the note to be analysed.
- At least one statistic obtained via the analysis in the spatial domain is then combined with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single numeric value, referred to as FV (Fitness Value).
- FV Fitness Value
- a number of reference values are obtained from the statistics with a greater correlation of the previous point for a note in perfect condition of the same type as the note to be analysed.
- individual images are captured of a set of new notes with the same denomination as the note to be analysed, which correspond to the captured image of the note to be analysed; the selected statistics that offer a greater correlation in the visual classification are calculated; and the minimum values are taken as the reference values.
- the invention fixes an arbitrary FV value threshold and, lastly, the calculated FV is compared with the FV threshold, and the note analysed is classified as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
- the method described avoids the problems mentioned in the state of the art regarding low repetitiveness and reproducibility, crossover error rates and low correlation with the human perception of soiling on notes, since the invention, compared to the systems that currently exist, has high repetitiveness and reproducibility, better FFR and FUR values and a high correlation with the human perception of soiling on notes.
- an algorithm which is calculated as the root n of the sum of the difference of the values raised to n of the selected statistics with respect to the reference values that correspond to a note in perfect condition with zero level of soiling, is used to obtain the calculated FV.
- the scale correction factors C i may be obtained conventionally.
- it comprises carrying out a selection of at least one region of interest of the captured image for characterising the state of use of the notes between the colour correcting phase of the captured image and the spatial domain analysing phase.
- both the spatial analysis and the frequency analysis are only carried out on said region of interest and similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected region of interest.
- the reference values are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
- the invention provides the possibility of selecting, within said region of interest, at least one area of the region of interest from the image referred to as "power spectrum".
- the spatial and frequency analysis is carried out on said area of the region of interest.
- the visual assessment by a group of experts is carried out paying particular attention to the selected area of the region of interest, and the reference values are obtained by selecting said areas of said region of interest in the individual images of the set of new notes of the same denomination as the note to be analysed.
- the captured images must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
- a region is selected that is not printed and represents a percentage equal to or greater than a previously established percentage of the total surface of the note. In the preferred embodiment this percentage is equal to or less than 10% of the total surface of the note.
- the invention relates to a device, functioning in accordance with the method described, which is characterised in that it comprises a module for digitally capturing images by reflection, via which at least one digital image of at least one of the faces of the note is captured. It also includes a captured digital image processor, which is configured to process the captured images in accordance with the previously described method.
- the module for digitally capturing images by reflection it has been provided that it may be a scanner, digital camera or CCD array (Charge Coupled Device).
- CCD array Charge Coupled Device
- Figure 1 Shows a schematic representation of the phases that make up the method of the invention.
- the invention relates to a method and a device that enables the characterisation of the soiling of banknotes and their classification as fit and unfit for circulation.
- images of the notes to be analysed are captured (1) by reflection. Only the front, only the back or the front and back may be captured via the use of a scanner, camera, CCD array (Charge Coupled Device) or any other type of module for digitally capturing images that forms part of the device of the invention.
- the image of the front of the note to be analysed is captured and the captured image of each note must have a sufficient resolution, the recommended minimum being 150 dpi (Dots Per Inch), a minimum colour depth of 24 bits per pixel and must be in a lossless compression format.
- the captured image is stored (2) in a standard lossless compression format and colour correction (3) of the captured image is carried out using the ICC colour profile of the module used for capturing the image.
- the ICC profile of the capturing module is built via a calibration process by capturing a calibration card and comparing the result to the reference values provided by the calibration card supplier, for which reason it is not described in greater detail.
- the object of the colour correction phase is to guarantee the repetitiveness and reproducibility of the method and the device, since the variations due to the capturing module are minimised as a result of this correction. Via this correction it is achieved that images of the same note originating from different capturing modules, or from the same module but captured at different times, are similar.
- One or several regions of interest are selected (16) on the colour corrected image.
- a single, rectangular-shaped region of interest comprising the watermark area of the note, which is the area where the paper is not printed, is selected.
- more than one region of interest and non-rectangular shapes may be used.
- the analysis (4) in the spatial domain of the selected region of interest is then carried out.
- said analysis in the spatial domain is carried out on said selected regions of interest.
- the analysis is carried out on the entire surface of the note.
- the colour corrected image is split (4a) into the three channels: Red (R), Green (G) and Blue (B).
- a number of statistics selected from between the arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient are calculated (4b) in each one of these channels.
- the analysis is carried out for a €50 note and it has been experimentally determined, as shall be explained below, that a good characterisation of the state of use is obtained for the analysis in the spatial domain by calculating the mean in the green channel and the standard deviation of the pixels contained in the region of interest in the blue channel. In the case where more than one region of interest has been defined, these calculations are carried out on the pixels contained in these regions of interest and if a region of interest has not been defined these calculations are done for all the pixels of the image.
- the analysis (5) in the frequency domain is then carried out on the green channel, for which reason the Fast Fourier Transform or FFT is applied (5a), thus obtaining a new image that is referred to as Power Spectrum.
- the Power Spectrum may be defined on the Power Spectrum, at least one statistic from between all of the aforementioned in the area or areas of interest of the Power Spectrum being calculated (5b).
- an area of interest of the region of interest has not been defined, for which reason this calculation is made on all the pixels of the Power Spectrum and it has been determined experimentally, as shall be explained below, that a good characterisation of the state of use is obtained via the calculation of the standard deviation of the value of the pixels in the image that is obtained when the Fast Fourier Transform is applied on the image of the region of interest.
- the value of the standard deviation in the power spectrum 19.436.
- the FFT is applied to the entire image of the note.
- a visual and individual classification (6) is carried out beforehand of a set of notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification (6) is carried out by a group of experts paying particular attention to the region of interest.
- the correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed (6) notes is then analysed (9) in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note and the minimum values are taken as reference values.
- the statistics calculated that offer a greater correlation correspond to those specified above; this is mean deviation in the green channel, standard deviation in the blue channel and standard deviation in the power spectrum of the green channel.
- a number of reference values (12) are obtained from the statistics with greater correlation of the point above, which corresponds to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of a set of new notes with the same denomination as the note to be analysed are captured (10), the same region of interest is selected as in the notes to be analysed, the selected statistics that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12).
- At least one statistic obtained via the analysis in the spatial domain is combined (13) with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single value referred to as FV, in the calculation of which the statistics of the analysis (9) described in the previous point that offer a greater correlation in the classification carried out visually by the group of experts is used.
- the value of the exponent n is obtained experimentally such that the coincidence percentage between the visual classification carried out and the analysis of said visually classified notes is maximised, as described above.
- m 3 which was indicated, is obtained experimentally (green channel mean, blue channel standard deviation and standard deviation of the FFT of the green channel).
- m is comprised between 2 and 4, since the FV calculation increases greatly in complexity with a greater number of statistics and the results in the classification of the notes as fit or unfit for circulation improves very little.
- an FV value threshold is fixed (14), which in the case of the €50 notes of the exemplary embodiment is 50.
- the calculated FV is compared (15) to the FV threshold, in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
- the note if the value obtained for the note, object of the study, is higher than the threshold value, the note is unfit to continue in circulation, and if not, it is fit to continue in circulation.
- the FV value obtained is 62.66, which is greater than the FV value threshold of 50 and, as such, the note analysed is unfit to continue in circulation.
- the minimum to be used in the FV calculation is a mean or deviation originating from the analysis in the spatial domain and a mean or standard deviation originating from the analysis in the frequency domain.
- the invention relates to a device that, as described, comprises a module for capturing images by reflection (scanner, etc.) that is completed with a captured digital image processor that is configured to process the captured digital image in accordance with the previous described phases of the method.
Description
- The present invention, as stated in the title of this specification, consists of a method and device for characterising the state of use of banknotes and their classification as fit or unfit for circulation, the object of which is to avoid errors in said classification, thus preventing unfit notes from being misclassified as fit and being put back into circulation when they should have been destroyed, and avoiding fit notes from being misclassified as unfit and being prematurely destroyed as a result.
- One of the jobs carried out by the National Central Banks in relation to cash is to classify the notes that return from circulation by the state of use by separating the notes fit to remain in circulation from those that are not.
- Normally, fit notes are prepared to be put back into circulation and unfit notes are destroyed.
- This classification by state of use is carried out via the detection and analysis of different features on the notes, such as, for example, the degree of soiling of the notes; the existence of holes and determining their size; the existence of folded corners and/or that they are not there and determining their size; the existence of tears and determining their size; the existence of adhesive tapes and determining their size; the existence of stains and determining their size; etc.
- The hardest one to measure out of all of them is the degree of soiling of the notes and it is also the one that causes the greatest number of notes to be classified as unfit and, therefore, destroyed.
- The systems and methods that currently exist are based exclusively on analysing the images of notes in the spatial domain, i.e. on splitting the images into three RGB (Red, Green, Blue) channels and measuring, thereon or on specific combinations of these channels, the mean and standard deviation of the pixels contained within certain regions of interest. As such, the patent document
WO 2012165959 regarding a "Method and device for classifying security documents such as banknotes" may be cited. - The document above, similarly to the rest of the methods and devices that currently exist, has a series of problems for characterising the soiling of notes and its classification as fit or unfit. Basically, there are three types of problems:
- 1. Low repetitiveness and reproducibility: the same note, processed several times in the same system, or in different systems of the same type, produces different results.
- 2. Crossover error rates: there is the "False Fit Rate, FFR" (soiled notes misclassified as clean) and the "False Unfit Rate, FUR" (clean notes misclassified as soiled). With the current systems, in order to achieve an FFR below 10%, an FUR above 15 and 20% is obtained. Whilst FFR "contaminates" the notes in circulation with contaminated notes, the FUR is a direct economic loss for the Central Banks as they are notes fit for continued circulation that are being prematurely destroyed.
- 3. Low correlation with the human perception of soiling on notes: often, notes classified as contaminated by the current systems are not perceived as such by human beings and vice versa, notes classified as clean have an unacceptable degree of soiling for a human observer.
- If it is taken into account that National Central Banks throughout the world classify billions of notes each year by state of use, an idea of the scale of the problem represented by the inconsistency of characterising the soiling of notes using the current methods or systems can appreciated.
- Document
US 2003/031340 A1 may be acknowledged. This document discloses techniques helpful in resolving the authentication of an identification document or banknote including a security enhancer. - Document
US 5175775 A may also be acknowledged, this document being related to a method of optical pattern recognition which can be applied to a joint transform correlator operative to effect correlation processing between a plurality of reference images and at least a single object image to produce correlation peaks corresponding to the respective reference images. - Document
EP 0691632 A1 refers to a method and an apparatus for detecting counterfeit bank-notes, the apparatus including an electronic camera which forms a digital representation of a small area of the bank-note. The digital image is stored in an image memory. The Fourier transform of the digital image is then computed and analyzed to determine if it contains high spatial frequency components, and if so, the note is identified as counterfeit. - Document
WO 2008/014090 A2 discloses techniques for authenticating security documents having security images that incorporate multiple security features. The techniques may be particularly useful in validating a security document having a security image composed of one or more "virtual" retroreflective images formed over a background of a repeating retroreflective image. - Lastly, document
EP 1011079 A1 may also be acknowledged, this document being referred to an apparatus for determining the soil degree of printed matter. - In order to achieve the objectives and solve the aforementioned drawbacks, the invention proposes a new method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation. The main objective of the invention is to characterise and detect the soiling of notes, although it also enables other features, such as stains and writing or "graffiti" for example, to be detected.
- It is known that to characterise the soiling of banknotes, methods are used that comprise capturing, by reflection, at least one image of at least one of the faces of a note, via a digital device for capturing images by reflection. The captured image is then stored in a standard format, the colour of the captured image is corrected using the ICC colour profile (colour profile standardised by the International Color Consortium) of the device used to capture the image. The captured image is then analysed in the spatial domain, to which end the image thereof is split into the RGB colour channels and one or more statistics (variables) of the image pixels of the captured image is calculated in at least one of the three RGB channels and on the value of the pixels comprising said captured image. The statistics calculated are selected from between the arithmetic mean, standard deviation, maximum, minimum, median, mode, Kurtosis coefficient and symmetry coefficient.
- The main novelty of the invention lies in that after the analysis in the spatial domain, an analysis in the frequency domain of the captured image is carried out, to which end the mathematical transformation known as Fast Fourier Transform (FFT) is applied to said captured image in at least one of the R, G and B channels, such that a new image, referred to as "power spectrum", is obtained. Furthermore, in order to carry out the analysis in the frequency domain, one or more of the aforementioned statistics is calculated on the pixels of the captured image of the power spectrum.
- Carrying out the analysis of the images in the frequency domain provides information that is complementary to the information obtained via traditional analysis in the spatial domain.
- The method of the invention also comprises visually and individually assessing beforehand a set of used notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification is carried out by a group of experts paying particular attention to the region of the captured image of the note to be analysed.
- It is provided that the individual images of the set of used notes that were previously assessed visually are captured, in order to then analyse these images in all the RGB channels, both in the spatial and frequency domain, and obtain all the aforementioned statistics. The captured images must correspond with the captured image of the note to be analysed.
- The correlation between the visual classification carried out by the group of experts and the data obtained in the analysis of the spatial and frequency domain of said individual images of the visually assessed notes is then analysed in order to determine the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note. This correlation analysis is carried out conventionally, for which reason it is not described in greater detail.
- At least one statistic obtained via the analysis in the spatial domain is then combined with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single numeric value, referred to as FV (Fitness Value). The statistics of the analysis described in the paragraph above, which offer a greater correlation in the classification carried out visually by the group of experts, is used to carry out this calculation.
- Subsequently, a number of reference values are obtained from the statistics with a greater correlation of the previous point for a note in perfect condition of the same type as the note to be analysed. In order to do so, individual images are captured of a set of new notes with the same denomination as the note to be analysed, which correspond to the captured image of the note to be analysed; the selected statistics that offer a greater correlation in the visual classification are calculated; and the minimum values are taken as the reference values.
- Moreover, it is provided that the invention fixes an arbitrary FV value threshold and, lastly, the calculated FV is compared with the FV threshold, and the note analysed is classified as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
- The method described avoids the problems mentioned in the state of the art regarding low repetitiveness and reproducibility, crossover error rates and low correlation with the human perception of soiling on notes, since the invention, compared to the systems that currently exist, has high repetitiveness and reproducibility, better FFR and FUR values and a high correlation with the human perception of soiling on notes.
- In the preferred embodiment of the invention, the FV is calculated as a distance, from the note object of the analysis to a perfectly clean note that is taken as a reference, as mentioned above, via the equation:
- In one embodiment of the invention, it is provided that it comprises carrying out a selection of at least one region of interest of the captured image for characterising the state of use of the notes between the colour correcting phase of the captured image and the spatial domain analysing phase. In this case, both the spatial analysis and the frequency analysis are only carried out on said region of interest and similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected region of interest. Furthermore, in this case, the reference values are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
- When at least one region of interest of the captured image is selected, the invention provides the possibility of selecting, within said region of interest, at least one area of the region of interest from the image referred to as "power spectrum". In this case, the spatial and frequency analysis is carried out on said area of the region of interest. Similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected area of the region of interest, and the reference values are obtained by selecting said areas of said region of interest in the individual images of the set of new notes of the same denomination as the note to be analysed.
- In the invention it is established that when the calculated FV value is lower than the FV value threshold, the note analysed is classified as fit and when it is higher then it is classified as unfit.
- Moreover, it is provided that the captured images must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
- In the region of interest selection phase that is carried out to characterise the soiling of notes, a region is selected that is not printed and represents a percentage equal to or greater than a previously established percentage of the total surface of the note. In the preferred embodiment this percentage is equal to or less than 10% of the total surface of the note.
- Furthermore, the invention relates to a device, functioning in accordance with the method described, which is characterised in that it comprises a module for digitally capturing images by reflection, via which at least one digital image of at least one of the faces of the note is captured. It also includes a captured digital image processor, which is configured to process the captured images in accordance with the previously described method.
- Regarding the module for digitally capturing images by reflection, it has been provided that it may be a scanner, digital camera or CCD array (Charge Coupled Device).
- For the purpose of helping to make this specification more readily understandable, a single drawing constituting an integral part of the same has been included below, wherein the subject matter of the invention is represented by way of illustrative and non-limiting examples.
-
Figure 1 .- Shows a schematic representation of the phases that make up the method of the invention. - For the purpose of helping to make the invention more readily understandable, an exemplary embodiment of the invention is described below.
- As mentioned above, the invention relates to a method and a device that enables the characterisation of the soiling of banknotes and their classification as fit and unfit for circulation.
- Regarding the method, it should be noted that in the state of the art it is known that in order to determine the state of soiling of banknotes and classify them as fit or unfit for circulation, images of the notes to be analysed are captured (1) by reflection. Only the front, only the back or the front and back may be captured via the use of a scanner, camera, CCD array (Charge Coupled Device) or any other type of module for digitally capturing images that forms part of the device of the invention. In the exemplary embodiment, the image of the front of the note to be analysed is captured and the captured image of each note must have a sufficient resolution, the recommended minimum being 150 dpi (Dots Per Inch), a minimum colour depth of 24 bits per pixel and must be in a lossless compression format.
- The captured image is stored (2) in a standard lossless compression format and colour correction (3) of the captured image is carried out using the ICC colour profile of the module used for capturing the image.
- It is known that the ICC profile of the capturing module is built via a calibration process by capturing a calibration card and comparing the result to the reference values provided by the calibration card supplier, for which reason it is not described in greater detail.
- The object of the colour correction phase is to guarantee the repetitiveness and reproducibility of the method and the device, since the variations due to the capturing module are minimised as a result of this correction. Via this correction it is achieved that images of the same note originating from different capturing modules, or from the same module but captured at different times, are similar.
- One or several regions of interest are selected (16) on the colour corrected image. In the exemplary embodiment of the invention, a single, rectangular-shaped region of interest comprising the watermark area of the note, which is the area where the paper is not printed, is selected. In any case, more than one region of interest and non-rectangular shapes may be used.
- The analysis (4) in the spatial domain of the selected region of interest is then carried out. In the case where more regions of interest are selected, said analysis in the spatial domain is carried out on said selected regions of interest.
- In the case where a region of interest is not defined, the analysis is carried out on the entire surface of the note.
- In order to carry out the analysis (4) in the spatial domain, the colour corrected image is split (4a) into the three channels: Red (R), Green (G) and Blue (B). A number of statistics selected from between the arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient are calculated (4b) in each one of these channels. In the exemplary embodiment of the invention, the analysis is carried out for a €50 note and it has been experimentally determined, as shall be explained below, that a good characterisation of the state of use is obtained for the analysis in the spatial domain by calculating the mean in the green channel and the standard deviation of the pixels contained in the region of interest in the blue channel. In the case where more than one region of interest has been defined, these calculations are carried out on the pixels contained in these regions of interest and if a region of interest has not been defined these calculations are done for all the pixels of the image.
- In the exemplary embodiment, the colour corrected image is 24 bits per pixel and is split into the three RGB channels, each one of which has 8 bits per pixel. Furthermore, the calculations carried out for the exemplary embodiment were obtained as a mean value in the green channel = 45.724 and a standard deviation value in the blue channel = 8.142.
- The analysis (5) in the frequency domain is then carried out on the green channel, for which reason the Fast Fourier Transform or FFT is applied (5a), thus obtaining a new image that is referred to as Power Spectrum. In turn, one or several regions of interest of any shape may be defined on the Power Spectrum, at least one statistic from between all of the aforementioned in the area or areas of interest of the Power Spectrum being calculated (5b). In the exemplary embodiment an area of interest of the region of interest has not been defined, for which reason this calculation is made on all the pixels of the Power Spectrum and it has been determined experimentally, as shall be explained below, that a good characterisation of the state of use is obtained via the calculation of the standard deviation of the value of the pixels in the image that is obtained when the Fast Fourier Transform is applied on the image of the region of interest.
- Specifically in the exemplary embodiment, the value of the standard deviation in the power spectrum = 19.436.
- In the case that no region of interest in the colour corrected images is defined, the FFT is applied to the entire image of the note.
- In order to experimentally establish the statistics that must be calculated, a visual and individual classification (6) is carried out beforehand of a set of notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification (6) is carried out by a group of experts paying particular attention to the region of interest.
- Furthermore, individual images of the set of notes previously assessed visually are captured (7) and analysed (8), in all the RGB channels, in the spatial and frequency domain, the individual images of the notes visually assessed, and all the statistics mentioned above (arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient) are obtained.
- The correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed (6) notes is then analysed (9) in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note and the minimum values are taken as reference values. In the exemplary embodiment, the statistics calculated that offer a greater correlation correspond to those specified above; this is mean deviation in the green channel, standard deviation in the blue channel and standard deviation in the power spectrum of the green channel.
- In the exemplary embodiment, the calculated statistics of the reference notes are: mean in the green channel = 39.440, standard deviation in the blue channel = 6.940, and standard deviation in the power spectrum = 10.183.
- A number of reference values (12) are obtained from the statistics with greater correlation of the point above, which corresponds to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of a set of new notes with the same denomination as the note to be analysed are captured (10), the same region of interest is selected as in the notes to be analysed, the selected statistics that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12).
- Then, at least one statistic obtained via the analysis in the spatial domain is combined (13) with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single value referred to as FV, in the calculation of which the statistics of the analysis (9) described in the previous point that offer a greater correlation in the classification carried out visually by the group of experts is used.
- The FV value is calculated as a distance, from the note object of the analysis to a perfectly clean note taken as a reference, mentioned above, via the equation:
- The value of the exponent n is obtained experimentally such that the coincidence percentage between the visual classification carried out and the analysis of said visually classified notes is maximised, as described above.
- In the exemplary embodiment the three aforementioned statistics have been selected and, therefore, m = 3, which was indicated, is obtained experimentally (green channel mean, blue channel standard deviation and standard deviation of the FFT of the green channel). Normally, m is comprised between 2 and 4, since the FV calculation increases greatly in complexity with a greater number of statistics and the results in the classification of the notes as fit or unfit for circulation improves very little.
- In this example, the scale correction factors Ci are 3.37, 107.88, and 26.96, which are obtained conventionally, and the exponent n = 2.4.
- Therefore, in accordance with the exemplary embodiment, the FV = (3.37*(45.724 - 39.440)2.4 + 107.88* (8.142 -6.940)2.4 + 26.96* (19.436 - 10.183)2.4)1/2.4 = 62.66.
- Moreover, an FV value threshold is fixed (14), which in the case of the €50 notes of the exemplary embodiment is 50.
- Lastly, the calculated FV is compared (15) to the FV threshold, in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
- In accordance with the exemplary embodiment, if the value obtained for the note, object of the study, is higher than the threshold value, the note is unfit to continue in circulation, and if not, it is fit to continue in circulation. In the case at hand, the FV value obtained is 62.66, which is greater than the FV value threshold of 50 and, as such, the note analysed is unfit to continue in circulation.
- In accordance with the description carried out for the FV calculation, all or part of the measurements obtained via the analysis in the spatial domain and frequency domain are taken, and those that are used and those that are rejected depends on the features of the note analysed and particularly the colour of the paper. In any case, the minimum to be used in the FV calculation is a mean or deviation originating from the analysis in the spatial domain and a mean or standard deviation originating from the analysis in the frequency domain.
- Furthermore, the invention relates to a device that, as described, comprises a module for capturing images by reflection (scanner, etc.) that is completed with a captured digital image processor that is configured to process the captured digital image in accordance with the previous described phases of the method.
Claims (10)
- A METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, comprising:- capturing (1) by reflection at least one image of at least one of the faces of the banknote to be analysed, via a digital module for capturing images by reflection,- storing (2) the captured image in a standard format,- correcting (3) the colour of the captured image by using the ICC colour profile of the module used for capturing the image,- analysing (4) the captured image in the spatial domain in accordance with the following phases:characterised in that it further comprises:∘ splitting (4a) each captured image into the three RGB channels: Red, Green and Blue,∘ calculating (4b) at least one statistic selected from between arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient in at least one of the three R, G and B channels, and on the value of the pixels comprising said captured image;- analysing (5) the captured image in the frequency domain in accordance with the following phases:∘ applying (5a) the mathematical transformation known as Fast Fourier Transform (FFT) to the captured image in at least one of the R, G and B channels in order to obtain a new image referred to as "power spectrum",∘ calculating (5b) at least one statistic selected from between arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient on the pixels of the captured image of the power spectrum,- visually and individually assessing (6) beforehand, as fit or unfit for circulation, a set of notes with the same denomination as the notes to be analysed, by a group of experts, paying particular attention to the region of the captured image of the note to be analysed,- capturing (7) the individual images of the set of notes assessed visually beforehand, where said images correspond to the captured images of the note to be analysed,- analysing (8), in all the RGB channels and the spatial and frequency domains, the individual images of the visually assessed notes and obtain all the aforementioned statistics,- analysing (9) the correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed notes in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note,- combining (13) at least one statistic obtained via the analysis in the spatial domain with at least one statistic obtained via the analysis in the frequency domain, of the note being analysed, in order to calculate a single numerical value, referred to as FV (Fitness Value), wherein in this calculation, the statistics of the analysis (9) described in the previous point offering a greater correlation in the visual classification (6) carried out by the group of experts are used,- obtaining a number of reference values (12) from the statistics with a greater correlation of the point above, which correspond to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of the set of new notes with the same denomination as the note to be analysed are captured (10), which correspond to the images captured of the note to be analysed, the statistics selected that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12),- fixing (14) an FV threshold and comparing (15) the calculated FV with the FV threshold in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION according to claim 1, characterised in that the FV is calculated as a distance, from the note object of the analysis to the perfectly clean note taken as a reference, via the equation:
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that, between the colour correction phase (3) of the captured image and the spatial domain analysis phase (4), it comprises selecting (16) at least one region of interest of the captured image for characterising the state of use of the notes, the spatial (4) and frequency (5) analysis only being carried out on said region of interest, where the visual assessment (6) by a group of experts is carried out paying particular attention to the selected region of interest, and where the reference values (12) are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 3, characterised in that it comprises selecting at least one area of the region of interest, from the image referred to as "power spectrum", the spatial (4) and frequency (5) analysis being carried out on said area of the region of interest, where the visual assessment (6) by a group of experts is carried out paying particular attention to the selected area of the region of interest, and where the reference values (12) are obtained by selecting said area of said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that when the calculated FV is lower than the threshold value, the analysed note is classified as fit and when it is higher, it is classified as unfit.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that the at least captured image must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 3, characterised in that the selection of a region of interest for characterising the soiling of notes comprises selecting a region that is not printed and represents a percentage equal to or greater than a percentage previously established of the total surface of the note.
- THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 7, characterised in that the previously established percentage that is not printed must be equal to or less than 10% of the total surface of the note.
- A DEVICE FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, characterised in that it comprises:- a module for digitally capturing images by reflection to capture at least one digital image of at least one of the faces of a note,- a captured digital image processor, configured to function in accordance with the method of claims 1 to 8.
- THE DEVICE FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 9, characterised in that the module for digitally capturing images by reflection is selected from between a scanner, digital camera and CCD array (Charge Coupled Device).
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
ES201430239A ES2549461B1 (en) | 2014-02-21 | 2014-02-21 | METHOD AND DEVICE FOR THE CHARACTERIZATION OF THE STATE OF USE OF BANK TICKETS, AND ITS CLASSIFICATION IN APTOS AND NOT SUITABLE FOR CIRCULATION |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2911123A1 EP2911123A1 (en) | 2015-08-26 |
EP2911123B1 true EP2911123B1 (en) | 2016-07-06 |
Family
ID=52544344
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15155865.7A Active EP2911123B1 (en) | 2014-02-21 | 2015-02-20 | A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation |
Country Status (2)
Country | Link |
---|---|
EP (1) | EP2911123B1 (en) |
ES (2) | ES2549461B1 (en) |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3023694B2 (en) * | 1990-07-27 | 2000-03-21 | セイコーインスツルメンツ株式会社 | Light pattern recognition method for multi-reference images |
GB9413413D0 (en) * | 1994-07-04 | 1994-08-24 | At & T Global Inf Solution | Apparatus and method for testing bank-notes |
US6978036B2 (en) * | 1998-07-31 | 2005-12-20 | Digimarc Corporation | Tamper-resistant authentication techniques for identification documents |
JP4180715B2 (en) * | 1998-12-14 | 2008-11-12 | 株式会社東芝 | Device for determining the degree of contamination of printed matter |
NL1011766C2 (en) * | 1999-04-09 | 2000-10-10 | Nl Bank Nv | Machine for sorting securities. |
US8055064B2 (en) * | 2005-03-30 | 2011-11-08 | National Printing Bureau, Incorporated Administrative Agency | Printed product, printed product detection method and detection apparatus, and authentication method and authentication apparatus |
US20070140551A1 (en) * | 2005-12-16 | 2007-06-21 | Chao He | Banknote validation |
US8096584B2 (en) * | 2006-07-24 | 2012-01-17 | 3M Innovative Properties Company | Document authentication using template matching with fast masked normalized cross-correlation |
NL2006990C2 (en) * | 2011-06-01 | 2012-12-04 | Nl Bank Nv | Method and device for classifying security documents such as banknotes. |
-
2014
- 2014-02-21 ES ES201430239A patent/ES2549461B1/en not_active Expired - Fee Related
-
2015
- 2015-02-20 ES ES15155865.7T patent/ES2591228T3/en active Active
- 2015-02-20 EP EP15155865.7A patent/EP2911123B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
ES2549461R1 (en) | 2015-12-01 |
ES2549461B1 (en) | 2016-10-07 |
ES2591228T3 (en) | 2016-11-25 |
EP2911123A1 (en) | 2015-08-26 |
ES2549461A2 (en) | 2015-10-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11450152B2 (en) | Detection of manipulated images | |
EP2713596B1 (en) | Image evaluation device, image evaluation method and program storage medium | |
TWI334108B (en) | Verification method for determining areas within an image corresponding to monetary banknotes | |
EP2518684B1 (en) | Fake finger determination device | |
KR100542365B1 (en) | Appratus and method of improving image | |
EP2713597B1 (en) | Image evaluation device, image evaluation method and program storage medium | |
CN106326887A (en) | Method and device for checking optical character recognition result | |
KR20110132770A (en) | Forgery detection system for security printing resource based on digital forensic technology and method therefor | |
EP2528037A1 (en) | Image processing device | |
US20110200257A1 (en) | Character region extracting apparatus and method using character stroke width calculation | |
EP3352146B1 (en) | Detection method and apparatus for overlapped notes | |
Ryu et al. | Document forgery detection with SVM classifier and image quality measures | |
EP3249617A1 (en) | Quality control method for paper money authentication and system therefor | |
US11842593B2 (en) | Systems and methods for detection of counterfeit documents | |
EP2911123B1 (en) | A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation | |
JP6534278B2 (en) | Paper sheet discrimination apparatus and paper sheet discrimination method | |
KR101232684B1 (en) | Method for detecting counterfeits of banknotes using Bayesian approach | |
EP4105815A1 (en) | Signal based machine learning fraud detection | |
US10438436B2 (en) | Method and system for detecting staining | |
Funk et al. | Intaglio quality measurement | |
CN108537945B (en) | Bill watermark detection method and system and self-service equipment | |
KR101158329B1 (en) | Apparatus and Method for Extracting Fluorescence Pattern for Automatic Paper Money Inspection | |
US11941901B2 (en) | Smartphone or tablet comprising a device for generating a digital identifier of a copy, including at least one print image of a printed product produced in a production system, and method for using this device | |
NL2011739C2 (en) | Banknote counterfeit classification. | |
CN111627145B (en) | Method and device for identifying fine hollow image-text of image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602015000104 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: G07D0007120000 Ipc: G07D0007187000 |
|
17P | Request for examination filed |
Effective date: 20151217 |
|
RAX | Requested extension states of the european patent have changed |
Extension state: BA Payment date: 20151217 Extension state: ME Payment date: 20151217 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G07D 7/187 20160101AFI20160122BHEP |
|
INTG | Intention to grant announced |
Effective date: 20160218 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 811192 Country of ref document: AT Kind code of ref document: T Effective date: 20160715 Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015000104 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20160706 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: FG2A Ref document number: 2591228 Country of ref document: ES Kind code of ref document: T3 Effective date: 20161125 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 3 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 811192 Country of ref document: AT Kind code of ref document: T Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161106 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161006 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161107 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015000104 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161006 |
|
26N | No opposition filed |
Effective date: 20170407 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 4 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170220 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180228 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20180228 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20150220 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20190220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20190220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20160706 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: ES Payment date: 20230301 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20230208 Year of fee payment: 9 Ref country code: DE Payment date: 20230222 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20231121 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: ES Payment date: 20240301 Year of fee payment: 10 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240226 Year of fee payment: 10 |