EP2911123B1 - A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation - Google Patents

A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation Download PDF

Info

Publication number
EP2911123B1
EP2911123B1 EP15155865.7A EP15155865A EP2911123B1 EP 2911123 B1 EP2911123 B1 EP 2911123B1 EP 15155865 A EP15155865 A EP 15155865A EP 2911123 B1 EP2911123 B1 EP 2911123B1
Authority
EP
European Patent Office
Prior art keywords
note
unfit
notes
classification
fit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP15155865.7A
Other languages
German (de)
French (fr)
Other versions
EP2911123A1 (en
Inventor
Eduardo Kropnick Carvajal
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Banco de Espana
Original Assignee
Banco de Espana
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Banco de Espana filed Critical Banco de Espana
Publication of EP2911123A1 publication Critical patent/EP2911123A1/en
Application granted granted Critical
Publication of EP2911123B1 publication Critical patent/EP2911123B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07DHANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/181Testing mechanical properties or condition, e.g. wear or tear
    • G07D7/187Detecting defacement or contamination, e.g. dirt
    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07DHANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/06Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
    • G07D7/12Visible light, infrared or ultraviolet radiation
    • GPHYSICS
    • G07CHECKING-DEVICES
    • G07DHANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/06Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
    • G07D7/12Visible light, infrared or ultraviolet radiation
    • G07D7/1205Testing spectral properties

Definitions

  • the present invention consists of a method and device for characterising the state of use of banknotes and their classification as fit or unfit for circulation, the object of which is to avoid errors in said classification, thus preventing unfit notes from being misclassified as fit and being put back into circulation when they should have been destroyed, and avoiding fit notes from being misclassified as unfit and being prematurely destroyed as a result.
  • One of the jobs carried out by the National Central Banks in relation to cash is to classify the notes that return from circulation by the state of use by separating the notes fit to remain in circulation from those that are not.
  • fit notes are prepared to be put back into circulation and unfit notes are destroyed.
  • This classification by state of use is carried out via the detection and analysis of different features on the notes, such as, for example, the degree of soiling of the notes; the existence of holes and determining their size; the existence of folded corners and/or that they are not there and determining their size; the existence of tears and determining their size; the existence of adhesive tapes and determining their size; the existence of stains and determining their size; etc.
  • Document US 5175775 A may also be acknowledged, this document being related to a method of optical pattern recognition which can be applied to a joint transform correlator operative to effect correlation processing between a plurality of reference images and at least a single object image to produce correlation peaks corresponding to the respective reference images.
  • Document EP 0691632 A1 refers to a method and an apparatus for detecting counterfeit bank-notes, the apparatus including an electronic camera which forms a digital representation of a small area of the bank-note.
  • the digital image is stored in an image memory.
  • the Fourier transform of the digital image is then computed and analyzed to determine if it contains high spatial frequency components, and if so, the note is identified as counterfeit.
  • Document WO 2008/014090 A2 discloses techniques for authenticating security documents having security images that incorporate multiple security features. The techniques may be particularly useful in validating a security document having a security image composed of one or more "virtual" retroreflective images formed over a background of a repeating retroreflective image.
  • document EP 1011079 A1 may also be acknowledged, this document being referred to an apparatus for determining the soil degree of printed matter.
  • the invention proposes a new method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation.
  • the main objective of the invention is to characterise and detect the soiling of notes, although it also enables other features, such as stains and writing or "graffiti" for example, to be detected.
  • the main novelty of the invention lies in that after the analysis in the spatial domain, an analysis in the frequency domain of the captured image is carried out, to which end the mathematical transformation known as Fast Fourier Transform (FFT) is applied to said captured image in at least one of the R, G and B channels, such that a new image, referred to as "power spectrum", is obtained. Furthermore, in order to carry out the analysis in the frequency domain, one or more of the aforementioned statistics is calculated on the pixels of the captured image of the power spectrum.
  • FFT Fast Fourier Transform
  • Carrying out the analysis of the images in the frequency domain provides information that is complementary to the information obtained via traditional analysis in the spatial domain.
  • the method of the invention also comprises visually and individually assessing beforehand a set of used notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification is carried out by a group of experts paying particular attention to the region of the captured image of the note to be analysed.
  • the individual images of the set of used notes that were previously assessed visually are captured, in order to then analyse these images in all the RGB channels, both in the spatial and frequency domain, and obtain all the aforementioned statistics.
  • the captured images must correspond with the captured image of the note to be analysed.
  • At least one statistic obtained via the analysis in the spatial domain is then combined with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single numeric value, referred to as FV (Fitness Value).
  • FV Fitness Value
  • a number of reference values are obtained from the statistics with a greater correlation of the previous point for a note in perfect condition of the same type as the note to be analysed.
  • individual images are captured of a set of new notes with the same denomination as the note to be analysed, which correspond to the captured image of the note to be analysed; the selected statistics that offer a greater correlation in the visual classification are calculated; and the minimum values are taken as the reference values.
  • the invention fixes an arbitrary FV value threshold and, lastly, the calculated FV is compared with the FV threshold, and the note analysed is classified as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
  • the method described avoids the problems mentioned in the state of the art regarding low repetitiveness and reproducibility, crossover error rates and low correlation with the human perception of soiling on notes, since the invention, compared to the systems that currently exist, has high repetitiveness and reproducibility, better FFR and FUR values and a high correlation with the human perception of soiling on notes.
  • an algorithm which is calculated as the root n of the sum of the difference of the values raised to n of the selected statistics with respect to the reference values that correspond to a note in perfect condition with zero level of soiling, is used to obtain the calculated FV.
  • the scale correction factors C i may be obtained conventionally.
  • it comprises carrying out a selection of at least one region of interest of the captured image for characterising the state of use of the notes between the colour correcting phase of the captured image and the spatial domain analysing phase.
  • both the spatial analysis and the frequency analysis are only carried out on said region of interest and similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected region of interest.
  • the reference values are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
  • the invention provides the possibility of selecting, within said region of interest, at least one area of the region of interest from the image referred to as "power spectrum".
  • the spatial and frequency analysis is carried out on said area of the region of interest.
  • the visual assessment by a group of experts is carried out paying particular attention to the selected area of the region of interest, and the reference values are obtained by selecting said areas of said region of interest in the individual images of the set of new notes of the same denomination as the note to be analysed.
  • the captured images must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
  • a region is selected that is not printed and represents a percentage equal to or greater than a previously established percentage of the total surface of the note. In the preferred embodiment this percentage is equal to or less than 10% of the total surface of the note.
  • the invention relates to a device, functioning in accordance with the method described, which is characterised in that it comprises a module for digitally capturing images by reflection, via which at least one digital image of at least one of the faces of the note is captured. It also includes a captured digital image processor, which is configured to process the captured images in accordance with the previously described method.
  • the module for digitally capturing images by reflection it has been provided that it may be a scanner, digital camera or CCD array (Charge Coupled Device).
  • CCD array Charge Coupled Device
  • Figure 1 Shows a schematic representation of the phases that make up the method of the invention.
  • the invention relates to a method and a device that enables the characterisation of the soiling of banknotes and their classification as fit and unfit for circulation.
  • images of the notes to be analysed are captured (1) by reflection. Only the front, only the back or the front and back may be captured via the use of a scanner, camera, CCD array (Charge Coupled Device) or any other type of module for digitally capturing images that forms part of the device of the invention.
  • the image of the front of the note to be analysed is captured and the captured image of each note must have a sufficient resolution, the recommended minimum being 150 dpi (Dots Per Inch), a minimum colour depth of 24 bits per pixel and must be in a lossless compression format.
  • the captured image is stored (2) in a standard lossless compression format and colour correction (3) of the captured image is carried out using the ICC colour profile of the module used for capturing the image.
  • the ICC profile of the capturing module is built via a calibration process by capturing a calibration card and comparing the result to the reference values provided by the calibration card supplier, for which reason it is not described in greater detail.
  • the object of the colour correction phase is to guarantee the repetitiveness and reproducibility of the method and the device, since the variations due to the capturing module are minimised as a result of this correction. Via this correction it is achieved that images of the same note originating from different capturing modules, or from the same module but captured at different times, are similar.
  • One or several regions of interest are selected (16) on the colour corrected image.
  • a single, rectangular-shaped region of interest comprising the watermark area of the note, which is the area where the paper is not printed, is selected.
  • more than one region of interest and non-rectangular shapes may be used.
  • the analysis (4) in the spatial domain of the selected region of interest is then carried out.
  • said analysis in the spatial domain is carried out on said selected regions of interest.
  • the analysis is carried out on the entire surface of the note.
  • the colour corrected image is split (4a) into the three channels: Red (R), Green (G) and Blue (B).
  • a number of statistics selected from between the arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient are calculated (4b) in each one of these channels.
  • the analysis is carried out for a €50 note and it has been experimentally determined, as shall be explained below, that a good characterisation of the state of use is obtained for the analysis in the spatial domain by calculating the mean in the green channel and the standard deviation of the pixels contained in the region of interest in the blue channel. In the case where more than one region of interest has been defined, these calculations are carried out on the pixels contained in these regions of interest and if a region of interest has not been defined these calculations are done for all the pixels of the image.
  • the analysis (5) in the frequency domain is then carried out on the green channel, for which reason the Fast Fourier Transform or FFT is applied (5a), thus obtaining a new image that is referred to as Power Spectrum.
  • the Power Spectrum may be defined on the Power Spectrum, at least one statistic from between all of the aforementioned in the area or areas of interest of the Power Spectrum being calculated (5b).
  • an area of interest of the region of interest has not been defined, for which reason this calculation is made on all the pixels of the Power Spectrum and it has been determined experimentally, as shall be explained below, that a good characterisation of the state of use is obtained via the calculation of the standard deviation of the value of the pixels in the image that is obtained when the Fast Fourier Transform is applied on the image of the region of interest.
  • the value of the standard deviation in the power spectrum 19.436.
  • the FFT is applied to the entire image of the note.
  • a visual and individual classification (6) is carried out beforehand of a set of notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification (6) is carried out by a group of experts paying particular attention to the region of interest.
  • the correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed (6) notes is then analysed (9) in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note and the minimum values are taken as reference values.
  • the statistics calculated that offer a greater correlation correspond to those specified above; this is mean deviation in the green channel, standard deviation in the blue channel and standard deviation in the power spectrum of the green channel.
  • a number of reference values (12) are obtained from the statistics with greater correlation of the point above, which corresponds to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of a set of new notes with the same denomination as the note to be analysed are captured (10), the same region of interest is selected as in the notes to be analysed, the selected statistics that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12).
  • At least one statistic obtained via the analysis in the spatial domain is combined (13) with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single value referred to as FV, in the calculation of which the statistics of the analysis (9) described in the previous point that offer a greater correlation in the classification carried out visually by the group of experts is used.
  • the value of the exponent n is obtained experimentally such that the coincidence percentage between the visual classification carried out and the analysis of said visually classified notes is maximised, as described above.
  • m 3 which was indicated, is obtained experimentally (green channel mean, blue channel standard deviation and standard deviation of the FFT of the green channel).
  • m is comprised between 2 and 4, since the FV calculation increases greatly in complexity with a greater number of statistics and the results in the classification of the notes as fit or unfit for circulation improves very little.
  • an FV value threshold is fixed (14), which in the case of the €50 notes of the exemplary embodiment is 50.
  • the calculated FV is compared (15) to the FV threshold, in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
  • the note if the value obtained for the note, object of the study, is higher than the threshold value, the note is unfit to continue in circulation, and if not, it is fit to continue in circulation.
  • the FV value obtained is 62.66, which is greater than the FV value threshold of 50 and, as such, the note analysed is unfit to continue in circulation.
  • the minimum to be used in the FV calculation is a mean or deviation originating from the analysis in the spatial domain and a mean or standard deviation originating from the analysis in the frequency domain.
  • the invention relates to a device that, as described, comprises a module for capturing images by reflection (scanner, etc.) that is completed with a captured digital image processor that is configured to process the captured digital image in accordance with the previous described phases of the method.

Description

    OBJECT OF THE INVENTION
  • The present invention, as stated in the title of this specification, consists of a method and device for characterising the state of use of banknotes and their classification as fit or unfit for circulation, the object of which is to avoid errors in said classification, thus preventing unfit notes from being misclassified as fit and being put back into circulation when they should have been destroyed, and avoiding fit notes from being misclassified as unfit and being prematurely destroyed as a result.
  • BACKGROUND OF THE INVENTION
  • One of the jobs carried out by the National Central Banks in relation to cash is to classify the notes that return from circulation by the state of use by separating the notes fit to remain in circulation from those that are not.
  • Normally, fit notes are prepared to be put back into circulation and unfit notes are destroyed.
  • This classification by state of use is carried out via the detection and analysis of different features on the notes, such as, for example, the degree of soiling of the notes; the existence of holes and determining their size; the existence of folded corners and/or that they are not there and determining their size; the existence of tears and determining their size; the existence of adhesive tapes and determining their size; the existence of stains and determining their size; etc.
  • The hardest one to measure out of all of them is the degree of soiling of the notes and it is also the one that causes the greatest number of notes to be classified as unfit and, therefore, destroyed.
  • The systems and methods that currently exist are based exclusively on analysing the images of notes in the spatial domain, i.e. on splitting the images into three RGB (Red, Green, Blue) channels and measuring, thereon or on specific combinations of these channels, the mean and standard deviation of the pixels contained within certain regions of interest. As such, the patent document WO 2012165959 regarding a "Method and device for classifying security documents such as banknotes" may be cited.
  • The document above, similarly to the rest of the methods and devices that currently exist, has a series of problems for characterising the soiling of notes and its classification as fit or unfit. Basically, there are three types of problems:
    1. 1. Low repetitiveness and reproducibility: the same note, processed several times in the same system, or in different systems of the same type, produces different results.
    2. 2. Crossover error rates: there is the "False Fit Rate, FFR" (soiled notes misclassified as clean) and the "False Unfit Rate, FUR" (clean notes misclassified as soiled). With the current systems, in order to achieve an FFR below 10%, an FUR above 15 and 20% is obtained. Whilst FFR "contaminates" the notes in circulation with contaminated notes, the FUR is a direct economic loss for the Central Banks as they are notes fit for continued circulation that are being prematurely destroyed.
    3. 3. Low correlation with the human perception of soiling on notes: often, notes classified as contaminated by the current systems are not perceived as such by human beings and vice versa, notes classified as clean have an unacceptable degree of soiling for a human observer.
  • If it is taken into account that National Central Banks throughout the world classify billions of notes each year by state of use, an idea of the scale of the problem represented by the inconsistency of characterising the soiling of notes using the current methods or systems can appreciated.
  • Document US 2003/031340 A1 may be acknowledged. This document discloses techniques helpful in resolving the authentication of an identification document or banknote including a security enhancer.
  • Document US 5175775 A may also be acknowledged, this document being related to a method of optical pattern recognition which can be applied to a joint transform correlator operative to effect correlation processing between a plurality of reference images and at least a single object image to produce correlation peaks corresponding to the respective reference images.
  • Document EP 0691632 A1 refers to a method and an apparatus for detecting counterfeit bank-notes, the apparatus including an electronic camera which forms a digital representation of a small area of the bank-note. The digital image is stored in an image memory. The Fourier transform of the digital image is then computed and analyzed to determine if it contains high spatial frequency components, and if so, the note is identified as counterfeit.
  • Document WO 2008/014090 A2 discloses techniques for authenticating security documents having security images that incorporate multiple security features. The techniques may be particularly useful in validating a security document having a security image composed of one or more "virtual" retroreflective images formed over a background of a repeating retroreflective image.
  • Lastly, document EP 1011079 A1 may also be acknowledged, this document being referred to an apparatus for determining the soil degree of printed matter.
  • DESCRIPTION OF THE INVENTION
  • In order to achieve the objectives and solve the aforementioned drawbacks, the invention proposes a new method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation. The main objective of the invention is to characterise and detect the soiling of notes, although it also enables other features, such as stains and writing or "graffiti" for example, to be detected.
  • It is known that to characterise the soiling of banknotes, methods are used that comprise capturing, by reflection, at least one image of at least one of the faces of a note, via a digital device for capturing images by reflection. The captured image is then stored in a standard format, the colour of the captured image is corrected using the ICC colour profile (colour profile standardised by the International Color Consortium) of the device used to capture the image. The captured image is then analysed in the spatial domain, to which end the image thereof is split into the RGB colour channels and one or more statistics (variables) of the image pixels of the captured image is calculated in at least one of the three RGB channels and on the value of the pixels comprising said captured image. The statistics calculated are selected from between the arithmetic mean, standard deviation, maximum, minimum, median, mode, Kurtosis coefficient and symmetry coefficient.
  • The main novelty of the invention lies in that after the analysis in the spatial domain, an analysis in the frequency domain of the captured image is carried out, to which end the mathematical transformation known as Fast Fourier Transform (FFT) is applied to said captured image in at least one of the R, G and B channels, such that a new image, referred to as "power spectrum", is obtained. Furthermore, in order to carry out the analysis in the frequency domain, one or more of the aforementioned statistics is calculated on the pixels of the captured image of the power spectrum.
  • Carrying out the analysis of the images in the frequency domain provides information that is complementary to the information obtained via traditional analysis in the spatial domain.
  • The method of the invention also comprises visually and individually assessing beforehand a set of used notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification is carried out by a group of experts paying particular attention to the region of the captured image of the note to be analysed.
  • It is provided that the individual images of the set of used notes that were previously assessed visually are captured, in order to then analyse these images in all the RGB channels, both in the spatial and frequency domain, and obtain all the aforementioned statistics. The captured images must correspond with the captured image of the note to be analysed.
  • The correlation between the visual classification carried out by the group of experts and the data obtained in the analysis of the spatial and frequency domain of said individual images of the visually assessed notes is then analysed in order to determine the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note. This correlation analysis is carried out conventionally, for which reason it is not described in greater detail.
  • At least one statistic obtained via the analysis in the spatial domain is then combined with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single numeric value, referred to as FV (Fitness Value). The statistics of the analysis described in the paragraph above, which offer a greater correlation in the classification carried out visually by the group of experts, is used to carry out this calculation.
  • Subsequently, a number of reference values are obtained from the statistics with a greater correlation of the previous point for a note in perfect condition of the same type as the note to be analysed. In order to do so, individual images are captured of a set of new notes with the same denomination as the note to be analysed, which correspond to the captured image of the note to be analysed; the selected statistics that offer a greater correlation in the visual classification are calculated; and the minimum values are taken as the reference values.
  • Moreover, it is provided that the invention fixes an arbitrary FV value threshold and, lastly, the calculated FV is compared with the FV threshold, and the note analysed is classified as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
  • The method described avoids the problems mentioned in the state of the art regarding low repetitiveness and reproducibility, crossover error rates and low correlation with the human perception of soiling on notes, since the invention, compared to the systems that currently exist, has high repetitiveness and reproducibility, better FFR and FUR values and a high correlation with the human perception of soiling on notes.
  • In the preferred embodiment of the invention, the FV is calculated as a distance, from the note object of the analysis to a perfectly clean note that is taken as a reference, as mentioned above, via the equation: FV = ( i = 1 m C i *
    Figure imgb0001
    X i X Ri n ) / n 1 ;
    Figure imgb0002
    where Xi are the selected statistics, XRi are the reference values for those statistics, Ci are the scale correction factors, n is an exponent that consists of a positive number and m is the number of statistics selected in the FV calculation. Therefore, an algorithm, which is calculated as the root n of the sum of the difference of the values raised to n of the selected statistics with respect to the reference values that correspond to a note in perfect condition with zero level of soiling, is used to obtain the calculated FV. The scale correction factors Ci may be obtained conventionally.
  • In one embodiment of the invention, it is provided that it comprises carrying out a selection of at least one region of interest of the captured image for characterising the state of use of the notes between the colour correcting phase of the captured image and the spatial domain analysing phase. In this case, both the spatial analysis and the frequency analysis are only carried out on said region of interest and similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected region of interest. Furthermore, in this case, the reference values are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
  • When at least one region of interest of the captured image is selected, the invention provides the possibility of selecting, within said region of interest, at least one area of the region of interest from the image referred to as "power spectrum". In this case, the spatial and frequency analysis is carried out on said area of the region of interest. Similarly, the visual assessment by a group of experts is carried out paying particular attention to the selected area of the region of interest, and the reference values are obtained by selecting said areas of said region of interest in the individual images of the set of new notes of the same denomination as the note to be analysed.
  • In the invention it is established that when the calculated FV value is lower than the FV value threshold, the note analysed is classified as fit and when it is higher then it is classified as unfit.
  • Moreover, it is provided that the captured images must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
  • In the region of interest selection phase that is carried out to characterise the soiling of notes, a region is selected that is not printed and represents a percentage equal to or greater than a previously established percentage of the total surface of the note. In the preferred embodiment this percentage is equal to or less than 10% of the total surface of the note.
  • Furthermore, the invention relates to a device, functioning in accordance with the method described, which is characterised in that it comprises a module for digitally capturing images by reflection, via which at least one digital image of at least one of the faces of the note is captured. It also includes a captured digital image processor, which is configured to process the captured images in accordance with the previously described method.
  • Regarding the module for digitally capturing images by reflection, it has been provided that it may be a scanner, digital camera or CCD array (Charge Coupled Device).
  • For the purpose of helping to make this specification more readily understandable, a single drawing constituting an integral part of the same has been included below, wherein the subject matter of the invention is represented by way of illustrative and non-limiting examples.
  • DESCRIPTION OF AN EXEMPLARY EMBODIMENT OF THE INVENTION
  • Figure 1.- Shows a schematic representation of the phases that make up the method of the invention.
  • DESCRIPTION OF AN EXEMPLARY EMBODIMENT OF THE INVENTION
  • For the purpose of helping to make the invention more readily understandable, an exemplary embodiment of the invention is described below.
  • As mentioned above, the invention relates to a method and a device that enables the characterisation of the soiling of banknotes and their classification as fit and unfit for circulation.
  • Regarding the method, it should be noted that in the state of the art it is known that in order to determine the state of soiling of banknotes and classify them as fit or unfit for circulation, images of the notes to be analysed are captured (1) by reflection. Only the front, only the back or the front and back may be captured via the use of a scanner, camera, CCD array (Charge Coupled Device) or any other type of module for digitally capturing images that forms part of the device of the invention. In the exemplary embodiment, the image of the front of the note to be analysed is captured and the captured image of each note must have a sufficient resolution, the recommended minimum being 150 dpi (Dots Per Inch), a minimum colour depth of 24 bits per pixel and must be in a lossless compression format.
  • The captured image is stored (2) in a standard lossless compression format and colour correction (3) of the captured image is carried out using the ICC colour profile of the module used for capturing the image.
  • It is known that the ICC profile of the capturing module is built via a calibration process by capturing a calibration card and comparing the result to the reference values provided by the calibration card supplier, for which reason it is not described in greater detail.
  • The object of the colour correction phase is to guarantee the repetitiveness and reproducibility of the method and the device, since the variations due to the capturing module are minimised as a result of this correction. Via this correction it is achieved that images of the same note originating from different capturing modules, or from the same module but captured at different times, are similar.
  • One or several regions of interest are selected (16) on the colour corrected image. In the exemplary embodiment of the invention, a single, rectangular-shaped region of interest comprising the watermark area of the note, which is the area where the paper is not printed, is selected. In any case, more than one region of interest and non-rectangular shapes may be used.
  • The analysis (4) in the spatial domain of the selected region of interest is then carried out. In the case where more regions of interest are selected, said analysis in the spatial domain is carried out on said selected regions of interest.
  • In the case where a region of interest is not defined, the analysis is carried out on the entire surface of the note.
  • In order to carry out the analysis (4) in the spatial domain, the colour corrected image is split (4a) into the three channels: Red (R), Green (G) and Blue (B). A number of statistics selected from between the arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient are calculated (4b) in each one of these channels. In the exemplary embodiment of the invention, the analysis is carried out for a €50 note and it has been experimentally determined, as shall be explained below, that a good characterisation of the state of use is obtained for the analysis in the spatial domain by calculating the mean in the green channel and the standard deviation of the pixels contained in the region of interest in the blue channel. In the case where more than one region of interest has been defined, these calculations are carried out on the pixels contained in these regions of interest and if a region of interest has not been defined these calculations are done for all the pixels of the image.
  • In the exemplary embodiment, the colour corrected image is 24 bits per pixel and is split into the three RGB channels, each one of which has 8 bits per pixel. Furthermore, the calculations carried out for the exemplary embodiment were obtained as a mean value in the green channel = 45.724 and a standard deviation value in the blue channel = 8.142.
  • The analysis (5) in the frequency domain is then carried out on the green channel, for which reason the Fast Fourier Transform or FFT is applied (5a), thus obtaining a new image that is referred to as Power Spectrum. In turn, one or several regions of interest of any shape may be defined on the Power Spectrum, at least one statistic from between all of the aforementioned in the area or areas of interest of the Power Spectrum being calculated (5b). In the exemplary embodiment an area of interest of the region of interest has not been defined, for which reason this calculation is made on all the pixels of the Power Spectrum and it has been determined experimentally, as shall be explained below, that a good characterisation of the state of use is obtained via the calculation of the standard deviation of the value of the pixels in the image that is obtained when the Fast Fourier Transform is applied on the image of the region of interest.
  • Specifically in the exemplary embodiment, the value of the standard deviation in the power spectrum = 19.436.
  • In the case that no region of interest in the colour corrected images is defined, the FFT is applied to the entire image of the note.
  • In order to experimentally establish the statistics that must be calculated, a visual and individual classification (6) is carried out beforehand of a set of notes, with the same denomination as the notes to be analysed, as fit or unfit for circulation. This classification (6) is carried out by a group of experts paying particular attention to the region of interest.
  • Furthermore, individual images of the set of notes previously assessed visually are captured (7) and analysed (8), in all the RGB channels, in the spatial and frequency domain, the individual images of the notes visually assessed, and all the statistics mentioned above (arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient) are obtained.
  • The correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed (6) notes is then analysed (9) in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note and the minimum values are taken as reference values. In the exemplary embodiment, the statistics calculated that offer a greater correlation correspond to those specified above; this is mean deviation in the green channel, standard deviation in the blue channel and standard deviation in the power spectrum of the green channel.
  • In the exemplary embodiment, the calculated statistics of the reference notes are: mean in the green channel = 39.440, standard deviation in the blue channel = 6.940, and standard deviation in the power spectrum = 10.183.
  • A number of reference values (12) are obtained from the statistics with greater correlation of the point above, which corresponds to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of a set of new notes with the same denomination as the note to be analysed are captured (10), the same region of interest is selected as in the notes to be analysed, the selected statistics that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12).
  • Then, at least one statistic obtained via the analysis in the spatial domain is combined (13) with at least one statistic obtained via the analysis of the frequency domain, of the note being analysed, in order to calculate a single value referred to as FV, in the calculation of which the statistics of the analysis (9) described in the previous point that offer a greater correlation in the classification carried out visually by the group of experts is used.
  • The FV value is calculated as a distance, from the note object of the analysis to a perfectly clean note taken as a reference, mentioned above, via the equation: FV = i = 1 m C i * X i X Ri n / n 1 ;
    Figure imgb0003
    where Xi are the selected statistics, XRi are the reference values for these statistics (e.g. the minimum values), Ci are the scale correction factors, the exponent n is a positive number and m is the number of statistics selected in the FV calculation.
  • The value of the exponent n is obtained experimentally such that the coincidence percentage between the visual classification carried out and the analysis of said visually classified notes is maximised, as described above.
  • In the exemplary embodiment the three aforementioned statistics have been selected and, therefore, m = 3, which was indicated, is obtained experimentally (green channel mean, blue channel standard deviation and standard deviation of the FFT of the green channel). Normally, m is comprised between 2 and 4, since the FV calculation increases greatly in complexity with a greater number of statistics and the results in the classification of the notes as fit or unfit for circulation improves very little.
  • In this example, the scale correction factors Ci are 3.37, 107.88, and 26.96, which are obtained conventionally, and the exponent n = 2.4.
  • Therefore, in accordance with the exemplary embodiment, the FV = (3.37*(45.724 - 39.440)2.4 + 107.88* (8.142 -6.940)2.4 + 26.96* (19.436 - 10.183)2.4)1/2.4 = 62.66.
  • Moreover, an FV value threshold is fixed (14), which in the case of the €50 notes of the exemplary embodiment is 50.
  • Lastly, the calculated FV is compared (15) to the FV threshold, in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
  • In accordance with the exemplary embodiment, if the value obtained for the note, object of the study, is higher than the threshold value, the note is unfit to continue in circulation, and if not, it is fit to continue in circulation. In the case at hand, the FV value obtained is 62.66, which is greater than the FV value threshold of 50 and, as such, the note analysed is unfit to continue in circulation.
  • In accordance with the description carried out for the FV calculation, all or part of the measurements obtained via the analysis in the spatial domain and frequency domain are taken, and those that are used and those that are rejected depends on the features of the note analysed and particularly the colour of the paper. In any case, the minimum to be used in the FV calculation is a mean or deviation originating from the analysis in the spatial domain and a mean or standard deviation originating from the analysis in the frequency domain.
  • Furthermore, the invention relates to a device that, as described, comprises a module for capturing images by reflection (scanner, etc.) that is completed with a captured digital image processor that is configured to process the captured digital image in accordance with the previous described phases of the method.

Claims (10)

  1. A METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, comprising:
    - capturing (1) by reflection at least one image of at least one of the faces of the banknote to be analysed, via a digital module for capturing images by reflection,
    - storing (2) the captured image in a standard format,
    - correcting (3) the colour of the captured image by using the ICC colour profile of the module used for capturing the image,
    - analysing (4) the captured image in the spatial domain in accordance with the following phases:
    ∘ splitting (4a) each captured image into the three RGB channels: Red, Green and Blue,
    ∘ calculating (4b) at least one statistic selected from between arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient in at least one of the three R, G and B channels, and on the value of the pixels comprising said captured image;
    characterised in that it further comprises:
    - analysing (5) the captured image in the frequency domain in accordance with the following phases:
    ∘ applying (5a) the mathematical transformation known as Fast Fourier Transform (FFT) to the captured image in at least one of the R, G and B channels in order to obtain a new image referred to as "power spectrum",
    ∘ calculating (5b) at least one statistic selected from between arithmetic mean, standard deviation, median, mode, maximum, minimum, Kurtosis coefficient and symmetry coefficient on the pixels of the captured image of the power spectrum,
    - visually and individually assessing (6) beforehand, as fit or unfit for circulation, a set of notes with the same denomination as the notes to be analysed, by a group of experts, paying particular attention to the region of the captured image of the note to be analysed,
    - capturing (7) the individual images of the set of notes assessed visually beforehand, where said images correspond to the captured images of the note to be analysed,
    - analysing (8), in all the RGB channels and the spatial and frequency domains, the individual images of the visually assessed notes and obtain all the aforementioned statistics,
    - analysing (9) the correlation between the visual classification carried out by the group of experts and the data obtained in the analysis in the spatial and frequency domains of the individual images of said visually assessed notes in order to establish the RGB channels and the statistics offering a greater correlation and a better characterisation of the state of use of the note,
    - combining (13) at least one statistic obtained via the analysis in the spatial domain with at least one statistic obtained via the analysis in the frequency domain, of the note being analysed, in order to calculate a single numerical value, referred to as FV (Fitness Value), wherein in this calculation, the statistics of the analysis (9) described in the previous point offering a greater correlation in the visual classification (6) carried out by the group of experts are used,
    - obtaining a number of reference values (12) from the statistics with a greater correlation of the point above, which correspond to a note in perfect condition of the same type as the note to be analysed, for which reason individual images of the set of new notes with the same denomination as the note to be analysed are captured (10), which correspond to the images captured of the note to be analysed, the statistics selected that offer a greater correlation in the visual classification are calculated (11), and the minimum values are taken as reference values (12),
    - fixing (14) an FV threshold and comparing (15) the calculated FV with the FV threshold in order to classify the note analysed as fit or unfit for circulation, depending on whether said calculated FV is lower or higher than the FV threshold.
  2. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION according to claim 1, characterised in that the FV is calculated as a distance, from the note object of the analysis to the perfectly clean note taken as a reference, via the equation: FV = i = 1 m C i * X i X Ri n / n 1 ;
    Figure imgb0004
    where Xi are the selected statistics, XRi are the reference values for those statistics, Ci are scale correction factors, m is the number of statistics selected in the FV calculation that is obtained experimentally, and the exponent n is a positive number that is obtained experimentally such that the coincidence percentage between the visual classification (6) carried out and the analysis (9) of said visually classified notes is maximised.
  3. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that, between the colour correction phase (3) of the captured image and the spatial domain analysis phase (4), it comprises selecting (16) at least one region of interest of the captured image for characterising the state of use of the notes, the spatial (4) and frequency (5) analysis only being carried out on said region of interest, where the visual assessment (6) by a group of experts is carried out paying particular attention to the selected region of interest, and where the reference values (12) are obtained by selecting said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
  4. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 3, characterised in that it comprises selecting at least one area of the region of interest, from the image referred to as "power spectrum", the spatial (4) and frequency (5) analysis being carried out on said area of the region of interest, where the visual assessment (6) by a group of experts is carried out paying particular attention to the selected area of the region of interest, and where the reference values (12) are obtained by selecting said area of said region of interest in the individual images of the set of new notes with the same denomination as the note to be analysed.
  5. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that when the calculated FV is lower than the threshold value, the analysed note is classified as fit and when it is higher, it is classified as unfit.
  6. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 1, characterised in that the at least captured image must have a resolution of at least 150 dots per inch, with a colour depth of at least 24 bits per pixel and be in a lossless compression format.
  7. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 3, characterised in that the selection of a region of interest for characterising the soiling of notes comprises selecting a region that is not printed and represents a percentage equal to or greater than a percentage previously established of the total surface of the note.
  8. THE METHOD FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 7, characterised in that the previously established percentage that is not printed must be equal to or less than 10% of the total surface of the note.
  9. A DEVICE FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, characterised in that it comprises:
    - a module for digitally capturing images by reflection to capture at least one digital image of at least one of the faces of a note,
    - a captured digital image processor, configured to function in accordance with the method of claims 1 to 8.
  10. THE DEVICE FOR CHARACTERISING THE STATE OF USE OF BANKNOTES, AND THEIR CLASSIFICATION AS FIT AND UNFIT FOR CIRCULATION, according to claim 9, characterised in that the module for digitally capturing images by reflection is selected from between a scanner, digital camera and CCD array (Charge Coupled Device).
EP15155865.7A 2014-02-21 2015-02-20 A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation Active EP2911123B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
ES201430239A ES2549461B1 (en) 2014-02-21 2014-02-21 METHOD AND DEVICE FOR THE CHARACTERIZATION OF THE STATE OF USE OF BANK TICKETS, AND ITS CLASSIFICATION IN APTOS AND NOT SUITABLE FOR CIRCULATION

Publications (2)

Publication Number Publication Date
EP2911123A1 EP2911123A1 (en) 2015-08-26
EP2911123B1 true EP2911123B1 (en) 2016-07-06

Family

ID=52544344

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15155865.7A Active EP2911123B1 (en) 2014-02-21 2015-02-20 A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation

Country Status (2)

Country Link
EP (1) EP2911123B1 (en)
ES (2) ES2549461B1 (en)

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3023694B2 (en) * 1990-07-27 2000-03-21 セイコーインスツルメンツ株式会社 Light pattern recognition method for multi-reference images
GB9413413D0 (en) * 1994-07-04 1994-08-24 At & T Global Inf Solution Apparatus and method for testing bank-notes
US6978036B2 (en) * 1998-07-31 2005-12-20 Digimarc Corporation Tamper-resistant authentication techniques for identification documents
JP4180715B2 (en) * 1998-12-14 2008-11-12 株式会社東芝 Device for determining the degree of contamination of printed matter
NL1011766C2 (en) * 1999-04-09 2000-10-10 Nl Bank Nv Machine for sorting securities.
US8055064B2 (en) * 2005-03-30 2011-11-08 National Printing Bureau, Incorporated Administrative Agency Printed product, printed product detection method and detection apparatus, and authentication method and authentication apparatus
US20070140551A1 (en) * 2005-12-16 2007-06-21 Chao He Banknote validation
US8096584B2 (en) * 2006-07-24 2012-01-17 3M Innovative Properties Company Document authentication using template matching with fast masked normalized cross-correlation
NL2006990C2 (en) * 2011-06-01 2012-12-04 Nl Bank Nv Method and device for classifying security documents such as banknotes.

Also Published As

Publication number Publication date
ES2549461R1 (en) 2015-12-01
ES2549461B1 (en) 2016-10-07
ES2591228T3 (en) 2016-11-25
EP2911123A1 (en) 2015-08-26
ES2549461A2 (en) 2015-10-28

Similar Documents

Publication Publication Date Title
US11450152B2 (en) Detection of manipulated images
EP2713596B1 (en) Image evaluation device, image evaluation method and program storage medium
TWI334108B (en) Verification method for determining areas within an image corresponding to monetary banknotes
EP2518684B1 (en) Fake finger determination device
KR100542365B1 (en) Appratus and method of improving image
EP2713597B1 (en) Image evaluation device, image evaluation method and program storage medium
CN106326887A (en) Method and device for checking optical character recognition result
KR20110132770A (en) Forgery detection system for security printing resource based on digital forensic technology and method therefor
EP2528037A1 (en) Image processing device
US20110200257A1 (en) Character region extracting apparatus and method using character stroke width calculation
EP3352146B1 (en) Detection method and apparatus for overlapped notes
Ryu et al. Document forgery detection with SVM classifier and image quality measures
EP3249617A1 (en) Quality control method for paper money authentication and system therefor
US11842593B2 (en) Systems and methods for detection of counterfeit documents
EP2911123B1 (en) A method and device for characterising the state of use of banknotes, and their classification as fit and unfit for circulation
JP6534278B2 (en) Paper sheet discrimination apparatus and paper sheet discrimination method
KR101232684B1 (en) Method for detecting counterfeits of banknotes using Bayesian approach
EP4105815A1 (en) Signal based machine learning fraud detection
US10438436B2 (en) Method and system for detecting staining
Funk et al. Intaglio quality measurement
CN108537945B (en) Bill watermark detection method and system and self-service equipment
KR101158329B1 (en) Apparatus and Method for Extracting Fluorescence Pattern for Automatic Paper Money Inspection
US11941901B2 (en) Smartphone or tablet comprising a device for generating a digital identifier of a copy, including at least one print image of a printed product produced in a production system, and method for using this device
NL2011739C2 (en) Banknote counterfeit classification.
CN111627145B (en) Method and device for identifying fine hollow image-text of image

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602015000104

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G07D0007120000

Ipc: G07D0007187000

17P Request for examination filed

Effective date: 20151217

RAX Requested extension states of the european patent have changed

Extension state: BA

Payment date: 20151217

Extension state: ME

Payment date: 20151217

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: G07D 7/187 20160101AFI20160122BHEP

INTG Intention to grant announced

Effective date: 20160218

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 811192

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160715

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602015000104

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20160706

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2591228

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20161125

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 3

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 811192

Country of ref document: AT

Kind code of ref document: T

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161106

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161006

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161107

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602015000104

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20161006

26N No opposition filed

Effective date: 20170407

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 4

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170220

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180228

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20150220

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20190220

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20160706

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20230301

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20230208

Year of fee payment: 9

Ref country code: DE

Payment date: 20230222

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231121

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20240301

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240226

Year of fee payment: 10