EP3598401A1 - Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier - Google Patents

Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier Download PDF

Info

Publication number
EP3598401A1
EP3598401A1 EP17900922.0A EP17900922A EP3598401A1 EP 3598401 A1 EP3598401 A1 EP 3598401A1 EP 17900922 A EP17900922 A EP 17900922A EP 3598401 A1 EP3598401 A1 EP 3598401A1
Authority
EP
European Patent Office
Prior art keywords
image data
banknote
sheet
image
collection unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17900922.0A
Other languages
German (de)
English (en)
Other versions
EP3598401B1 (fr
EP3598401A4 (fr
Inventor
Ryo Ikemoto
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Glory Ltd
Original Assignee
Glory Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Glory Ltd filed Critical Glory Ltd
Publication of EP3598401A1 publication Critical patent/EP3598401A1/fr
Publication of EP3598401A4 publication Critical patent/EP3598401A4/fr
Application granted granted Critical
Publication of EP3598401B1 publication Critical patent/EP3598401B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/16—Testing the dimensions
    • G07D7/162—Length or width
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D11/00—Devices accepting coins; Devices accepting, dispensing, sorting or counting valuable papers
    • G07D11/20—Controlling or monitoring the operation of devices; Data handling
    • G07D11/22—Means for sensing or detection
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/06—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
    • G07D7/12—Visible light, infrared or ultraviolet radiation
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/06—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency using wave or particle radiation
    • G07D7/12—Visible light, infrared or ultraviolet radiation
    • G07D7/121—Apparatus characterised by sensor details
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07D—HANDLING OF COINS OR VALUABLE PAPERS, e.g. TESTING, SORTING BY DENOMINATIONS, COUNTING, DISPENSING, CHANGING OR DEPOSITING
    • G07D7/00—Testing specially adapted to determine the identity or genuineness of valuable papers or for segregating those which are unacceptable, e.g. banknotes that are alien to a currency
    • G07D7/20—Testing patterns thereon

Definitions

  • the present invention relates to sheet detection devices, sheet handling devices, and sheet detection methods.
  • the present invention specifically relates to a sheet detection device, a sheet handling device, and a sheet detection method suitable for detecting the external shape and/or the presence or absence of a sheet.
  • Sheets such as banknotes (printed money), gift vouchers, and checks have a variety of security characteristics for anti-counterfeiting.
  • paper made of vegetable fibers is usually used for sheets
  • paper made of synthetic fibers or a polymer sheet made of synthetic resin may be used in order to improve the properties such as durability, water resistance, and security.
  • Banknotes made of polymer sheets are called polymer banknotes. Polymer banknotes having a transparent portion such as a clear window (transparent window) are difficult to counterfeit.
  • an optical sensor such as an optical line sensor is usually used.
  • a transparent portion transmits light emitted from the optical sensor, and a sheet having a transparent portion may therefore need to undergo different processes from a common sheet having no transparent portion.
  • Patent Literature 2 discloses a banknote image detection device which enables cost reduction.
  • the device includes detection units arranged to sandwich a banknote conveying passage therebetween, with each detection unit including an image detecting sensor and a light emitting means.
  • the image is usually taken by an image sensor module including an optical line sensor.
  • the light such as infrared light, emitted from a light source of the image sensor module passes through a transparent portion of a sheet.
  • a sheet having a transparent portion needs to undergo different processes from a common sheet having no transparent portion.
  • the external shape (outline) of a sheet is first detected (extracted) from the image data acquired by an image sensor module.
  • the image taken by the image sensor module includes not only the sheet but also the background thereof, a region corresponding to the sheet in the overall image needs to be specified and the external shape of the region needs to be extracted.
  • a region corresponding to the sheet may not be correctly extracted from the overall image.
  • a transmission image of a banknote having transparent portions when taken using infrared light, the infrared light may pass through the transparent portions. This may cause assimilation of transparent regions 1001 and 1002, which correspond to the transparent portions, in a medium region 1000, which corresponds to the banknote, into a background region 1003, as illustrated in Fig. 14(a) .
  • Fig. 14(b) when the transmission image is processed such that multiple points on the edges of the medium region 1000 are detected as edge computed points (white dots and black dots in Fig. 14(b) ) corresponding to the edges of the banknote, these edge computed points may include some edge computed points (the black dots in Fig. 14(b) ) inside the actual edges of the banknote. Based on these inappropriate edge computed points, a region 1004 within the medium region 1000 may be computed as the banknote, as illustrated in Fig. 14(c) .
  • the presence of a transparent portion at an edge of a banknote as illustrated in Fig. 14(a) may cause a failure in detecting the edge, possibly causing incorrect detection of passing of the banknote.
  • transmissive light alone of such light of different wavelengths fails to cause correct detection of the external shape and the presence or absence of a variety of sheets having a transparent portion.
  • some characteristics of a material thereof enable detection of the clear window from transmissive light alone of different wavelengths.
  • the combination of wavelengths to be used cannot be fixed in the stage of extracting the external shape at which the kind of document has not yet been determined.
  • the above technique fails to detect clear windows of every kind of documents of different countries or different dates.
  • Patent Literature 2 does not disclose the above technical issue caused by a transparent portion of a sheet, and does not aim to solve the above technical issue.
  • One aspect of the present invention is a sheet detection device comprising: a first image collection unit configured to collect reflection image data of a sheet under transport on a transport path; a second image collection unit configured to collect transmission image data of the sheet under transport on the transport path; and a sheet detector configured to detect at least one of the external shape or the presence or absence of the sheet based on the reflection image data and the transmission image data.
  • the first image collection unit and the second image collection unit share a light source.
  • the sheet detector is configured: to binarize the reflection image data and the transmission image data to generate binarized reflection image data and binarized transmission image data, respectively; to execute OR processing on the binarized reflection image data and the binarized transmission image data to generate OR-processed image data; and to detect at least one of the external shape or the presence or absence of the sheet based on the OR-processed image data.
  • the sheet detector is configured to output outline information of an OR-processed image derived from the OR-processed image data.
  • the sheet detector is configured: to generate a 2D reflection image and a 2D transmission image from the reflection image data and the transmission image data, respectively; and to detect at least one of the external shape or the presence or absence of the sheet based on the 2D reflection image and the 2D transmission image.
  • the sheet detector is configured to detect at least one of the external shape or the presence or absence of the sheet based on data of each line of the reflection image data and data of each line of the transmission image data corresponding to the data of each line of the reflection image data.
  • the first image collection unit and the second image collection unit are configured to apply light of multiple wavelengths including infrared light to the sheet.
  • the sheet has a base material that is a polymer or a composite of paper and a polymer.
  • Another aspect of the present invention is a sheet handling device comprising the sheet detection device.
  • Still another aspect of the present invention is a sheet detection method comprising: a first image collection step of collecting reflection image data of a sheet under transport on a transport path; a second image collection step of collecting transmission image data of the sheet under transport on the transport path; and a sheet detection step of detecting at least one of the external shape or the presence or absence of the sheet based on the reflection image data and the transmission image data.
  • the sheet detection device, the sheet handling device, and the sheet detection method of the present invention can improve the precision of detecting at least one of the external shape or the presence or absence of a sheet having a transparent portion.
  • the banknote to be handled is preferably a polymer banknote having a transparent portion such as a clear window that transmits light such as infrared light applied.
  • a banknote having no transparent portion such as a paper banknote, may also be handled.
  • the transparent portion is preferably made from a synthetic resin (polymer).
  • the banknote to be handled is preferably formed from a polymer sheet.
  • the banknote to be handled may also be a sheet (hybrid banknote) whose transparent portion is formed from a polymer sheet and whose opaque portion is formed from paper made of vegetable fibers or synthetic fibers.
  • the base material of the banknote to be handled is preferably a polymer or a composite of paper and a polymer.
  • the transparent portion may partially include an optically variable device (OVD) such as rainbow hologram.
  • ODD optically variable device
  • the presence or absence of the banknote (medium) is detected based on a reflection image acquired by the image sensor module 15 (step S2).
  • the reflection image is binarized based on a predetermined threshold to generate a binarized reflection image.
  • the pixel values of reflection image data are compared with the predetermined threshold; the pixels with a pixel value of not lower than the threshold are determined as medium-present, and the pixel data thereof are replaced by 1 (white), while the pixels with a pixel value of lower than the threshold are determined as medium-absent, and the pixel data thereof are replaced by 0 (black).
  • the step S1 and the step S2 may be processed in the reversed order, or may be processed simultaneously.
  • a binarized image is generated based on the binarized transmission image acquired in the step S1 and the binarized reflection image acquired by the step S2 (step S3).
  • the binarized transmission image and the binarized reflection image are OR-processed.
  • the corresponding pixels of the binarized transmission image and the binarized reflection image are compared; for the banknote-present (white) pixels in at least one of the images, the pixel data are set to 1 (white), while for the banknote-absent (black) pixels in both images, the pixel data are set to 0 (black).
  • the external shape of the banknote is extracted based on the OR-processed image acquired in the step S3 (step S4). Specifically, a region (partial image region) corresponding to the banknote in the whole OR-processed image including the banknote and the background thereof is specified, and the external shape (outline) of this region is extracted.
  • the transmission image and the reflection image are complementary to each other for extraction of the external shape of a banknote. This can improve the precision of extracting the external shape of a banknote having a transparent portion, and can increase the chance of correctly extracting the external shape of even a banknote having a transparent portion at an edge.
  • the sensor unit 10 has a structure in which a photo sensor 13a, the image sensor module 15, a thickness detection sensor 17, a magnetic sensor module 19, and a photo sensor 13b are arranged in line along the transport path 12 on which banknotes BN are transported.
  • the image sensor module 15, the thickness detection sensor 17, and the magnetic sensor module 19 are sufficiently long relative to the width W of the transport path 12 in the direction perpendicular to the transport direction of banknotes BN, i.e., the main scanning direction, and thus can detect the entire surface of a banknote BN.
  • the transport direction of banknotes BN corresponds to the sub-scanning direction.
  • the sensor unit 10 is provided with a transport mechanism 11 so as to move a banknote BN in the transport path 12.
  • the transport mechanism 11 used may be, but is not limited to, one in which a roller, a belt, or the like is driven by a driver such as a motor.
  • the transport mechanism 11 may be connected with an amount-of-rotation detector such as a rotary encoder, although not illustrated. This enables detection of the distance of transporting a banknote BN from the detected amount of rotation.
  • the image sensor module 15 acquires the image information of the banknote BN.
  • the external shape of the banknote BN is detected based on the acquired image information.
  • the banknote BN may have a transparent portion T that transmits light applied in the image sensor module 15.
  • the photo sensor 13a detects banknotes BN successively transported to the sensor unit 10 and generates banknote detection signals each for determining the timing of starting detection of a banknote BN by the sensor unit 10.
  • the photo sensor 13b detects passing of a banknote BN.
  • the photo sensor 13b detects arrival of a banknote BN
  • the photo sensor 13a detects passing of a banknote BN.
  • the photo sensors 13a and 13b used are light-reflective or light-transmissive photo sensors. Instead of the photo sensors 13a and 13b, sensors that mechanically detect passing of a banknote BN may be provided.
  • the thickness detection sensor 17 detects the thickness of a banknote BN.
  • An example of the thickness detection sensor 17 is one detecting the displacement of rollers facing across the transport path 12 during passing of a banknote BN by sensors provided for the respective rollers.
  • the magnetic sensor module 19 is used to detect the magnetic information contained in a banknote BN under transport on the transport path 12.
  • the magnetic sensor module 19 detects the magnetic information such as a security thread that is a metal or resin narrow band-shaped article or a magnetic ink printed on a banknote BN.
  • the magnetic sensor module 19 is preferably a magnetic sensor in which magnetic detection elements (magnetic heads) are arranged in line.
  • the image sensor module 15 includes an optical line sensor. For example, it includes a light-receiving sensor (image sensor) in which imaging elements such as CCD or CMOS are arranged in line and imaging optical systems such as a light source and a lens.
  • image sensor image sensor
  • the image sensor module 15 detects the image data of a banknote BN under transport on the transport path.
  • the image data may be in an imaged form as described above, or may be in the form of combination of non-imaged coordinates and measured values.
  • the image data used may include transmission image data (transmissive light image data) generated from the intensity distribution of light passed through a banknote BN and reflection image data (reflective light image data) generated from the intensity distribution of light reflected on the banknote BN.
  • the reflection image data used may include at least one of front reflection image data based on the light reflected on the front surface of the banknote BN or back reflection image data based on the light reflected on the back surface of the banknote BN.
  • the transmission image data alone is commonly used for detection of the external shape of a banknote BN. Still, in the present embodiment, the transmission image data and the reflection image data are used.
  • the wavelength of light used for acquisition of image data (imaging) may appropriately selected in accordance with the banknote BN to be imaged. For example, visible light such as single color light of red, green, or blue or white light, infrared light, or ultraviolet light may be used. If necessary, light beams of different spectra may be used to image multiple times.
  • light beams of different spectra may be applied one by one to a banknote BN.
  • the front reflection image data, the back reflection image data, and the transmission image data each may include multiple sets of image data imaged by light beams of different spectra.
  • infrared light is suitable to detect the external shape of a banknote BN and infrared light reflection image data are preferably used. This is because infrared light enables relatively clear imaging of a reflection image even when a banknote BN is soiled or stained, while visible light may provide an entirely black reflection image including the background when a banknote BN is soiled or stained, making it difficult to detect the external shape of the banknote BN.
  • the image sensor modules 15 illustrated in Figs. 5 and 6 each include an upper unit 15A and a lower unit 15B arranged to face each other across the transport path, and include a first image collection unit 15a and a second image collection unit 15b.
  • the upper unit 15A includes a light source 15Aa that applies light to a banknote BN, a condenser 15Ab that condenses reflective light reflected on the banknote BN or transmissive light passed through the banknote BN, a light-receiving sensor 15Ac that receives reflective light or transmissive light condensed by the condenser 15Ab, a transparent plate 15Ad at a lower portion facing the transport path, and a substrate 15Ae on which the light-receiving sensor 15Ac is mounted.
  • the light-receiving sensor 15Ac includes imaging elements (pixels) arranged in line in the direction (main scanning direction) perpendicular to the transport direction of banknotes BN.
  • the lower unit 15B includes a light source 15Ba that applies light to the banknote BN and a transparent plate 15Bd at an upper portion facing the transport path.
  • the first image collection unit 15a includes the light source 15Aa, the condenser 15Ab, and the light-receiving sensor 15Ac.
  • the second image collection unit 15b includes the light source 15Ba, the condenser 15Ab, and the light-receiving sensor 15Ac. As described above, the condenser 15Ab and the light-receiving sensor 15Ac are shared between the first image collection unit 15a and the second image collection unit 15b.
  • the light source 15Aa applies light to a banknote BN, the condenser 15Ab condenses reflective light reflected on the banknote BN, and the light-receiving sensor 15Ac receives reflective light condensed by the condenser 15Ab (first image collection step).
  • the first image collection unit 15a can acquire the reflection image data of the upper surface of the banknote BN.
  • the light source 15Ba applies light to the banknote BN, the condenser 15Ab condenses transmissive light passed through the banknote BN, and the light-receiving sensor 15Ac receives transmissive light condensed by the condenser 15Ab (second image collection step).
  • the second image collection unit 15b can acquire the transmission image data of the banknote BN.
  • the upper unit 15A includes the light source 15Aa that applies light to a banknote BN, the condenser 15Ab that condenses reflective light reflected on the banknote BN, the light-receiving sensor 15Ac that receives reflective light condensed by the condenser 15Ab, the transparent plate 15Ad at a lower portion facing the transport path, and the substrate 15Ae on which the light-receiving sensor 15Ac is mounted.
  • the light-receiving sensor 15Ac includes imaging elements (pixels) arranged in line in the direction (main scanning direction) perpendicular to the transport direction of banknotes BN.
  • the lower unit 15B includes a condenser 15Bb that condenses transmissive light passed through the banknote BN, light-receiving sensors 15Bc that receive transmissive light condensed by the condenser 15Bb, the transparent plate 15Bd at an upper portion facing the transport path, and a substrate 15Be on which the light-receiving sensor 15Bc is mounted.
  • the light-receiving sensor 15Bc includes imaging elements (pixels) arranged in line in the direction (main scanning direction) perpendicular to the transport direction of banknotes BN.
  • the first image collection unit 15a can acquire the reflection image data of the upper surface of the banknote BN (first image collection step).
  • the light source 15Aa applies light to the banknote BN
  • the condenser 15Bb condenses transmissive light passed through the banknote BN
  • the light-receiving sensor 15Bc receives transmissive light condensed by the condenser 15Bb.
  • the second image collection unit 15b can acquire the transmission image data of the banknote BN (second image collection step).
  • the first image collection unit 15a collects the reflection image data of a banknote BN under transport on the transport path and outputs the collected reflection image data to a sensor information acquirer to be described later
  • the second image collection unit 15b collects the transmission image data of a banknote BN under transport on the transport path and outputs the collected transmission image data to the sensor information acquirer to be described later.
  • the first image collection unit 15a and the second image collection unit 15b each collect the image data of the whole banknote BN under transport on the transport path line by line.
  • each image collection unit repeats imaging of a banknote BN under transport in the transport direction (sub-scanning direction) at certain time intervals, with one imaging (exposure of pixel to light) taken as one line, to acquire the image data of the whole banknote BN.
  • a banknote BN may be transported in either the face-up state or the face-down state in any orientation.
  • the denomination and the directions, i.e., either the face-up state or the face-down state and the orientation of a banknote BN are determined by the image information acquired by the image sensor module 15.
  • a banknote recognition device 100 of the present embodiment includes a sensor group that includes the photo sensors 13a and 13b, the image sensor module 15 (the first image collection unit 15a and the second image collection unit 15b), the thickness detection sensor 17, and the magnetic sensor module 19 as illustrated in Fig. 4 , a controller 20 connected with each sensor of the sensor group, and a memory 30 connected with the controller 20.
  • the controller 20 composed of a logical device such as a field programmable gate array (FPGA) includes a sensor information acquirer 21, a banknote detector 22, and a recognizer 23.
  • FPGA field programmable gate array
  • the sensor information acquirer 21 has a function of acquiring data relating to a banknote BN from the sensors constituting the sensor group.
  • the sensor information acquirer 21 also appropriately executes a variety of processes, such as amplification, analog-to-digital conversion (digitization), imaging, image correction, and storage to the memory 30, on the reflection image data and the transmission image data input from the first image collection unit 15a and the second image collection unit 15b.
  • the sensor information acquirer 21 executes processing (averaging processing) of averaging the output values (pixel values) of multiple consecutive pixels (e.g., six pixels) of the reflection image data of each line for every input of the reflection image data of each line from the first image collection unit 15a, and stores the calculated N average values (wherein N is an integer of 2 or greater) in memory regions corresponding to the 1st channel to the Nth channel of the line.
  • the sensor information acquirer 21 also executes averaging processing on the transmission image data of each line input from the second image collection unit 15b for every input of the transmission image data of each line.
  • the averaged resolutions in the main scanning direction of the reflection image data and the transmission image data are both about 33 dpi.
  • the reflection image data and the transmission image data input from the first image collection unit 15a and the second image collection unit 15b each have a total number of lines of M (wherein M is an integer of 2 or greater)
  • the averaged reflection image data and transmission image data are each constituted by N ⁇ M pixel data.
  • the arrangement directions of the channels and the lines respectively correspond to the main scanning direction and the sub-scanning direction.
  • the sensor information acquirer 21 stores the output values (pixel values) of the reflection image data and transmission image data of each line input from the first image collection unit 15a directly in memory regions corresponding to the 1st channel to the N'th channel (wherein N' is an integer satisfying N' > N) of the line.
  • the output of each imaging (exposure of pixel) by the first image collection unit 15a may not be averaged and constitute one pixel value of the reflection image data as it is, or the outputs of multiple consecutive imaging operations (exposure operations of pixel) by the first image collection unit 15a may be averaged for each channel by the sensor information acquirer 21 to constitute one pixel value of the reflection image data.
  • These embodiments may be switched by parameter settings. The same applies to the second image collection unit 15b.
  • the resolutions in the main scanning direction of the reflection image data and the transmission image data are preferably the same as each other, but they are not necessarily the same as each other.
  • the resolutions in the sub-scanning direction of the reflection image data and the transmission image data are also preferably the same as each other, but they are not necessarily the same as each other.
  • the banknote detector 22 includes a binarization processor 22a, an OR processor 22b, and an edge extractor 22c, and detects (extracts) the external shape (outline) of a banknote BN based on the reflection image data of the banknote BN acquired by the first image collection unit 15a and the transmission image data of the banknote BN acquired by the second image collection unit 15b (banknote detection step).
  • a banknote detection device 101 of the present embodiment includes the image sensor module 15 (the first image collection unit 15a and the second image collection unit 15b) and the banknote detector 22. The banknote detector 22 is specifically described later.
  • the recognizer 23 utilizes the data acquired by the sensor information acquirer 21 to execute recognition processing.
  • the recognizer 23 recognizes at least the denomination and authenticity of a banknote BN.
  • the recognizer 23 may have a function of determining the fitness of a banknote BN.
  • the recognizer 23 has a function of detecting defects such as soil, fold, and tear of a banknote BN and detecting material such as tape attached to a banknote BN based on the thickness of the banknote BN, and thereby determining whether the banknote BN is processed as a fit note to be reused in the market or as an unfit note unsuitable to circulation in the market.
  • the recognizer 23 uses an image of a banknote BN taken by the image sensor module 15 for recognition of the information such as the denomination, the authenticity, and the fitness, it utilizes the outline information of the banknote BN acquired by the banknote detector 22. For example, based on the outline information of a banknote BN acquired by the banknote detector 22, the recognizer 23 defines a region corresponding to the banknote BN as a recognition target area within the whole image including the banknote BN and the background thereof, divides the image data within the area into blocks, and executes recognition processing by, for example, pattern matching.
  • the memory 30 is a memory device including, for example, a semiconductor memory or a hard disk, and stores therein determination data 31 for recognition of the information such as the denomination, the authenticity, and the fitness.
  • the determination data 31 includes a variety of templates 31A and a variety of thresholds 31B. Examples of the templates 31A stored include a reference image for comparison with an image of a banknote BN taken by the image sensor module 15 so as to recognize the information such as the denomination, the authenticity, and the fitness, and a reference waveform and a reference image for comparison with a waveform or an image indicating the magnetic properties acquired from a banknote BN.
  • Examples of the thresholds 31B stored include values for determining a variety of characteristic amounts acquired from a banknote BN so as to recognize the information such as the denomination, the authenticity, and the fitness of the banknote BN or to extract the shape of the banknote BN.
  • Predetermined templates 31A and predetermined thresholds 31B are prepared in advance for the respective denominations of the banknotes BN to be handled by the banknote recognition device 100.
  • the memory 30 also stores setting data of the methods for measuring a variety of data for recognition of a banknote BN.
  • the memory 30 is also used to store the image data and the measured values detected by the sensors and the results of recognizing a banknote BN.
  • the banknote detector 22 executes processing of generating a 2D reflection image and a 2D transmission image respectively from the reflection image data and the transmission image data (optionally the averaged reflection image data and the averaged transmission image data), and of detecting the external shape of a banknote BN based on the 2D reflection image and the 2D transmission image, as described in the summary of the banknote detection method.
  • the banknote detector 22 executes processing of detecting the external shape of a banknote BN based on the averaged reflection image data and averaged transmission image data of each line.
  • the binarization processor 22a binarizes the average value based on a predetermined threshold 31B to generate binarized reflection image data, while simultaneously after every averaging processing of the transmission image data of each line, the binarization processor 22a binarizes the average value based on the predetermined threshold 31B to generate binarized transmission image data (binarization processing step).
  • the binarization processor 22a replaces the banknote-present average values with 1 (white), while replacing the medium-absent average values with 0 (black).
  • the OR processor 22b executes OR processing on the binarized reflection image data and binarized reflection image data of the line to generate OR-processed image data (OR processing step).
  • the OR processor 22b compares the same channel of the binarized reflection image data and binarized reflection image data of the line; for the channels having a value of 1 (white) in at least one data, the pixel data thereof are set to 1 (white), while for the channels having a value of 0 (black) in both data, the pixel data thereof are set to 0 (black), whereby OR-processed image data are generated.
  • the edge extractor 22c detects the both edges of a banknote BN in the OR-processed image data of each line (first edge extraction step).
  • the channels corresponding to the respective edges (the left and right edges relative to the transport direction) of the banknote BN in the main scanning direction are detected.
  • the edge extractor 22c detects both edges of the banknote BN in the OR-processed image data of each channel (second edge extraction step).
  • the lines corresponding to the respective edges (the front and back edges relative to the transport direction) of the banknote BN in the sub-scanning direction are detected.
  • the banknote detector 22 outputs the outline information of an OR-processed image derived from the OR-processed image data (outline information output step). Specifically, the banknote detector 22 images all OR-processed image data to generate an OR-processed image, and then specifies a region (partial image region) corresponding to the banknote BN in the OR-processed image based on the left edge channel, right edge channel, front edge line, and back edge line corresponding to the upper, lower, left, and right edges of the banknote BN relative to the transport direction, and then extracts the external shape (outline) of the region.
  • a region partial image region
  • Any method may be used to extract the shape of the partial image region from the channels and the lines corresponding to the upper, lower, left, and right edges, and an example thereof may be Hough transform.
  • This is a technique of computing the straight lines passing any of the channels and lines of the respective sides of a banknote BN and determining the four apexes corresponding to the four corners of the banknote BN. Then, the banknote detector 22 outputs the information of the extracted partial image region, i.e., the outline information of the OR-processed image to the recognizer 23.
  • the banknote detector 22 may detect the presence or absence of a banknote BN based on the reflection image data and the transmission image data. Thereby, the banknote detector 22 can detect even an edge of a banknote BN at which a transparent portion T is present as described above, and thus can correctly detect passing of the banknote BN. As described above, the banknote detection device 101 can suitably be used as a tracking sensor for detecting the presence or absence of a banknote BN under transport.
  • the banknote handling device of the present embodiment may have a structure illustrated in Fig. 11 or Fig. 12 , for example.
  • a banknote handling device 200 illustrated in Fig. 11 includes a hopper 210 capable of supporting a plurality of banknotes, a transport path 211 that transports banknotes supported on the hopper 210, the sensor unit 10 that executes processing of recognizing banknotes, a stacker 213 that accumulates the banknotes recognized by the sensor unit 10, and a rejecter 214 that accumulates banknotes satisfying predetermined conditions separate from the other banknotes.
  • the use of the sensor unit 10 integrated into such a banknote device 200 enables continuous handling of banknotes supported on the hopper 210 and returning of banknotes determined as any of counterfeit notes, unfit notes, and suspect notes to the rejecter 214 for separation.
  • the banknote handling device 200 illustrated in Fig. 11 or the banknote handling device 300 illustrated in Fig. 12 may execute two banknote handling processes; in the first handling, the banknote handling device may determine the denomination and the authenticity to sort the banknotes by the denomination, and in the second handling, the banknote handling device may determine the fitness of the sorted banknotes. Alternatively, the banknote handling device may determine the authenticity of banknotes whose authenticity has been determined at a different site. When the banknote handling device is a device for handling banknotes BN sorted as genuine notes, an authenticity determiner 25b may be omitted.
  • the device includes the first image collection unit 15a that collects reflection image data of a banknote BN under transport on the transport path 12 (first image collection step), the second image collection unit 15b that collects transmission image data of the banknote BN under transport on the transport path 12 (second image collection step), and the banknote detector 22 that detects at least one of the external shape or the presence or absence of the banknote BN based on the reflection image data and the transmission image data (banknote detection step).
  • the transmission image data and the reflection image data are complementary to each other and the device and the method can detect at least one of the external shape or the presence or absence of a banknote BN based on the complementary data. This can improve the precision of detecting at least one of the external shape or the presence or absence of a banknote BN having a transparent portion T.
  • the device and the method can highly precisely detect a banknote BN regardless of the presence or absence of a transparent portion T.
  • the device and the method can be applied to banknotes BN of a variety of countries, and the technique of the above embodiment can be expanded as a standard specification to many countries.
  • the device and the method can highly precisely detect the shape of a banknote BN having a transparent portion T.
  • the device can highly precisely calculate the size of a banknote BN such as the length of a banknote and the parameters relating to the state of the banknote BN under transport such as the skewing angle. This can reduce a decrease in passage of banknotes BN due to rejection and can reduce the chance of miscalculation due to misrecognition.
  • the first image collection unit 15a and the second image collection unit 15b share the light-receiving sensor 15Ac or the light source 15Aa.
  • the former can lead to better cost efficiency than the latter, while the latter can lead to better precision of detecting the external shape of a banknote BN than the former.
  • the banknote detector 22 binarizes the reflection image data and the transmission image data to generate binarized reflection image data and binarized transmission image data, executes OR processing on the binarized reflection image data and the binarized transmission image data to generate the OR-processed image data, and detects at least one of the external shape or the presence or absence of a banknote BN based on the OR-processed image data. This enables more secure detection of at least one of the external shape or the presence or absence of a banknote BN.
  • the banknote detector 22 detects at least one of the external shape or the presence or absence of a banknote BN based on the data of each line of the reflection image data and the data of each line of the transmission image data corresponding to the data of each line of the reflection image data. This can shorten the detection time, and thus can shorten the recognition processing time.
  • the present invention provides a technique useful for detecting the external shape and/or the presence or absence of a sheet.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Toxicology (AREA)
  • Inspection Of Paper Currency And Valuable Securities (AREA)
EP17900922.0A 2017-03-15 2017-03-15 Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier Active EP3598401B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2017/010442 WO2018167876A1 (fr) 2017-03-15 2017-03-15 Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier

Publications (3)

Publication Number Publication Date
EP3598401A1 true EP3598401A1 (fr) 2020-01-22
EP3598401A4 EP3598401A4 (fr) 2020-12-02
EP3598401B1 EP3598401B1 (fr) 2022-01-12

Family

ID=63521956

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17900922.0A Active EP3598401B1 (fr) 2017-03-15 2017-03-15 Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier

Country Status (3)

Country Link
EP (1) EP3598401B1 (fr)
AU (1) AU2017403990A1 (fr)
WO (1) WO2018167876A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022189038A1 (fr) * 2021-03-12 2022-09-15 Giesecke+Devrient Currency Technology Gmbh Dispositif destiné à déterminer la présence d'un objet

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7391668B2 (ja) * 2019-01-11 2023-12-05 グローリー株式会社 画像取得装置、紙葉類処理装置、紙幣処理装置及び画像取得方法
JP7227818B2 (ja) * 2019-03-27 2023-02-22 グローリー株式会社 紙幣識別装置、紙幣処理装置及び紙幣識別方法
CN112509210A (zh) * 2019-08-26 2021-03-16 深圳怡化电脑股份有限公司 纸币图像处理方法、装置、验钞设备及可读存储介质
JP2023135883A (ja) * 2022-03-16 2023-09-29 グローリー株式会社 紙葉類識別装置、紙葉類処理装置及び紙葉類識別方法

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000307819A (ja) * 1999-04-26 2000-11-02 Glory Ltd 多波長光源を用いた画像読取装置及びその制御方法
JP4334913B2 (ja) 2003-05-28 2009-09-30 ローレル精機株式会社 紙幣画像検出装置
JP2006202075A (ja) * 2005-01-21 2006-08-03 Mars Engineering Corp 紙幣識別装置および紙幣識別方法
EP2525332B1 (fr) * 2010-01-12 2019-03-06 Glory Ltd. Dispositif d'identification de feuille de papier et procédé d'identification de feuille de papier
JP5752549B2 (ja) 2011-09-30 2015-07-22 株式会社東芝 光学読取装置および紙葉類処理装置
DE102013006925A1 (de) * 2013-04-22 2014-10-23 Giesecke & Devrient Gmbh Vorrichtung und Verfahren zur Prüfung von Wertdokumenten, insbesondere Banknoten, sowie Wertdokumentbearbeitungssystem
EP2993648B1 (fr) * 2013-04-30 2021-06-09 Glory Ltd. Dispositif d'acquisition d'images et procédé d'acquisition d'images
JP2017027431A (ja) * 2015-07-24 2017-02-02 グローリー株式会社 紙葉類処理機および紙葉類処理方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022189038A1 (fr) * 2021-03-12 2022-09-15 Giesecke+Devrient Currency Technology Gmbh Dispositif destiné à déterminer la présence d'un objet

Also Published As

Publication number Publication date
EP3598401B1 (fr) 2022-01-12
EP3598401A4 (fr) 2020-12-02
WO2018167876A1 (fr) 2018-09-20
AU2017403990A1 (en) 2019-10-31

Similar Documents

Publication Publication Date Title
EP3598401B1 (fr) Dispositif de détection de feuilles de papier, appareil de traitement de feuilles de papier et procédé de détection de feuilles de papier
US8509515B2 (en) Paper sheet identifying device and paper sheet identifying method
US8989433B2 (en) Paper sheet recognition apparatus and paper sheet recognition method
AU2005315600A1 (en) Acceptor device for sheet objects
EP1601599B1 (fr) Detection optique de l'introduction simultanee de deux elements
JP2022129805A (ja) 紙葉類識別装置及び紙葉類識別方法
EP3723054B1 (fr) Unité de reconnaissance de billets de banque, dispositif de manipulation de billets de banque et procédé de reconnaissance de billets de banque
US8265336B2 (en) Paper identifying apparatus and paper identifying method
US8948489B2 (en) Paper sheet recognition device and paper sheet recognition method
KR101479850B1 (ko) 시트류 판별 장치
EP3598400B1 (fr) Dispositif d'acquisition d'image de feuille de papier, dispositif de traitement de feuille de papier et procédé d'acquisition d'image de feuille de papier
EP3680867B1 (fr) Dispositif d'acquisition d'images, dispositif de manipulation de feuilles, dispositif de manipulation de billets de banque et procédé d'acquisition d'images
EP3480794B1 (fr) Système et procédé d'identification de feuille de papier
US20230015962A1 (en) Optical sensor and sheet recognition unit
CN108604397B (zh) 有价票券处理装置及有价票券处理方法
CN108496207B (zh) 有价票券识别装置、有价票券处理机、图像传感器单元及光学可变元件区域的检测方法
EP3467794B1 (fr) Dispositif de collecte d'informations de feuilles de papier et procédé de collecte d'informations de feuilles de papier
JPWO2020208806A1 (ja) 画像取得装置、紙葉類処理装置及び画像取得方法
US20240029497A1 (en) Multifeed detection device and multifeed detection method
JP2023137760A (ja) 紙葉類識別装置、紙葉類処理装置及び紙葉類識別方法
JP2021012587A (ja) 記番号読取装置、紙葉類処理装置、及び記番号読取方法

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20191015

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20201029

RIC1 Information provided on ipc code assigned before grant

Ipc: G07D 11/22 20190101ALI20201023BHEP

Ipc: G07D 7/20 20160101ALI20201023BHEP

Ipc: G07D 7/162 20160101ALI20201023BHEP

Ipc: G07D 7/12 20160101AFI20201023BHEP

Ipc: G07D 7/121 20160101ALI20201023BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G07D 7/12 20160101AFI20210707BHEP

Ipc: G07D 7/20 20160101ALI20210707BHEP

Ipc: G07D 7/162 20160101ALI20210707BHEP

Ipc: G07D 7/121 20160101ALI20210707BHEP

Ipc: G07D 11/22 20190101ALI20210707BHEP

INTG Intention to grant announced

Effective date: 20210810

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017052397

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1462884

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220215

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20220112

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1462884

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220112

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220512

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220412

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220412

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220413

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220512

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602017052397

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220331

26N No opposition filed

Effective date: 20221013

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220315

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220331

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220315

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220331

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221001

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220331

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220112

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240320

Year of fee payment: 8