EP3262562A1 - Système et procédé permettant l'identification de produits dans un panier d'achat - Google Patents

Système et procédé permettant l'identification de produits dans un panier d'achat

Info

Publication number
EP3262562A1
EP3262562A1 EP16707406.1A EP16707406A EP3262562A1 EP 3262562 A1 EP3262562 A1 EP 3262562A1 EP 16707406 A EP16707406 A EP 16707406A EP 3262562 A1 EP3262562 A1 EP 3262562A1
Authority
EP
European Patent Office
Prior art keywords
product
shopping cart
products
data
software
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP16707406.1A
Other languages
German (de)
English (en)
Other versions
EP3262562B1 (fr
Inventor
Frank Heinrich
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Pentland Firth Software GmbH
Original Assignee
Pentland Firth Software GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Pentland Firth Software GmbH filed Critical Pentland Firth Software GmbH
Publication of EP3262562A1 publication Critical patent/EP3262562A1/fr
Application granted granted Critical
Publication of EP3262562B1 publication Critical patent/EP3262562B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/0054—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/0081—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader the reader being a portable scanner or data reader
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62B—HAND-PROPELLED VEHICLES, e.g. HAND CARTS OR PERAMBULATORS; SLEDGES
    • B62B2202/00—Indexing codes relating to type or characteristics of transported articles
    • B62B2202/26—Handbags, e.g. shopping bags
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62B—HAND-PROPELLED VEHICLES, e.g. HAND CARTS OR PERAMBULATORS; SLEDGES
    • B62B2203/00—Grasping, holding, supporting the objects
    • B62B2203/02—Grasping, holding, supporting the objects suspended
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
    • B62B—HAND-PROPELLED VEHICLES, e.g. HAND CARTS OR PERAMBULATORS; SLEDGES
    • B62B2203/00—Grasping, holding, supporting the objects
    • B62B2203/50—Grasping, holding, supporting the objects comprising weighing means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/68—Food, e.g. fruit or vegetables
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/10—Recognition assisted with metadata

Definitions

  • the subject matter described herein relates generally to systems, methods, and computer-readable media for the identification of products in a shopping cart in a retail environment, such as a super market. Particularly, it relates to systems, methods, and computer-readable media for the identification of products by optical image recognition and comparison with a database comprising product specific data, and to an automated checkout of a customer in a shopping environment.
  • Another proposed system is based on the automated recognition or identification of products by image recognition. This is based on comparing images obtained from the chosen products with pre-stored images of the assortment of products in a database.
  • a comprehensive and reliable comparison of a product image against the large number of products, such as 30,000 to 100,000 items that might be carried by a large retailer typically requires more time than would be acceptable for a customer or for the merchant.
  • shorter, more acceptable time spans would tend to render the efficiency and accurateness of the recognition process too low for practical use, because for the method to be ready for productive use, the recognition accuracy should be at least 99 % or more - so that a mis-recognition would occur on average only for a fraction of the customers.
  • Further questions involved pertain to the question of how the images can be taken during the shopping process to yield a sufficient quality for the method, and how this may be combined or integrated with the well-known and expected shopping experience of the customer.
  • a system for the identification of products in a shopping cart in a retail environment comprises: a computer comprising a database with product specific data related to products from an assortment of products in a retail environment; an optical camera connected to the computer, for monitoring the shopping cart; a software stored in a memory of the computer, for identifying products in the shopping cart, which comprises an image recognition software adapted for discerning between different products in the shopping cart, and to determine a product and image areas attributed to it, and to calculate product specific information from the optical data of the image areas; an optional barcode scanner, which may be integrated in the software; wherein the product specific data in the data base comprises signature data derived from previously obtained optical images of the assortment of products, and wherein the software is adapted to identify products in the shopping cart by comparing obtained product specific information of a product in the shopping cart, and wherein the software is adapted to use at least one auxiliary information from at least one further sensing device, in order to improve the accuracy and/or speed of identification of products in the data base
  • a method for the identification of products in a shopping cart in a retail environment comprises: Providing a system according to the first aspect; Providing a database with product specific data related to products of the assortment of products in the retail environment; monitoring the shopping cart with the optical camera and/or a movement sensor; identifying products in the shopping cart, by an image recognition software adapted for discerning between different products in the shopping cart by determining a product and attributing image areas to it, calculating product specific information from the optical data of the image area, and comparing this signature information data with data in the database; optionally reading a barcode on the product with a barcode scanner or via the optical camera; wherein the product specific data in the database comprises signature data derived from the obtained optical images of the assortment of products, using at least one auxiliary information from at least one further sensing device for improving the accuracy and/or speed of identification of products in the data base.
  • FIG. 1 is a schematic view of a system according to embodiments
  • FIG. 2 is a schematic view of a system according to further embodiments.
  • FIG. 3 is a schematic view of a method according to embodiments.
  • FIG. 4 shows details of a system according to embodiments
  • FIG. 5 shows a cross-sectional view of details of a system according to embodiments
  • FIG. 6 shows details of a system applied to a shopping cart, according to further embodiments.
  • Embodiments described herein provide a system and method which serve for reliably identifying products inserted into a shopping cart by a customer in a retail environment, such as a super market.
  • the products are automatically identified during the shopping process, ideally, but not necessarily almost in real time, and the prices and numbers of each product are so determined. Consequently, the checkout may be carried out fully automatic. That is, the customer can for example pay at a cashier - with or without interaction with a human employee - with cash or a credit card or debit card and may receive a printed bill from, e.g., an automated checkout teller.
  • the customer leaves the shop without any interaction, while his smartphone or tablet computer is automatically used for a wireless payment process, for example over NFC via a mobile app.
  • the customer may receive the bill during the payment process, or prior to paying may receive the list of goods in the shopping cart, and can authorize the payment with a simple press of a button or touchscreen.
  • the authorization of the customer may also be carried out by using a computer, such as a tablet computer with a touchscreen, built into or mounted to the shopping cart.
  • the customer may authorize himself before the shopping process, as non-limiting examples only, with an electronic ID, an online account such as a Google account, his/her credit card data (and optionally a PIN code), or internet payment accounts such as, for example, Paypal or the German service Clickandbuy.
  • the loading volume of the shopping cart is supervised by an optical camera mounted to the body of the shopping cart, which is connected to a computer having a central processing unit (CPU) and an internal memory.
  • a software is running on the computer and processes the images transmitted from the at least one optical camera.
  • one or two further optical cameras also supervise the shopping cart from a different perspective, so that the computer is fed with images of the products in the shopping cart from two or three different viewing angles.
  • the software comprises an image processing and recognition software, which is adapted to determine products in the images by analyzing the image information and by attributing defined image areas to a single product (herein also called item).
  • Information extracted from the optical camera images is then compared to product specific information, called signature data, from a database which is stored in the memory of the computer.
  • This database comprises previously derived information (signature data) about each product of the assortment of products of the retail environment, for example a super market, in which the system and methods according to embodiments are employed.
  • this database may comprise from about 30.000 to about 100.000 different products.
  • various features and measures are employed to provide a high reliability of the recognition process of products in the shopping cart, while at the same time delivering a reasonable average recognition time for each item.
  • This is due to the fact that the recognition under the given circumstances is a relatively difficult task - on the one hand, the number of products in the database, to which the current images of the optical camera have to be compared, is up to 100.000 and thus very high, opening possibility for errors as well as requiring a long computation time.
  • the product has to be reliably identified in the image of the whole shopping cart, and the product may be partly obscured by other products, and its position is typically such that only some sides are visible for the optical camera or optical cameras.
  • images from several sides of each product are provided, such that a view from any external viewing perspective on the product is reproducible from the images in the database.
  • the product has a typical cuboid shape
  • images taken from a normal direction to each of the six sides are provided - from left, right, front, rear, top and down.
  • the complete image database is typically not provided in the computer of the shopping cart, but on a server in a backoffice.
  • signature data In the memory of the computer of the shopping cart, only data derived from the original high resolution images of each product is stored as so called signature data.
  • the entire database may also be stored on this computer.
  • additional parameters for each product are provided in the database in embodiments. These may for example comprise a bar code present on the product, and also information about text printed on the surfaces of the product.
  • an optional movement sensor may give a signal to the computer, so that the image identification process is started.
  • the software compares a current image of the shopping cart with the previous image before the new product was added. The calculated delta in the comparison yields the partial image exhibiting the newly added product(s).
  • the software searches if a bar code is visible in the partial image and if so, directly identifies the product in the database.
  • text information is identified in the partial image by image recognition.
  • a character recognition process (OCR) is carried out and the yielded text is compared to entries in the database. This will in many cases directly lead to the identification of the product in the database.
  • a recognition based on the image information is started.
  • the recognition is carried out by comparing so called signature data calculated form the partial image with respective signature data stored in the database.
  • This signature data may in embodiments comprise a variety of different parameters for each product. These are provided in the database and are typically calculated offline when a product is added to the database, typically when it is added to the assortment of products of the retail environment. Also, if the appearance of a product is altered, for example, shape, size, colors, and text information printed thereon, the product is (in an offline-process) again entered into the database, respectively its entry is modified.
  • the various possible signature data parameters may, as non-limiting examples, include: A value indicating the volume of a product; a value about at least one outer dimension of a product; a value representative of the geometrical shape of a product; a value representative of geometrical relations of a product, preferably a relation between at least two of width, height and length; a set of at least two values related to colors of a product, a set of values related to the area which at least one specific color takes up in a product, the percentage that areas with a certain color take up with respect to at least one side of the outer surface of the product; data related to the color taking up the biggest fraction, optionally the second biggest fraction, etc. of at least one side of the outer surface of the product. It is understood that further parameters may be employed or added to the product recognition process, which are regarded to fall under the scope of the present disclosure.
  • one or all of the above parameters may be stored as signature data for each product in the database.
  • the software may be adapted to use one, several or all of the above parameters for the identification of the newly added product. This may be carried out by sequentially calculating and comparing the individual parameters with the database, until a match with the signatures of a product in the database has been found.
  • a confidentiality interval may be defined defining how many parameters of a product have to individually match with an entry in the database as a prerequisite for the decision of the software that the product is regarded as identified.
  • the computer may be a tablet computer mounted to the shopping cart.
  • the price of the added product is added to the sum which has to be paid at checkout.
  • the sum is transmitted to a payment and authorization process, which was described above.
  • Fig. 1 shows an exemplary shopping cart 1 with components of the system 10 according to embodiments, and the components of the system which are external to the shopping cart.
  • Two optical cameras 15, 16 are mounted to the shopping cart 1 and are connected to the computer 20.
  • the database 22 comprising product information, in the form of signature data and/or reduced image data, is stored in a memory 24 of the computer.
  • the system 10 may optionally comprise a server 31 external to the shopping cart 1.
  • the complete database 33 with high resolution image data on each product of the assortment of products may be provided on the server 31, in this case only signature data and reduced image data on each product is stored in the memory 24 of the computer 20. If the computer 20 has enough memory and computing power, the complete image information database 33 may also be stored in the computer 20 of the shopping cart 1.
  • one optical camera 15 may be sufficient in certain configurations or with small shopping carts 1
  • a second optical camera 16 and even third optical camera may be typically provided in order to improve the recognition process.
  • Two optical cameras 15, 16 may for example be provided at both ends of the handle 26 of the shopping cart, left and right to the computer 20 mounted in the area of the middle of the handle 26.
  • one optical camera 15 is provided at the handle, and the second optical camera 16 at the opposite end of the shopping cart 1, distant from the typical position of the customer 2 at the cart.
  • a barcode scanner device 30 is mounted to the cart.
  • the barcode scanning function may typically be integrated in the software 23 in the memory 24 of the computer 20, and the barcode recognition may also be carried out by analyzing the optical camera image data for respective bar code information.
  • an additional infrared camera 35 may be provided for supervising the shopping cart 1.
  • the software 23 can be adapted to calculate temperature related data of products in the shopping cart from obtained infrared images.
  • a further optical camera (not shown) may be provided on a lower side of the main compartment of the shopping cart 1, in order to supervise the storing compartment intended for huge goods (bottom-of-the-basket), which is often provided below the main compartment.
  • the software 23 may be adapted to identify at least one surface temperature of a newly added or any product in the shopping cart 1 from infrared images taken. The temperature is used as auxiliary information, in conjunction with typical product temperatures in the shopping environment stored in the database 22, in the recognition process.
  • the software can be adapted to recognize areas of varying temperature on a product, and to recognize if a varying temperature distribution is due to a recent temperature imprint from a hand of a human. This information can be used as additional information to the optical images, to determine if a product was recently added to the shopping cart 1.
  • the system 10 can have a location subsystem which comprises a sensor 40 for obtaining information about the location of the shopping cart 1 in the retail environment.
  • the sensor may be an RFID sensor, for example, which receives signals from antennas 45 distributed over the retail environment, which may be connected to the server 31 wirelessly or via a cable system 46.
  • the location subsystem may be employed by various methods known in the art, for example infrared, using additional data from a WIFI system, infrared data, direction finding, etc.
  • the information about the current position may be used for various purposes in embodiments.
  • the database may comprise information about the storing location of each product in the shopping environment. This is used in conjunction with the current location of the shopping cart 1 to determine a limited choice / group of products from the database 22, to which a product 50, 51, 52 newly added to the shopping cart 1 is compared by the software 23 in a first approach of the identification process.
  • the database 22 may comprise information which indicates, for a number of locations in the retail environment, which products have often been added to the shopping cart 1 at the respective locations in the past. This information is used in conjunction with the actual location of the shopping cart 1 to determine a limited choice of products from the database to which a product newly added to the shopping cart 1 is compared by the software 23 in a first approach of the identification process.
  • the database may also comprise information about the colors of a product as signature data.
  • the color data of a newly added product is derived from optical images of the product by reducing the image information by an algorithm This may comprise rasterization of the images, reducing the resolution of the rasterization, calculating characteristic color signatures or color tables for a product and/or for different sides of a product.
  • characteristic signature information about colors may be stored in the database and may be obtained by the computer in the identification process.
  • the product identification process is typically further assisted by using data from the infrared camera.
  • data from the infrared camera can be analyzed by the software for the thermal imprint of a human hand, which will typically show as locally enhanced temperature for products generally having room temperature or lower.
  • newly added products can be identified by analyzing infrared images, respectively this data may be combined with data from the image recognition process described further above.
  • the temperature distribution on the respective product may further be used to estimate, for example, how long the customer carried the product, which may be taken as an indicator - at least as an approximation - in which distance from the cart the product was grabbed.
  • This may in turn be used to delimit the range of products in which is searched - as the products are stored with their location in the shop in the database 22 on computer 20 - to improve product recognition.
  • the average walking speed of the customer during shopping may be accounted for.
  • Images from the infrared camera may also be used to recognize if a person different from the customer puts a product into the cart, by discerning between the spectra of the infrared emissions of a hand in an image. Then, a signal may be displayed or a sound message may be played like "Attention, not your shopping cart” or the like.
  • a typical method for the identification of products is as follows: Beforehand, the database with product specific data related to all products of the assortment of products in the retail environment is provided. Thereby, images are taken and stored in the database 33, and the signature information is calculated for each product.
  • the shopping cart 1 is continuously observed with the optical camera(s) 15, 16 and/or the movement sensor. Newly added products are identified in the shopping cart by the image recognition software, which is adapted for identifying between different products in the shopping cart by determining the image areas belonging to a product and attributing the image areas to it. From that partial image, product specific information (signature data) is calculated from the optical data of the partial image. This signature information is compared with signature data from the database 22.
  • the search for barcode information in the partial image is carried out as one of the first method steps for the product identification.
  • an optical character recognition is carried out on the partial image which was identified as belonging to the last added product.
  • the extracted text information if any, is then compared to signature data in the database 22 in order to identify the product via text printed on its surface. Thereby, text printed in larger fonts can be typically used first for the search, as this has a greater chance to be indicative of the product type indicated by brand name and/or product name. If there is a match in the database with identified text, a cross-check may be conducted by the software using other signature data, for example about product size or other parameters described herein.
  • the software starts to calculate signature data from the optical camera image and compares it with the database 22.
  • "Reliable" can for example be defined such that a defined number of different parameters of signature data are all within a 90% match with stored signature data in the database 22.
  • At least one auxiliary information from at least one further sensing device, apart from the optical camera(s) 15, 16 and barcode scanner device 30, may be used for improving the accuracy and/or speed of identification of products in the database 22.
  • images from an infrared camera 35 may be used as was described further above, to obtain signature data about the temperature of the product.
  • the infrared camera 35 may also be used, optionally in conjunction with the movement sensor, to detect that a human hand is moved into the shopping cart, which may be used as an indicator for a product being added or removed. This may be combined with data from the optical camera.
  • the product specific signature data in the data base may typically comprise at least signature data obtained from the optical camera images beforehand; text phrases on the outside of the product, preferably attributed to specific side surfaces of the product; data about typical surface temperatures of the products; and optionally data about natural radiation of a product, which may optionally be used as additional signature data which is compared with data from an optional radiation sensor.
  • the signature data stored in the data base may typically comprise the following parameters: a value indicating the volume of a product; a value about at least one outer dimension of a product; a value representative of the geometrical shape of a product; a value representative of geometrical relations of a product, preferably a relation between at least two of width, height and length; a set of at least two values related to colors of a product, set of values related to the area which at least one specific color takes up in a product, the percentage that areas with a certain color take up with respect to at least one side of the outer surface of the product; and data related to the color taking up the biggest fraction, optionally the second biggest fraction, etc. of at least one side of the outer surface of the product.
  • the signature data may further comprise data about acoustical properties of the product, such as a sound when the product is put into a cart and hits various other products. Therefore, a microphone may be used to record sounds in the shopping cart as additional identification information.
  • the signature data may comprise data obtained by a fast fourier transformation (FFT) of parts or the complete image data from the optical camera.
  • FFT fast fourier transformation
  • this may be an FFT of a line of the image parallel to and in a certain distance from (identified by the image recognition software) edges of one surface of the product, for example 1 cm, or 2 cm, or in the middle of the surface between two identified edges.
  • This FFT data yields a characteristic signature data for the optical appearance of the product and is highly individual for each product.
  • the respective data has to be stored in the database 22, typically at least one FFT-obtained value for each side of the products, such that a comparison can be made for identification. It is understood that this feature may be greatly modified, while those modifications are regarded to fall under the disclosure herein.
  • the optical camera images, or partial image for the product in question are sent to the server 31 via, e.g., WIFI (WLAN) or another communication technology.
  • the server 31 typically comprises a complete image database 33 with high resolution images of the assortment of products of the retail environment.
  • a search is then carried out using the high resolution images. If even this fails, the process is transmitted to a human operator.
  • the resulting options are described further above and below, usually the operator may solve the problem at his own computer, only as a last resort he may go to the shopping cart with the problematic unidentified product.
  • a weighing device may be provided at the cart. This may be a typical small scale, roughly similar to the type used for home cooking purposes. It may be attached to the upper rim of the cart, for example.
  • the scale may also be realized with a hook on which a plastic bag with the goods has to be placed for weighing, wherein the customer can choose on the display of the computer between various items, such that the price can be determined by the computer employing the scale-provided weight data. It would also be possible to have the customer weigh the goods on a stationary scale, which then prints an adhesive label with a bar code or another machine-readable code, which the customer can attach to the plastic bag with the vegetables/fruits and which may be read by the software of the computer.
  • the identification process may further comprise obtaining information about the location of the shopping cart 1 in the retail environment, using hardware as described further above.
  • information about the typical storing location of products in the retail environment may then be used in conjunction with the current location of the shopping cart to determine a limited choice of products from the database, to which a product newly added to the shopping cart is compared in a first approach.
  • the database may contain information about which products are often added by customers to the shopping cart at the respective locations, which is then used in conjunction with the actual location of the shopping cart to determine a limited choice of products from the database to which a product newly added to the shopping cart is compared by the software in a first approach. While the former is based on the average customer, the computer may also comprise data in a database in its memory about prior shopping activities of the current customer logged in to the computer 20. Thus, the computer may for example indicate to the customer, typically on the screen of the computer 20, that he has formerly added a certain product to the shopping cart 1 at the current location in the shop. This information may also contain a hint that a previously bought product was reduced in price, or that there is a special offer about that product.
  • the previously bought number of the respective product may be indicated.
  • the computer may for example indicate that the customer has usually bought a certain product - e,g., a certain pasta sauce - in conjunction with the choice of another product the customer has just put into the shopping cart, e.g. spaghetti. If the customer agrees by clicking "ok" on a touchscreen of the computer 20, for example, he might also be guided by a map of the shop displayed on the computer screen to the specific product location.
  • the computer indicates which other product(s) might be bought, or might be added to the current shopping list entered by the customer - which, if the recommendations fit the customer's needs at least in a number of cases, can be a significant benefit for the merchant by emotionally binding the customer to his business and also by fostering sales.
  • Another mechanism is the above recommendation mechanism in conjunction with a database different from the one run by the retail environment.
  • the computer 20 may be connected via a WIFI (WLAN) over the server 31 to the internet and to large consumer behavior databases like the bonus systems run by joint organizations of large retail companies.
  • WIFI wireless local area network
  • the customer specific data if present in such a database, opens a variety of options for recommendations based on the individual shopping habits, similar to the recommendation process described above.
  • the location data may also be used to guide the customer, on the basis of a shopping list he has entered in the system, in an optimized path through the retail environment.
  • the computer 20 may also be adapted to show special offers, to recommend products to the customer based on his previous shopping behavior, and the like. It may also be used to guide the customer if he searches a certain product. Also, previous shopping lists may be displayed, and the delta to the actual list entered by the customer may be calculated and shown. The products already in the shopping cart 1 may be displayed on a list on the computer, and the current sum to be paid can be displayed. If the identification of a product fails, the computer 20 may send a signal wirelessly to staff of the shop.
  • an augmented reality view of the cart may be displayed on the computer, and the respective product 50, 51, 52 may be marked in color thereon, or the like.
  • the staff can then manually enter the product code or use a barcode scanner device. Subsequently, there may be an optical signal like a blinking LED on the shopping cart to indicate to staff where a failure occurred.
  • the personnel may then also regard a video of about 5 to 20 seconds length on the screen of computer 20, which shows events before and after the product identification failure occurred, which is typically centered around the addition of a product to the shopping cart.
  • the staff can then manually enter the product code or use the barcode scanner, or the like.
  • the failure may be signaled by the computer 20 of the shopping cart 1 to a control person in a back-office, which can carry out the formerly described activities on a screen of his own personal computer. He may for example remotely log on to the computer 20 of the respective problem-causing shopping cart 1, view the video, and may manually intervene by identifying the product in the video himself and to type a product number into his own computer, or by walking towards the customer and to solve the issue at the computer of the shopping cart. This may even happen without any interaction with the customer.
  • the optical camera(s), infrared camera, movement sensor(s), and barcode scanning device are mounted to the shopping cart or at the shopping cart.
  • the computer such as a tablet computer or generally a typical single-board computer (SBC)
  • a server having the complete image data base is typically provided in a backoffice or the like.
  • a plurality of optical cameras 15 and/or a plurality of infrared cameras 35 are mounted stationary in the retail environment, for example at a ceiling 70, and/or at walls and/or pillars.
  • the computer at each shopping cart may in this case be replaced by a central computer or server 31 in the backoffice 60, the computer at the shopping cart may however also be used for interaction with the customer, about his shopping list and items in the cart, for example, and may thus be operably connected at the cart such as in Fig. 1.
  • the optical and infrared cameras are distributed over the area of the retail environment such that typically, a shopping cart is in the field of view of at least one optical camera, or preferably two or more optical cameras, regardless of the cart's position in the retail environment.
  • the optical cameras may be stationary, while the infrared cameras are mounted to the carts, or vice versa. It is understood that the skilled person may derive a number of variations of the embodiments described, in particular with respect to variations of the placing of the various cameras, which are regarded to fall into the scope of this disclosure.
  • the method of recognition of products is, apart from the above differences of the placing and location of the components of the recognition system, basically the same as in the previous embodiments.
  • the products may already be supervised by the stationary optical cameras when they are taken from the shelf, whereby the location in the shelf may be used in conjunction with previously obtained data - or data stored in a location part of the database in the server - to delimit the range of products with which the chosen product has to be compared in the image recognition process.
  • Fig. 3 shows a method 100 for the identification of products in a shopping cart in a retail environment according to embodiments.
  • the method comprises: Providing a system as described with respect to Fig. 1 or Fig. 2 in a block 110; Providing a database with product specific signature data related to products of the assortment of products in the retail environment in a block 120; monitoring the shopping cart with an optical camera and optionally a movement sensor in a block 130; identifying products in the shopping cart, by an image recognition software adapted for discerning between different products in the shopping cart, by identifying an image area and attributing it to a product, and to calculate signature data from the optical data of the attributed image area in a block 140; comparing the calculated signature data with signature data in the database in a block 150; and using at least one auxiliary information from at least one further sensing device for improving the accuracy and/or speed of identification of products in the data base in a block 160.
  • an age verification method which gives a first approach if a minor wants to buy goods to which he is not entitled to. This applies particularly to alcohol and tobacco products, but depending on national or state legislation, also other products may be affected.
  • an optical camera 15 (like in Fig. 2) takes a picture of every customer - or only when needed - and therefrom, his/her height is estimated by an analysis of the image data by the software. This may typically, but not necessarily, be carried out in a centralized computer such as server 31. If certain values - which may also, e.g., include the body outline and therefor the estimated weight of the person - is below a certain threshold, it is assumed that the person may be a minor.
  • the at least further sensing device comprises at least one of a barcode scanner device, and a movement sensor.
  • the at least one further sensing device may comprise a sensor for obtaining information about the location of the shopping cart in the retail environment.
  • the database in the computer of the shopping cart may in embodiments comprise information about the storing location of products in the shopping environment. This can be used in conjunction with the current location of the shopping cart to determine a limited choice of products from the database, to which a product newly added to the shopping cart is compared by the software in a first approach of an identification process.
  • the data base may comprise information which indicates, for a number of locations in the retail environment, which products are often added to the shopping cart at the respective locations, which is used in conjunction with the actual location of the shopping cart to determine a limited choice of products from the database to which a product newly added to the shopping cart is compared by the software in, for example, a first approach of an identification process.
  • the database may further also comprise information about the colors of a product, respectively the color palette, which is derived from optical images of the product by reducing the image information by an algorithm.
  • This process may comprise rasterization of the images, reducing the resolution of the rasterization, calculating characteristic color signatures or color tables for a product and/or for different sides of a product.
  • the system may further be configured to recognize that a product is newly added to the shopping cart and that is subsequently removed from the shopping cart.
  • Fig. 4 shows a shopping cart 1 as employed in embodiments, wherein the shopping cart 1 comprises at least one extra compartment.
  • the compartment is typically realized as a bag 101.
  • the bag 101 or several such bags are, for example, mounted to an upper rim of the shopping cart.
  • the bag 101 comprises a container body 102, which is typically made of flexible material, such as (synthetic) fabric or the like, and has a mouth or inlet 103. It may have grip elements 104 on mutually opposite edges.
  • the grip elements 104 have an elongated bar 108, from which a handle 106 protrudes upwardly and, laterally, a wing 107, which can straddle the edge (upper rim) of the cart 101.
  • closure means may be provided on the grip elements 104, constituted by hook-shaped portions (not shown) formed on the grip elements 104, for example. It is also possible to provide buttons, a zip fastener or any other element that allows to provide a detachable connection.
  • the bag can be closed easily, by utilizing the presence of the hook-like element provided on the handle, which mates within a seat (not shown) formed on the opposite handle.
  • the cameras etc. as of Fig. 1 are not shown in Figs. 4, 5, and 6 for illustrational purposes only.
  • the bag 101 serves, on the one hand, the purpose that once the customer has paid at checkout - which procedure is described above - he may remove the bag or bags from the shopping cart 1, which spares the effort of reloading the bought items into own bags as done conventionally. Also, the customer may roll the shopping cart to the parking lot and conveniently take the bags out and load them into his parking car.
  • the bag 101 serves a further purpose.
  • the bag is supported by at least one weight sensor 100.
  • it is supported by at least two weight sensors 100, 110 which are attached in the vicinity of, or directly at, an upper rim of the shopping cart 1.
  • the sensor(s) may be attached to the upper rim of the metal frame of the shopping cart, for example.
  • the weight of the bag may be continuously monitored.
  • the weight of the item can be determined by a comparison before and after.
  • the data about the weight may be used by the computer 20 as an additional parameter for the product identification process, as the weight has a great significance to discern between products which may, for example, share some optical features and are thus difficult to discern with the conventional optical-based method as described further above.
  • overall identification accuracy can be further enhanced by using the weight as a further parameter.
  • this may be added as a method step to the method shown in Fig. 3.
  • the bag 101 may in embodiments also have a rigid structure, hence be a box, but have nearly the same dimensions and properties as above.
  • magnets 120, 130 may optionally be employed close to lower corners (e.g. at the four lower corners) or edges of the bags as shown in Fig. 4.
  • the magnets may be magnetically attached to counterparts 121, 131 comprising a ferromagnetic material, mounted to the inner side walls - or bottom - of the shopping cart 1.
  • the bag 101 comprises a net of thin wires, which are, for example, woven into the fabric.
  • the wires are connected to a battery, for example located in the bottom of the shopping cart 1.
  • a contacts which may be integrated in the weight sensors 100, 110 described above, or with the magnets 120, 130, as examples, the wire mesh is subjected to a current and will heat up slightly.
  • the resistance of the wires and the battery are adapted such that the wires may heat up for some degrees over ambient temperature, for example from 5 degrees to 25 degrees, more typically from 7 degrees to 15 degrees.
  • the infrared camera supervises the bag 101 in the shopping cart 1, it will recognize the heated wires clearly, and any product being placed in the bag 101 will obstruct a part of the net of wires. This may be used by the computer 20 as an additional information for product identification. Especially cold products from a refrigerator then produce a clear change of the image provided by the infrared camera.
  • the bottom of the shopping cart may be equipped with a pressure-sensing sensor plate 150.
  • the sensor plate, or the individual sensors arranged as the sensor plate, are connected to the computer 20.
  • the computer 20 By checking the signal of the different pressure sensors, which may for example be arranged in a matrix with from 4 x 6, up to 30 x 50 sensors 151 typically covering the whole bottom of the cart, it can be detected where a newly added product has been placed inside the shopping cart. Further, from the quantitative analysis of the sensor signals, the actual weight of a product which was just added may be derived. Thus, consecutively, the weight of each added product may be derived. The weight information can thus be used as additional discriminating information in the product identification process.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Cash Registers Or Receiving Machines (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un système pour l'identification de produits dans un panier d'achat dans un environnement de détail, lequel système comprend : un ordinateur comprenant une base de données avec des données spécifiques à un produit associées à des produits provenant d'un assortiment de produits dans un environnement de détail ; une caméra optique connectée à l'ordinateur, pour surveiller le panier d'achat ; un logiciel stocké dans une mémoire de l'ordinateur, pour identifier des produits dans le panier d'achat, qui comprend un logiciel de reconnaissance d'image conçu pour effectuer une distinction entre différents produits dans le panier d'achat, et déterminer un produit et des zones d'image attribuées à ce dernier, et calculer des informations spécifiques à un produit à partir des données optiques des zones d'image ; un dispositif facultatif de balayage de code-barres, qui peut être intégré dans le logiciel ; les données spécifiques à un produit dans la base de données comprenant des données de signature dérivées à partir d'images optiques obtenues précédemment de l'assortiment de produits, et le logiciel étant conçu pour identifier des produits dans le panier d'achat par comparaison d'informations spécifiques à un produit obtenues d'un produit dans le panier d'achat, et le logiciel étant conçu pour utiliser au moins des informations auxiliaires provenant d'au moins un autre dispositif de détection, de façon à améliorer la précision et/ou la vitesse d'identification de produits dans la base de données. En outre, l'invention concerne un procédé et un produit programme d'ordinateur respectifs.
EP16707406.1A 2015-02-23 2016-02-23 Système et procédé permettant l'identification de produits dans un panier d'achat Active EP3262562B1 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
DE102015102550 2015-02-23
DE102015106659 2015-04-29
PCT/EP2016/053777 WO2016135142A1 (fr) 2015-02-23 2016-02-23 Système et procédé permettant l'identification de produits dans un panier d'achat

Publications (2)

Publication Number Publication Date
EP3262562A1 true EP3262562A1 (fr) 2018-01-03
EP3262562B1 EP3262562B1 (fr) 2022-10-19

Family

ID=55451166

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16707406.1A Active EP3262562B1 (fr) 2015-02-23 2016-02-23 Système et procédé permettant l'identification de produits dans un panier d'achat

Country Status (3)

Country Link
EP (1) EP3262562B1 (fr)
DE (1) DE202016009151U1 (fr)
WO (1) WO2016135142A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4118574A4 (fr) * 2020-03-11 2024-03-06 Gatekeeper Systems, Inc. Surveillance de chariots de supermarché à l'aide de la vision artificielle

Families Citing this family (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017192457A1 (fr) 2016-05-02 2017-11-09 Focal Systems, Inc. Système de paiement de sortie accéléré à l'aide d'unités de commande portables
WO2018057459A1 (fr) * 2016-09-20 2018-03-29 Walmart Apollo, Llc Systèmes et procédés d'identification d'article autonome
CA3040701A1 (fr) * 2016-10-18 2018-04-26 Walmart Apollo, Llc Systemes, dispositifs et procedes de surveillance d'objets dans un chariot
EP3346439A1 (fr) 2017-01-04 2018-07-11 OÜ Virca Système et procédé de fixation de marchandises pouvant être achetées, avec calcul et règlement a posteriori des obligations monétaires dans les magazins
GB2561527A (en) * 2017-01-09 2018-10-24 Jarosz Mateusz One step shopping cart
US20180197218A1 (en) * 2017-01-12 2018-07-12 Verizon Patent And Licensing Inc. System and method for object detection in retail environment
WO2018144650A1 (fr) * 2017-01-31 2018-08-09 Focal Systems, Inc. Système de caisse automatisée par l'intermédiaire d'unités d'achat mobiles
US10410025B2 (en) * 2017-04-19 2019-09-10 Nec Corporation Product checkout and interest detection in retail environment using radio-frequency identification
GB2562095B (en) * 2017-05-05 2020-07-15 Arm Kk An electronic label and methods and system therefor
US10963740B2 (en) 2017-05-12 2021-03-30 Focal Systems, Inc. Automatic labeling of products via expedited checkout system
TR201712155A2 (tr) * 2017-08-16 2018-02-21 Heksagon Muehendislik Ve Tasarim Anonim Sirketi Bi̇r alişveri̇ş si̇stemi̇ ve yöntemi̇
CN109697801A (zh) * 2017-10-23 2019-04-30 北京京东尚科信息技术有限公司 自助结算设备、方法、装置、介质及电子设备
CN107944348A (zh) * 2017-11-07 2018-04-20 杭州盟山网络科技有限公司 基于摄像头的购物系统
JP6330115B1 (ja) * 2018-01-29 2018-05-23 大黒天物産株式会社 商品管理サーバ、自動レジシステム、商品管理プログラムおよび商品管理方法
FR3077404B1 (fr) 2018-01-31 2022-07-15 Afraite Seugnet Mehdi Methodes et systemes d'assistance a l'achat en un point de vente physique
TWI652634B (zh) * 2018-01-31 2019-03-01 緯創資通股份有限公司 自助結帳方法及其系統
DE202018000613U1 (de) 2018-02-07 2019-02-11 Surim Eberlein Von Hand bewegbarer Transportwagen mit einer Wiegeeinrichtung
UA126771C2 (uk) 2018-02-07 2023-02-01 Мартін Еберлайн Візок, що пересувається вручну
DE202018000661U1 (de) 2018-02-09 2019-02-12 Surim Eberlein Von Hand bewegbarer Transportwagen mit einer Wiegeeinrichtung
CN108376448A (zh) * 2018-02-24 2018-08-07 广州逗号智能零售有限公司 商品校验方法及装置
ES2924375T3 (es) * 2018-03-02 2022-10-06 Martin Eberlein Equipo de transporte con al menos un dispositivo de transporte y método para fabricar, operar y mantener el equipo de transporte
DE102018004282A1 (de) 2018-05-29 2019-12-05 Surim Eberlein Transporteinrichtung mit wenigstens einem Transportgerät sowie Verfahren zur Herstellung, zum Betrieb und zur Wartung der Transporteinrichtung
CN110656824A (zh) * 2018-06-28 2020-01-07 南京铭富信息科技有限公司 一种购物车电子锁
JP7156860B2 (ja) * 2018-08-23 2022-10-19 東芝テック株式会社 情報処理装置
CN108734459A (zh) * 2018-08-23 2018-11-02 深圳码隆科技有限公司 一种3d超市商品核验系统
US12033481B2 (en) 2018-09-07 2024-07-09 Gatekeeper Systems, Inc. Shopping basket monitoring using computer vision and machine learning
CN109741552B (zh) * 2019-01-03 2021-06-15 中国联合网络通信集团有限公司 一种购物车、购物系统及方法
AU2020209288A1 (en) * 2019-01-16 2021-08-26 Edi BAHOUS System and methods for automatic detection of product insertions and product extraction in an open shopping cart
FR3094687B1 (fr) * 2019-04-03 2022-07-15 Knap Chariot pour supporter et peser un contenant
JP2022527714A (ja) * 2019-04-03 2022-06-03 クナップ コンテナを支持および計量するための手押し車
JP7410668B2 (ja) * 2019-08-13 2024-01-10 東芝テック株式会社 商品管理装置及びその制御プログラム、セルフ登録システム
US10745039B1 (en) * 2019-08-19 2020-08-18 Sbot Technologies, Inc. Accurate weight measurement system and method operable with a movable device
CN110689677A (zh) * 2019-09-30 2020-01-14 安徽信息工程学院 一种购物车智能开锁装置及其方法
US11017229B2 (en) * 2019-10-25 2021-05-25 7-Eleven, Inc. System and method for selectively verifying algorithmically populated shopping carts
US11023728B1 (en) * 2019-10-25 2021-06-01 7-Eleven, Inc. Machine learning algorithm trained to identify algorithmically populated shopping carts as candidates for verification
US10607080B1 (en) * 2019-10-25 2020-03-31 7-Eleven, Inc. Feedback and training for a machine learning algorithm configured to determine customer purchases during a shopping session at a physical store
DE202021002295U1 (de) 2021-07-03 2021-07-23 Stefan Matthiae System zur Erfassung von Waren in einem Einkaufwagen
KR102433786B1 (ko) * 2021-10-13 2022-08-18 주식회사 케이티 모듈형 전동 카트 및 이를 이용한 원격 작업 지시 방법
AT526299A1 (de) * 2022-07-06 2024-01-15 Manuel Loew Beer Mba Computerimplementiertes Verfahren zur Identifikation eines Artikels, Computerprogrammprodukt, sowie System
DE102022131968A1 (de) 2022-12-02 2024-06-13 Wanzl GmbH & Co. KGaA Einkaufswagen mit Energieversorgung und Wägezelle
WO2024115735A2 (fr) 2022-12-02 2024-06-06 Wanzl GmbH & Co. KGaA Chariot de supermarché à alimentation en énergie et cellule de pesée
DE102022134802A1 (de) 2022-12-24 2024-06-27 KBST GmbH Kontrollverfahren bei digitalen Einkaufswagen
EP4390883A1 (fr) * 2022-12-24 2024-06-26 KBST GmbH Procédé de contrôle pour chariots de supermarché numériques

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7100824B2 (en) * 2004-02-27 2006-09-05 Evolution Robotics, Inc. System and methods for merchandise checkout
US7487913B2 (en) * 2006-07-12 2009-02-10 International Business Machines Corporation Method and system for reducing waste due to product spoilage within a grocery environment
US7988045B2 (en) * 2007-05-31 2011-08-02 International Business Machines Corporation Portable device-based shopping checkout
KR101053366B1 (ko) * 2008-09-08 2011-08-01 주식회사 유로원 쇼핑카트용 가방
US8751318B2 (en) * 2011-05-30 2014-06-10 Lg Electronics Inc. Method for managing and/or controlling store and system for the same
EP2751748B1 (fr) * 2011-08-30 2019-05-08 Digimarc Corporation Procédés et agencements d'identification d'objets

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4118574A4 (fr) * 2020-03-11 2024-03-06 Gatekeeper Systems, Inc. Surveillance de chariots de supermarché à l'aide de la vision artificielle

Also Published As

Publication number Publication date
WO2016135142A1 (fr) 2016-09-01
EP3262562B1 (fr) 2022-10-19
DE202016009151U1 (de) 2023-04-13

Similar Documents

Publication Publication Date Title
EP3262562B1 (fr) Système et procédé permettant l'identification de produits dans un panier d'achat
CN208188867U (zh) 一种用于无人自动售货的商品识别系统
US20200193404A1 (en) An automatic in-store registration system
KR20200051705A (ko) 중량 모니터링 기반의 물품 감지 시스템 및 물품 감지 방법
US10290031B2 (en) Method and system for automated retail checkout using context recognition
CN108780596A (zh) 信息处理系统
US10990945B2 (en) Self-checkout system
JP2019528537A (ja) スマートショッピングカート、スマートショッピングシステム及びその実現方法
JP7257403B2 (ja) 物理的な売り場で購買を支援するための方法及びシステム
EP4439506A2 (fr) Imagerie permettant de réconcilier un écart de poids de chariot
CN107808469A (zh) 自助超市系统
CN112567419A (zh) 使用计算机视觉和人工智能进行的自助产品退还
US20180260799A1 (en) Smart Shopping Object Detection System
CN111899402A (zh) 基于人脸识别的零散物品自助购物结算系统和方法
KR102082988B1 (ko) 무게검증이 가능한 바코드 인식 쇼핑 카트
SE1751401A1 (en) Training of an automatic in-store registration system
CN104778618A (zh) 一种购物储物设备
US20210049575A1 (en) Self-checkout terminal, method and non-transitory computer-readable medium
US10769713B1 (en) Electronic shopping cart
CN209980386U (zh) 一种新型箱式防盗无人收银台
KR20170109989A (ko) 장보기펜을 이용한 무인계산 활용방안
CN207586994U (zh) 无人超市智能采购装置
AU2014240256A1 (en) Method and system for identifying at least one shopping item
KR20030039001A (ko) 쇼핑몰의 무인화 시스템
WO2019245441A1 (fr) Système d'enregistrement automatique en magasin

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170922

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
RIN1 Information on inventor provided before grant (corrected)

Inventor name: HEINRICH, FRANK

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190314

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602016075708

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06K0009000000

Ipc: G06V0020520000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G07G 1/00 20060101ALI20220406BHEP

Ipc: G06V 20/52 20220101AFI20220406BHEP

INTG Intention to grant announced

Effective date: 20220510

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: PENTLAND FIRTH SOFTWARE GMBH

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602016075708

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1526047

Country of ref document: AT

Kind code of ref document: T

Effective date: 20221115

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1526047

Country of ref document: AT

Kind code of ref document: T

Effective date: 20221019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230220

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230119

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230219

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230120

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602016075708

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

26N No opposition filed

Effective date: 20230720

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230223

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230228

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230228

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602016075708

Country of ref document: DE

Representative=s name: TER MEER STEINMEISTER & PARTNER PATENTANWAELTE, DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602016075708

Country of ref document: DE

Representative=s name: TER MEER STEINMEISTER & PARTNER PATENTANWAELTE, DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230223

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20240220

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240222

Year of fee payment: 9

Ref country code: GB

Payment date: 20240222

Year of fee payment: 9

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221019

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240222

Year of fee payment: 9

Ref country code: BE

Payment date: 20240219

Year of fee payment: 9