SE1850757A1 - An automatic in-store registration system - Google Patents
An automatic in-store registration systemInfo
- Publication number
- SE1850757A1 SE1850757A1 SE1850757A SE1850757A SE1850757A1 SE 1850757 A1 SE1850757 A1 SE 1850757A1 SE 1850757 A SE1850757 A SE 1850757A SE 1850757 A SE1850757 A SE 1850757A SE 1850757 A1 SE1850757 A1 SE 1850757A1
- Authority
- SE
- Sweden
- Prior art keywords
- article
- image
- articles
- customer
- registration
- Prior art date
Links
- 238000000034 method Methods 0.000 claims abstract description 118
- 230000008859 change Effects 0.000 claims abstract description 56
- 210000001364 upper extremity Anatomy 0.000 claims abstract description 56
- 230000004044 response Effects 0.000 claims abstract description 10
- 238000012549 training Methods 0.000 claims description 28
- 238000004891 communication Methods 0.000 claims description 18
- 210000003414 extremity Anatomy 0.000 claims description 5
- 230000008569 process Effects 0.000 description 16
- 230000002829 reductive effect Effects 0.000 description 14
- 230000015654 memory Effects 0.000 description 12
- 238000005303 weighing Methods 0.000 description 12
- 238000004422 calculation algorithm Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 6
- 230000005684 electric field Effects 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 201000002266 mite infestation Diseases 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 235000013334 alcoholic beverage Nutrition 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004611 spectroscopical analysis Methods 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000032258 transport Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000013585 weight reducing agent Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/30—Payment architectures, schemes or protocols characterised by the use of specific devices or networks
- G06Q20/32—Payment architectures, schemes or protocols characterised by the use of specific devices or networks using wireless devices
- G06Q20/322—Aspects of commerce using mobile devices [M-devices]
- G06Q20/3224—Transactions dependent on location of M-devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
- G06Q20/203—Inventory monitoring
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B62—LAND VEHICLES FOR TRAVELLING OTHERWISE THAN ON RAILS
- B62B—HAND-PROPELLED VEHICLES, e.g. HAND CARTS OR PERAMBULATORS; SLEDGES
- B62B5/00—Accessories or details specially adapted for hand carts
- B62B5/0096—Identification of the cart or merchandise, e.g. by barcodes or radio frequency identification [RFID]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/08—Payment architectures
- G06Q20/20—Point-of-sale [POS] network systems
- G06Q20/208—Input by product or record sensing, e.g. weighing or scanner processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q20/401—Transaction verification
- G06Q20/4014—Identity check for transactions
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/0036—Checkout procedures
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
- G07G1/00—Cash registers
- G07G1/0036—Checkout procedures
- G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
- G07G1/0081—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader the reader being a portable scanner or data reader
-
- A—HUMAN NECESSITIES
- A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
- A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
- A47F10/00—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for
- A47F10/02—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for for self-service type systems, e.g. supermarkets
- A47F2010/025—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for for self-service type systems, e.g. supermarkets using stock management systems
-
- A—HUMAN NECESSITIES
- A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
- A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
- A47F9/00—Shop, bar, bank or like counters
- A47F9/02—Paying counters
- A47F9/04—Check-out counters, e.g. for self-service stores
Abstract
A method for automatic registration of an article in a store having at least one article containing area (A-1) and an associated automatic in-store registration system (100) is provided. The method comprises associate an article carrying arrangement (5) with a specific registration account (130), wherein the registration account (130) is associated with a dynamic list of articles, continuously monitor the at least one article containing area (A-1) by means of a sensor arrangement (110), and detect a change in the at least one article containing area (A-I) and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list. The method further comprises continuously monitor the article carrying arrangement (5) by means of the sensor arrangement (110), receive a first image from the sensor arrangement (110) of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement (5), determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list and add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account (130).
Description
AN AUTOMATIC IN-STORE REGISTRATION SYSTEM TECHNICAL FIELDThe present invention relates to an automatic in-store registration system being configured to automatically registering articles in a store.
BACKGROUND In today°s stores many different articles, such as food products, hygienearticles, clothing, etc. may be purchased Which all have different sizes and shapes, asWell as visual appearance. Norrnally, a checkout operator handles each article manuallyand makes sure that the article is associated With the correct pricing at checkout forcorrect payment by the customer. This is traditionally done either by manually inputtingthe price, by scanning a barcode attached to the article, or a combination of those.However, semi- or fully automatic checkout counters are becoming an interestingaltemative for retail stores and supermarkets. Such checkout counters provides robustand easy identification and handling of articles and they norrnally include differentcomponents for identifying the article accurately.
However, in prior art systems the customer still needs to perform the criticalstep of feeding the articles through a checkout counter in order for the articles to beidentified. The customer is required to queue in checkout lines and pass each productfrom the shopping cart or basket to a scanning system. Therefore, there is a need for animproved registration system that is fast, reliable, and robust and thus has an increased security level While at the same time being easy and convenient for the customer.
SUMMARY An object of the present invention is therefore to provide an improved methodand system for automatic registration of articles in a store.
In a first aspect, a method for automatic registration of an article in a storehaving at least one article containing area and an associated automatic in-storeregistration system is provided. The method comprises associate an article carrying arrangement With a specific registration account, Wherein the registration account is associated With a dynamic list of articles, continuously monitor the at least one articlecontaining area by means of a sensor arrangement, and detect a change in the at leastone article containing area and, in response to said detected change, and identify one ormore possible articles causing the change in a first identification step and save said oneor more possible articles to a temporary article list. The method further comprisescontinuously monitor the article carrying arrangement by means of the sensorarrangement, receive a first image from the sensor arrangement of at least a portion of acustomer°s upper extremity being captured When at least a part of the customer°s upperextremity is moving across a virtual boundary set at a predeterrnined distance from thearticle carrying arrangement, determine, based at least in part on the first image, if thefirst image includes a representation of an article, and if so, identify said article in asecond identification step from the article identities currently included in the temporaryarticle list, and add the identity of the article Which Was identified in the secondidentification to the dynamic list of articles of the registration account.
In one embodiment the method further comprises the step of receiving a secondimage from the sensor arrangement of at least a portion of a customer°s upper extremitybeing captured When the at least a part of the customer°s upper extremity is movingaway from the virtual boundary set and determine, based at least in part on acomparison of the first image and the second image, if the first image includes arepresentation of an article and if so determine if the second image does not include arepresentation of the article of the first image. If so, identify the article of the first imagein a second identification step from the article identities currently included in thetemporary article list, and add the identity of the article Which Was identified in thesecond identification to the dynamic list of articles of the registration account.
In one embodiment, if it is deterrnined that the first image includes arepresentation of an article and that the second image include a representation of thearticle of the first image, the method further comprises the step of determine if the firstimage includes a representation of an article and that the second image includes arepresentation of the same article as of the first image. And if the first and secondimages includes representation of different articles: identify the article of the first image and add the identity of the article to the dynamic list of articles of the registration account, and identify the article of the second image and remove the identity of thearticle from the dynamic list of articles of the registration account.
In one embodiment, if it is deterrnined that the first image includes arepresentation of an article and that the second image includes a representation of thesame article as of the first image, the article is not added to the dynamic list of articlesof the registration account.
In one embodiment, if it is deterrnined, based at least in part on a comparisonof the first image and the second image, that the first image does not include arepresentation of an article and that the second image includes a representation of anarticle, the method further comprises the step of identifying said article from the secondimage in a second identification step from the article identities currently included in thedynamic list of articles of the registration account, and removing the identity of thearticle Which Was identified in the second identification from the dynamic list of articlesof the registration account.
The method may further comprises the step of receive a third image from thesensor arrangement of the virtual boundary set prior to at least a part of the customer"supper extremity is moving across the virtual boundary set, receive a fourth image fromthe sensor arrangement of the virtual boundary set after to at least a part of thecustomer°s upper extremity is moving away from the virtual boundary set, determine,based at least in part on a comparison of the first, third and fourth image, if the articlehas been added to or removed from the article carrying arrangement, identify said articlein a second identification step from the article identities currently included in thetemporary article list, and Wherein if is deterrnined that the article has been added to thearticle carrying arrangement add the identity of the article Which Was identified in thesecond identification to the dynamic list of articles of the registration account andWherein if is deterrnined that the article has been removed from the article carryingarrangement remove the identity of the article Which Was identified in the secondidentification from the dynamic list of articles of the registration account.
In one embodiment, the first identification step is performed by identifying anarticle by at least using information relating to the position of the change in the article containing area, Wherein the position of each one of the plurality of articles arranged in the article containing area is predeterrnined and accessible for the automatic in-storeregistration system.
The first identification step may be perforrned by reducing the total number ofpossible article identities using the position information of the change in the articlecontaining area. The article containing area may be arranged With a plurality ofsegments, Wherein the position of each segment is known. Each segment of the articlecontaining area may comprise at least one capacitive sensor arranged to detect a changein the article containing area.
In one embodiment, the step of associating the article carrying arrangementWith a specific registration account is performed using Near Field Communication.
In a second aspect, an automatic in-store registration system in a store havingat least one article containing area is provided. The system comprises at least oneprocessor, at least one registration account being associated With an article carryingarrangement, and a sensor arrangement comprising a plurality of sensors, the sensorarrangement being configured to, in a first identification step, continuously monitor thearticle containing area and to detect a change in the article containing area and, inresponse to said detected change, to transmit a first identification signal comprisinginformation relating to one or more possible articles to the processor, and Wherein thesensor arrangement is further configured to, in a second identification step, continuouslymonitor the article carrying arrangement and to detect a change in or near the articlecarrying arrangement and, in response to said detected change, to transmit a secondidentification signal comprising a first image to the processor, Wherein the processor isconfigured to identify one or more possible articles from the first identification signaland to save said articles to a temporary article list, Wherein the processor is furtherconfigured to identify one or more possible articles from the second identification signalby: receiving a first image of at least a portion of a customer°s upper extremity beingcaptured When at least a part of the customer°s upper extremity is moving across avirtual boundary set at a predeterrnined distance from the article carrying arrangement;deterrnining, based at least in part on the first image, if the first image includes arepresentation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list; and Wherein the processor is further configured to determine a final article by comparing the one ormore articles identified from the second identification signal With the articles stored inthe dynamic article list, and to add said final article to the registration account.
In a third aspect, a method for training an in-store registration systemconfigured to identify and register articles in a store having at least one articlecontaining area is provided. The method comprising performing a training sequence inorder to improve the accuracy of the identification of articles by using training data torecognize different movements of a user°s upper extremity While being in or near thearticle containing area and/or the article carrying arrangement, and to detect at least aportion of a upper extremity of a person When at least a part of the person"s upperextremity is moving in or near the article containing area and/or the article carryingarrangement, registering one or more signatures of each article by means of a sensorarrangement, and adding or updating information relating to signature and/or theposition of each article to the in-store registration system.
In one embodiment, the training data is composed of a synthetically generated3D-model of at least a part of the store and/or the customer, Wherein the synthetictraining synthetic data is a 3D-model of at least a part of the store. The synthetic datamay be enhanced using Generative Adversarial Networks.
In a fourth aspect, a method for automatic registration of an article in a storehaving at least one article containing area and an associated automatic in-storeregistration system is provided. The method comprises associate an article carryingarrangement With a specific registration account, Wherein the registration account isassociated With a dynamic list of articles, continuously monitor the article carryingarrangement by means of the sensor arrangement, receive a first image from the sensorarrangement of at least a portion of a customer°s upper extremity being captured Whenat least a part of the customer°s upper extremity is moving across a virtual boundary setat a predeterrnined distance from the article carrying arrangement, determine, based atleast in part on the first image, if the first image includes a representation of an article,and if so identify said article, and add the identity of the identified article to the dynamic list of articles of the registration account.
In one embodiment, the method further comprises the step of receive a secondimage from the sensor arrangement of at least a portion of a customer°s upper extremitybeing captured When at least a part the customer°s upper extremity is moving away fromthe virtual boundary set, determine, based at least in part on a comparison of the firstimage and the second image, if the first image includes a representation of an article andif so determine if the second image does not include a representation of the article of thefirst image, and if so, identify the article of the first image.
In one embodiment, Wherein if it is deterrnined that the first image includes arepresentation of an article and that the second image include a representation of thearticle of the first image, the method further comprises the step of determine if the firstimage includes a representation of an article and that the second image includes arepresentation of the same article as of the first image, and if the first and second imagesincludes representation of different articles: identify the article of the first image andadd the identity of the article to the dynamic list of articles of the registration account;and identify the article of the second image and remove the identity of the article fromthe dynamic list of articles of the registration account.
In one embodiment, Wherein if it is deterrnined, based at least in part on acomparison of the first image and the second image, that the first image includes arepresentation of an article and that the second image includes a representation of thesame article as of the first image, the article is not added to the dynamic list of articlesof the registration account.
In one embodiment, if it is deterrnined, based at least in part on a comparisonof the first image and the second image, that the first image does not include arepresentation of an article and that the second image includes a representation of anarticle, the method further comprises the step of identify said article at least in partbased on the second image from the article identities currently included in the dynamiclist of articles of the registration account, and remove the identity of the article WhichWas identified in the second identification from the dynamic list of articles of theregistration account.
In a fifth aspect, an automatic in-store registration system in a store having at least one article containing area is provided. The system comprises at least one processor, at least one registration account being associated With an article carryingarrangement, and a sensor arrangement comprising a plurality of sensors, the sensorarrangement being conf1gured to continuously monitor the article carrying arrangement.The processor is configured to receive, from the sensor arrangement, a first image of atleast a portion of a customer°s upper extremity being captured When at least a part of thecustomer°s upper extremity is moving across a virtual boundary set at a predeterrnineddistance from the article carrying arrangement, determine, based at least in part on thefirst image, if the first image includes a representation of an article, and if so, identifysaid article, and Wherein the processor further is configured to add said final article tothe registration account.
In a sixth aspect, a method for training an in-store registration systemconfigured to identify and register articles in a store is provided. The method comprisesperforming a training sequence in order to improve the accuracy of the identification ofarticles by def1ning a virtual boundary set at a predeterrnined distance from the articlecarrying arrangement, using training data to recognize different movements of a user°supper extremity While being in or near the virtual boundary set, and to detect at least aportion of a upper extremity of a person When at least a part of the person"s upperextremity is moving across the virtual boundary set, registering one or more signaturesof each article by means of a sensor arrangement, and adding or updating informationrelating to signature and the position of each article to the in-store registration system.
In one embodiment, the training data is composed of a synthetically generated3D-model of at least a part of the store and/or the customer, Wherein the synthetictraining synthetic data is a 3D-model of at least a part of the store. The synthetic datamay be enhanced using Generative Adversarial Networks.
In a seventh aspect, a method for registration of an article in a store having atleast one article containing area and an associated automatic in-store registration systemis provided. The method comprises to associate a customer With a registration account,monitor the movement of the customer through the store, continuously monitor the atleast one article containing area by means of a sensor arrangement, and detect a changein the at least one article containing area and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list, Wherein, if saidtemporary article list includes tWo or more articles, positively identifying the articleWhich caused the change in a second identification step from the articles currentlyincluded in the temporary article list by receiving a first image from the sensorarrangement of at least a portion of a user°s upper extremity being captured after at leasta part of the user°s upper extremity is removed from the article containing area, Whereinthe first image includes a representation of an article, and add the identity of the articleWhich Was positively identified in the first or second identification to the registrationaccount.
In one embodiment, the method further comprises the step of receiving asecond image from the sensor arrangement of at least a portion of a user°s upperextremity being captured When at least a part of the user°s upper extremity is inside thearticle containing area, Wherein the first and second images are used to identifying thearticle in the second identification step.
In one embodiment, the second identification step further comprises accessingthe article identities saved in the temporary article list and identifying the article as oneof the accessed article identities originally positioned Within a predeterrnined distancefrom the customer and added to the temporary article list Within a predeterrnined time.
In one embodiment, the method further comprises identifying a plurality ofcustomers, and associating two or more customers to the same registration account.
The step of associating the customer With the registration account may beperformed using Near Field Communication.
In one embodiment, the first identification step is performed by reducing thetotal number of possible article identities using the position information of the change inthe article containing area. The article containing area may be arranged With a pluralityof segments, Wherein the position of each segment is known. Each segment of thearticle containing area may comprise at least one capacitive sensor arranged to detect achange in the article containing area.
In an eight aspect, an automatic in-store registration system in a store having atleast one article containing area is provided. The system comprises at least one processor, at least one registration account being associated With a customer in the associated store, and a sensor arrangement comprising a plurality of sensors, the sensorarrangement being configured to, in a first identification step, continuously monitor thearticle containing area and to detect a change in the article containing area and, inresponse to said detected change, to transmit a first identification signal comprisinginforrnation relating to one or more possible articles to the processor, wherein the sensorarrangement further is configured to continuously monitor the customer. The processoris configured to identify one or more articles from said first identification signal and tosave the identified one or more articles to a temporary article list, wherein if thetemporary article list includes two or more articles, the processor is configured topositively identify the article which caused the change in a second identification stepfrom the articles currently included in the temporary article list by receiving a firstimage from the sensor arrangement of at least a portion of a user°s upper extremitybeing captured after at least a part of the user°s upper extremity is removed from thearticle containing area, wherein the first image includes a representation of an article,and to add the article being positively identified in the first or second identification step to the registration account.
BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will be described in the following; referencebeing made to the appended drawings which illustrate non-limiting examples of how theinventive concept can be reduced into practice.
Fig. l schematically shows a top view of a store having a registration systemaccording to an embodiment of the present invention; Figs. 2a-b show isometric views of parts of a store having a registration systemaccording to an embodiment of the present invention; Fig. 3 shows a schematic view of a registration system according to anembodiment of the present invention; Fig. 4a shows a schematic method for automatic registration according to anembodiment of the present invention; Fig. 4b shows a schematic method for automatic registration according to an embodiment of the present invention; Fig. 4c shoWs a schematic method for automatic registration according to anembodiment of the present inVention; Fig. 4d shoWs a schematic method for automatic registration according to anembodiment of the present inVention; Fig. 5 shoWs a schematic View of the limitation process of a registrationmethod according to an embodiment of the present inVention; and Fig. 6 shoWs a schematic method for automatic registration according to an embodiment of the present invention.
DETAILED DESCRIPTION OF EMBODIMENTS Fig. 1 shows an illustrative View of a designated area, such as a retail orgrocery store 10. The store 10 comprises a plurality of article containing areas A-I, suchas shelves, displaying articles available for purchase. The store 10 may also include anentrance or check-in area 12, an entrance gate 14, an exit or checkout area 16, an exitgate 18 and at least one checkout station 20. The entrance gate 14 and the exit gate 18may be the same gate. Furthermore, the entrance area 12 and the exit area 16 may be thesame or they may overlap.
A customer enters the store 10 by passing through the entrance gate 14 and intothe entrance area 12, Whereby the customer Walks through the aisles between the articlecontaining areas A-I and collects the articles to be purchased. The article Will be, at leastin part, identified by the automatic in-store registration system 100 described hereinonce the article is removed from its original position at the article containing areas A-I.Providing that the identification by the registration system 100 is successful, thecustomer does not need to scan the article, or provide any other manual input. Once thecustomer is finished fetching the articles to be purchased, the customer pays the amountcorresponding to the chosen articles in the checkout station 20 and exits through the exitarea 16 out of the exit gate 18. The total amount to be paid is automatically deterrninedby the registration system 100. The automatic in-store registration system 100 may alsouse information relating to discounts, coupons, sales-items etc. in the deterrnination of the total amount. ll Fig. 2a shows an embodiment of the registration system 100 monitoring onearticle containing area A, here in the forrn of a shelf, comprising a plurality of differentarticles. The registration system 100 comprises a sensor arrangement 110 comprising aplurality of sensors 112, Which at least monitors the shelf A per se and/or the articlesarranged on the shelf A. In some embodiments, the sensors 112 also monitor a customer1 and/or an article carrying arrangement 5 used by the customer 1, as Will be describedin further details beloW. Furthermore, the registration system 100 comprises aregistration account 130, Which together With the sensor arrangement 110 Will bedescribed more in detail With reference With Fig. 3.
Although the article containing area A shoWn in Fig. 2a is the form of a shelf,the article containing area may also take the form of a basket, tray, freezer, fridge and/orother similar arrangements Which can contain and display articles to a customer. Thearticles may be of different dimensions, structure and colors as illustrated in Fig. 2a.Preferably, all articles being the same type and/or brand are arranged next to each otherto facilitate selection and retrieving by the customer 1.
The article carrying arrangement, or device, 5 in Fig. 2a is in the form of ashopping cart, but it should be understood that the article carrying arrangement 5 alsomay be a basket, a bag or similar container Which the customer 1 may use to collectand/or transports its chosen articles. The article carrying arrangement may thus be usedto collect the by a customer chosen article. The article carrying device may be amovable device or arrangement. In another embodiment, the article carryingarrangement 5 may be a stationary, but pre-defined area.
In yet one embodiment, the article carrying arrangement 5 could be the upperextremity of a user. More preferably, the article carrying arrangement is the hand and/orarm of the customer. The article carrying arrangement 5 could thus be a part of thecustomers upper body. This is for example the case When a customer is pickingarticle(s) from an article containing area A and carries the articles in his/her arms. Thecustomer may carry the article(s) in his arms during the Whole shopping session, or fortime period before he/she adds it to a second article carrying device (such as a shopping cart, basket, or a stationary pre-defined area). 12 In Fig. 2a the customer 1 has fetched an article 7 from the shelf A and has theintention to put down the article into the article carrying device 5. This selected article 7thus needs to be identified in order to be added to the registration account 130 so thatthe customer can pay for the article 7. For this purpose the sensors 112 monitor thearticle containing area A and all the articles arranged thereon, at least the articlesarranged at the front. By continuously monitoring the article containing area A, it ispossible to recognize When a change in the article containing A has occurred. Such achange may indicate that a customer 1 has picked an article 7 from the article containingarea A. The registration system 100 is configured to have access to information relatingto Which types of articles being present on each article containing A-I and theregistration system 100 is configured to use such position information together Withanalyzing data from the sensors 112 in order to identify the identity of the article 7being selected and picked by the customer 1.
Fig. 2b illustrates an article containing area A being divided into a plurality ofspecified segments aa, ab, ac, ad. The segments provide a more refined position data ofthe articles arranged in the article containing area A. The segments may vary in size orall segments may be of the same size. The segments may seen as an area or a volume.
The segment may comprise one kind of article of one single brand, or one kindof article of different brands or each segment comprises a plurality of article types.
In one embodiment, all segments are arranged With at least one sensor. As Willsoon be described more in detail, the sensors may for example be a Weight sensor, LIDAR-sensor, capacitive sensor and/or a camera.
TlIC SCIISOI' affangCmCnt As explained above the sensor arrangement 110 is provided in order toautomatically detect the articles 7 being picked by the customer 1 throughout the store10. The sensor arrangement 110 thus forms part of the registration system 100, Whichalso, as indicated in Fig. 3, comprises a processor 120, being connected to a memory122, and a registration account 130 being in operative communication With theprocessor 120. The sensor arrangement 110 is configured to be in communication With the processor 120 as Will be explained further below. 13 The sensor arrangement 110 comprises at least one sensor 112. HoWever, asindicated by the dotted lines in Fig. 3, the sensor arrangement 110 may comprise aplurality of sensors 112. Preferably, the sensor arrangement 110 comprises a plurality ofsensors 112 to create a full coverage of the articles 7 arranged in the store. Furtherrnore,a plurality of sensors 112 may be arranged to cover different views of the article, so thatfor example an upper, lower, left, right, front, and back side of the article 7 could bemonitored by the sensors 112. The sensors 112 may be arranged on the article containeritself, on the Walls surrounding the article containers and/or on the ceiling of the store.
Additionally or altematively, one or a plurality of sensors may be arranged onan article carrying device 5, such as on a shopping cart. If at least one sensor is arrangedon the article carrying device 5, the sensor can be used to continuously generate dataand thus generate a geometry of the store.
The sensor 112 may be conf1gured to provide inforrnation corresponding to thedimensions and outer appearance of the article, as Well as the position of Where thearticle Was picked in the article containing area A-I. The sensor 112 may furtherprovide inforrnation relating to size, color, shape, contour, marking With barcode and/ortext and/or figure and/or pattern and/or reflection spectrums. The sensor 112 may alsoprovide inforrnation relating to Weight, movement, different triggers, presence,sWitching, acceleration, vibration, temperature and/or the picking position in the articlecontainer.
The sensor arrangement 110 may preferably comprise different types ofsensors 112. The sensors 112 included in the sensor arrangement 110 may for examplebe any of: a camera, a spectroscopy sensor, a RFID sensor, a contour sensor, a Weightsensor (i.e. a scale), a symbol or text recognizing sensor, a color texture sensor, a colorhistogram sensor, microwave sensor, OCR, 3D-sensor or camera, time of flight sensor,presence sensor, switch sensor, accelerometer, movement sensor, temperature sensorand/or an object sensor and/or microwave sensor and/or a light curtain, an IR camera,and/or LIDAR sensor. The contour sensor, the symbol or text recognizing sensor, thecolor texture sensor, the color histogram sensor, and the object sensor may all be implemented by one or several cameras that are connected to the processor 120, Which 14 processor 120 is configured to execute a specific image-processing algorithm forextracting the required inforrnation from the captured images.
In one embodiment at least one sensor 112 in the sensor arrangement 110 is acamera. The camera may be a 2D and/or 3D camera. The camera may have associatedimage-processing software, or the required processing software may be provided by theprocessor 120.
The at least one sensor 112 Will during operation detect a sensor signal, such asan measured signature, from an article, Whereby the processor 120 is configured tomatch the measured signature With a database of reference signatures. Each referencesignature is associated With a unique article identity. Hence, different articles Willprovide a unique signature Whereby an accurate identification may be achieved. Thesensor must thus be capable of detecting article signatures for a Wide variety of articleshaving different dimensions. For example, such articles may have small dimensions, asWell as large dimensions. The memory unit 122 of the processor 120 may comprises oneor more digital reference signatures created by the at least one sensor 112, Wherein eachone of said reference signature is associated With a corresponding article identity. Theidentity of the article may be deterrnined by matching a measured signature to adatabase comprising properties of a plurality of articles. The database may be stored inthe memory unit 122 and examples of properties are Weight, size, color, shape, contour,pattem, barcode pattems and/or text and/or figure and/or reflection spectrum. In oneembodiment, the sensor arrangement 110 comprises at least one sensor in the form of acamera. The camera detects an image from an article, Whereby the processor 120 isconfigured to match the detected image, or a processed digital representation of theimage, With a database of reference images, Whereby each reference image is associatedWith a unique article identity. Hence, different articles Will provide a unique digitalrepresentation of the image Whereby an accurate identification may be achieved. Thememory unit 122 of the processor 120 may comprises one or more reference images (orreference processed digital representations of the images) created by the at least onecamera, Wherein each one of said reference images/processed digital representations isassociated With a corresponding article identity. The identity of the article may be deterrnined by matching a deterrnined image or a processed digital representation of the image to a database comprising properties of a plurality of articles. The database may bestored in the memory unit 122 and examples of properties are size, color, shape,contour, barcode pattems and/or text and/or figure.
As Will be described more in detail later on, the processor 120 may also beconfigured to receive images of article containing area, article carrying device and/orimages of a customer°s upper extremity (such for example the hand and/or the arrn)indifferent situations.
A number of different sensors 112 may be used to provide correctidentification of an article. For example, a shelf A is filled With one hundred differentarticle types. Each type of article is represented by a certain number of individualarticles, Which means that the shelf A stores several hundred, or even thousands ofindividual articles. There is thus a need for a reliable identification process.
The shelf A may e.g. be equipped With one or more scales forrning part of thesensor arrangement 110. When a customer 1 picks an article from the shelf A the totalWeight of the shelf A Will be reduced, and the information relating to the change inWeight Will be transmitted to the processor 120. If the Weight reduction is deterrnined tobe 250g, the processor 120 Will determine that the article being picked must belong toone of those article identities not Weighing more than 25 0g. Preferably, the sensorarrangement 110 may comprise a plurality of scales being positioned at a common shelfA. For example, if a shelf A stores ten different article types, ten individual scales maybe provided at the shelf A. Each scale may then be conf1gured to measure the Weight ofa separate part of the shelf A such that the article type may be identified by detecting achange in Weight of the part of the shelf A storing the particular article type. Yet further,the data measured by the scales may further be used to identify the number of articlesbeing picked from the shelf A.
In one embodiment the one or more scales are arranged to determine if anarticle has been picked from or being placed in the article containing area based on achange in Weight. Hence, the exact change in Weight may not be needed or be connectedto a specific article identity, since it may be sufficient to detect that the Weight hasincreased or decreased in order to determine if an aticle has been picked or placed in the article containing area. 16 In one embodiment the one or more scales are arranged to determine if anarticle has been picked from or being placed in the article containing area based on achange in "weight". Hence, the exact change in weight is not known, but samples fromthe load cells will be monitored and leamed. Based on the leamed samples the articletype may be identified by detecting the change in the load cells of the part of the shelf Astoring the particular article type. Yet further, the data measured by the load cells mayfurther be used to identify the number of articles being picked from the shelf A.
The sensor arrangement 110 may also include one or more capacitive sensors.The one or more capacitive sensors are preferably arranged in conjunction with theshelfs A. The capacitive sensors may be arranged on the bottom of the shelf and/or onthe top of the shelf. The capacitive sensors are arranged to detect when a customer°shand and/or arm (or other part of the upper extremity of the customer) is approaching ortouching the article containing area. The capacitive sensor preferably comprises areceiver and a transmitter, where an electric field is formed between the receiver andtransmitter. An electric field, so called fringe electric field, also extends out from thetransmitter. Once at least a part of the upper extremity of the customer is approachingthe sensor, the fringe electric field is changed and a change in environment is detected.As will be described more in detail later, this information may be used to trigger anothersensor(s) of the sensor arrangement 110.
The sensor arrangement may also include one or more LIDAR sensors, whichmay be attached to a shelf or in the ceiling of the article containing area. The LIDARsensor is preferably mounted so that it creates a vertical plane in front of an articlecontainer, e. g. a shelf A. When a customer or attendant reaches for an article in the shelfA the LIDAR sensor will detect an exact x-y coordinate of where the plane wasobstructed. The system may be configured to have information of every unique article"sexact position in the shelf A, thus by using the coordinate information from one or moreLIDAR sensor the article identity can be accurately deterrnined. The information fromthe LIDAR sensors may also be used as leaming data to the processor 120 when articlesare put into the shelf, e. g. as performed when attendants are unpacking new articles for display. 17 The sensor arrangement 130 may also include one or more IR cameras Whichmay be attached to a shelf or in the ceiling of the article containing area A-I. IR camerasmay be specifically advantageous for identifying cold and/or hot articles.
As is evident from the foregoing description, the above example onlyrepresents some possible uses of different sensors 112. The above-mentioneddescription could thus also be implemented using another type of sensor, such as a 3D- camera, a spectrometer, etc., The communication interface The sensor arrangement 110, and thus the individual sensors 112,communicates With the processor 120 by the use of a communication interface. Thecommunication interface is adapted to allow the sensor arrangement 110 tocommunicate With the processor 120, through the use of different communicationtechnologies. Such communication technologies may be Wired or Wireless. Examples ofsuch Wired technologies are Universal Serial Bus (USB) and Ethemet to name a few. Itshould be noted that other technologies exist and are taken to be an obvious equivalentfor such Wired communication interfaces. Examples of such Wireless technologies areIEEE 802.11, IEEE 802.15, ZigBee, WirelessHART, WIFI, Bluetooth®, W-CDMA/HSPA, GSM, UTRAN and LTE to name a feW. It should be noted that othertechnologies exist and are taken to be an obvious equivalent for such Wireless communication interfaces.
The processor The processor 120, or a plurality of processors 120 arranged in communicationWith each other, is connected to the sensor arrangement 110. The at least one processor120 is conf1gured to determine the identity of an article picked by the customer 1. Theprocessor 120 is responsible for the overall operation of the sensor arrangement 120 andthe registration account 130 and is preferably implemented by any commerciallyavailable CPU ("Central Processing Unit"), DSP ("Digital Signal Processor") or anyother electronic programmable logic device. The processor could by a physical device, or being part of a netWork computing system such as a cloud. The processor 120 is 18 configured to read instructions from a memory 122 and to execute these instructions tocontrol the operation of the sensor arrangement 110 and/or the registration account 130.The memory 122 may be implemented using any commonly known technology forcomputer-readable memories such as ROM, RAM, SRAM, DRAM, CMOS, FLASH,DDR, SDRAM or some other memory technology. The memory 122 is used for Variouspurposes by the processor 120, such as for storing application data and programinstructions, as well as storing reference data, such as reference signatures, for thearticles of the store 10.
The processor 120 is further connected to the communication interface in orderto communicate with the sensor arrangement 110, the registration account 130 and/or anextemal electronic device (such as a mobile phone, tablet or any other computingdevice). Such communication technologies may be Wired or wireless, as described in relation to the communication interface arranged in the sensor arrangement 110.
The registration account In some embodiments the registration account 130 is created and connected tothe customer in a direct way, i.e. by tracking a customer inside a store, and in someembodiments the registration account 130 is created and connected to an accompanyingunit such as an article carrying device 5. The respective embodiments will now bedescribed together with preferred methods of adding identified articles to the registration account.
Article carrying device - registration account 1st embodiment In some embodiments the registration account 130 is created and connected toan accompanying unit such as an article carrying device. The article carrying device 5may for example be a shopping cart, basket or a bag. The article carrying arrangement 5could also be the upper extremity of the customer, and more specifically the handsand/or arms of the customer. The article carrying arrangement 5 could thus be aphysical or virtual arrangement, and the arrangement 5 could be a part of the customer.In the following, the article device or arrangement 5 will be embodied as a shopping cart, basket or a bag, however it should be understood that the description is applicable 19 to the embodiments where the article carrying arrangement 5 is the upper extremity ofthe customer and the embodiments where the article carrying arrangement 5 is astationary predeterrnined area.
The registration account 130 keeps a list of the articles that has been collectedin the store 10. The registration account 130 is in communication with the processor 120and during checkout the list of articles is automatically transferred from the registrationaccount 130 to an associated payment system. The registration account 130 comprises alink to an article carrying device 5, represented by means of a unique ID or similar, anda dynamic article list of the items being picked from the article containing areas A-I.
As soon as the person pulls out the article carrying device 5 form its idlelocation, the article carrying device 5 will be associated with a unique registrationaccount 130. Altematively, the customer may e.g. press an activation button arranged onthe article carrying device 5 or at another position at the entrance area for allowing thearticle carrying device 5 to be associated with the registration account 130. Associationof the article carrying device 5 and the registration account 130 is required in order tomake use of the registration system 100. The registration account 130 may either bemanaged using the article carrying device 5 itself or by connecting the article carryingdevice 5 to a mobile electronic device, as will be described further below.
In one embodiment, the article carrying device 5 in the form of a shopping cart5 is used to manage the registration account 130. The shopping cart 5 thus needs to havenecessary hardware for allowing the customer to interact with the registration account130. In one embodiment, the article carrying device 5 may comprise an interactivedisplay and equipment for allowing for manual registration of articles if needed. Sucharticles may e. g. by bulk articles or similar. The communication interface of the articlecarrying device 5 may also allow the customer to review and/or mange the registrationaccount 130. It is beneficial if the in-store registration system 100 can gain informationof the position of the article carrying device 5. This may be done in several ways. In oneembodiment the article carrying device 5 comprises a unique identification tag. Thisidentification tag is read by the sensor arrangement 110, wherein the positions of theindividual sensors are known, in order to determine a current position of the article carrying device 5. The identification tag may for example be a QR-code or another unique pattern Which can be used for the sensor arrangement 110 to identify the articlecarrying device 5. In an alternative embodiment the article carrying device 5 maycomprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE),short-range radio frequency positioning and/or Wifi-based positioning systems.
The use of a smart article carrying device 5 alloWs the registration account 130to be associated With a current position in the store Without tracking the individualcustomer itself. It can be assumed that the customer picks an article in an articlecontaining area A-I Which is Within a specified distance from the article carrying device5. Such presumption reduces the number of possible articles to choose from Whenidentifying a fetched article. This information is used in the identification process of thearticle, as Will be described more in detail With reference to Fig. 5.
Here the customer does not need any other physical device than the articlecarrying device 5 in order to properly use the registration system 100. Additionally,there is no need for the customer 1 to manually check in When entering the store andmultiple customers may use the same article carrying device 5 and thus the sameaccount 130 Without requiring adaptation of the registration account 130. Having theregistration account 130 associated With an article carrying device 5 has the benefit thatthe account is not directly associated With the customer and his/her identity.
The embodiment Where the registration account 130 is managed using thearticle carrying device 5 per se is preferably used When the probability of correct identification of an article is very high, so that manual registration is rarely needed.
Article carrying device - registration account. 2nd embodiment An embodiment in Which the registration account 130 is associated With anarticle carrying device 5 Will now be described. In some embodiments, the registrationaccount 130 is managed by connecting the article carrying device 5 to a separatephysical device such as an extemal electronic device. The extemal device may be amobile device, such as a mobile phone or other suitable devices for allowing thecustomer to interact With the registration account 130 In either case, the article carrying device 5 does not need to have any special hardware, display or position detection unit. 21 As soon as the person pulls out the article carrying device 5 form its idlelocation, the article carrying device 5 will be associated with a unique registrationaccount 130. Altematively, the customer may e.g. press an activation button arranged atthe entrance area for allowing the article carrying device 5 to be associated with theregistration account 130, or use the external device to make the association between theregistration account 130 and the article carrying device 5. Association of the articlecarrying device 5 and the registration account 130 is required in order to make use ofthe registration system 100.
Preferably, the article carrying device 5 comprises a unique identification tagwhich the sensors arrangement 120 reads in order to gain the position of the articlecarrying device 5. Since the positions of the individual sensors in the sensorarrangement are known, it is possible to determine the position of the article carryingdevice 5. The identification tag may for example be a QR-code or another uniquepattem which can be used for the sensor arrangement 110 to identify the article carryingdevice 5.
When an extemal device, such as a mobile phone or similar, is used formanaging the registration account 130 the article carrying device 5 may be providedwith a unique code arranged to be scanned by an application on the external device.Such a code may be in the form of a QR-code, EAN, UPC code or an RFID tag. Byscanning the code arranged on the article carrying device 5, the registration account 130can e.g. be managed using the application of the extemal device. This allows thecustomer to manually scan articles using the extemal device when the registrationsystem 100 does not manage to correctly identify an article, and to manually removeidentified articles which for some reason were decided not to be bought. The applicationof the extemal device must thus preferably be able to scan codes of an article. Theextemal device may also allow the customer to review and/or mange its registrationaccount 130. For this purpose the extemal device may e.g. be a dedicated portable codescanning unit.
In one embodiment, the article carrying device 5 is associated with a specificregistration account 130 by using Near Field Communication. one embodiment, the article carrying device 5 is arranged with a NFC-tag. The article carrying device 5 can 22 thus be connected With the external device having a NFC-reader, such as a mobilephone. Hence, in one embodiment a NFC-tag, being readable by an external device ofthe customer, is arranged on the article carrying device 5. In an alternative embodiment,a NFC-reader is arranged on the article carrying device 5, is conf1gured to read a NFC-tag on an external device of the customer. The extemal device may for example be atag arranged in a credit card.
The tags used by the customer When linking the article carrying device 5 to theregistration account 130, and the tag used for identifying the article carrying device 5 bymeans of the sensor arrangement 110 may not necessarily be the same. In someembodiments, the position of the article carrying device 5 may be deterrnined using apositioning unit on the extemal device.
The embodiment Where the registration account 130 is managed using anextemal device connected to the article carrying device 5 is preferably used When theprobability of correct identification of an article is very high, so that manual registration is rarely needed.
Article carrying device - First identification step: Limitation The method of automatic registration of an article Will now be described inmore detail With reference to Fig. 4a. The following description is related to the caseWhere the registration account 130 is associated With an article carrying device 5, Withor Without the use of an extemal device. As been mentioned above, in a first step 202the sensor arrangement 110 monitors all article containing areas A-I arranged in thestore. By continuously monitoring an article containing area A-I, it is possible torecognize 204 When a change in the article containing area A-I has occurred.
This may be done by comparing data streams generated by the sensorarrangement 110 and by comparing the data frame by frame. Additionally oraltematively, the change in the article containing area A-I may be detected by at leastone sensor in the sensor arrangement 110. The detected change Will then trigger, i.e.activate, at least one other sensor in the sensor arrangement 110 and/or the process of identifying the article causing the change. 23 The sensor(s) detecting the change may for example be at least one capacitivesensor arranged in the article containing area A-I. The capacitive sensor detects thatsomething is causing a change in the electromagnetic environment of the containingarea A-I. Such a change may for example be the a part of the customer°s upperextremity (such as the hand and/or arm) that is approaching, retracting, occupying ortouching the article containing area A-I. This information will be used by the system inorder to initiate the identification process.
A change in article containing area A-I may indicate that a customer 1 hasselected and removed an article 7 from the containing area A-I. This article 7 needs tobe, at least in part, identified 206 in order to finally be added 216 to the registrationaccount 130 associated with the article carrying device 5. In one embodiment, the firstidentification step is performed once the system 100 recognizes a change in the articlecontaining area A-I and a second identification step is performed once the customer putsan article into the article carrying device 5. In another embodiment, the article is in factcompletely identified directly as it is removed from the article container A-I. However,in order to connect the identified article with the correct article carrying device 5, thestep of adding the article to a list of possible article identities and having a secondidentification step at the article carrying device 5 may still be preferred.
The first identification step 206 comprises two main steps; limitation ofpossible article identities and identification of a unique article, which steps will now bedescribed more in detail. These two steps may be performed in sequence orsimultaneously. The limitation process is based on the fact that the article containingarea A-I has a known position in the store and a known range of articles and thelimitation process uses this information in order to at least partly identify the article. Ifan article is removed from one article containing area A, it is a very high probabilitythat said article is one of the articles which usually is accommodated in that articlecontaining area A. This limitation process based on position decreases the number ofpossible articles.
As previously mentioned, each article containing area may be divided into aplurality of segments. The segmentation of the article containing areas further increases the position information of the articles contained therein. Moreover, capacitive sensors 24 may be arranged in each segment of the article containing area to further specify theexact position of the article being removed from the area.
Once the number of possible articles is reduced, the system 100 needs tonarrow down the possible articles to only one unique article. This identification may beachieved using different identification techniques. One such technique may be anidentification algorithm, which uses information gathered from the sensor arrangement110 as an input. The sensor arrangement 110 comprises at least one sensor 112 which isconfigured to determine a measured signature of the article being taken from the articlecontainer. The sensors 112 may be switched on, i.e. activated, in sequences to be able tofind beneficial combinations or a partial set of or all the sensors may be active until oneof the combinations provides a positive decision, wherein one or more of the redundantsensors may be disconnected. A positive decision is here when all sensors in thecombination have detected and identified a predeterrnined property of an article, wherethe properties in combination together give the article an identity. The identity may bedeterrnined through control against a database comprising properties, or signatures, ofan amount of articles. The database may be stored in the memory unit according to whathas previously been described. Examples of properties are size, color, shape, contour,marking with barcode and/or text and/or figure and/or pattem. In order to provide asuccessful classification, an activated sensor deterrnines a measured signature of anarticle. The measured signature is associated with the sensed signal, and may thus be adigital representation of a number of different article properties. The processor 120 isfor this purpose configured to compare the measured signature with the digital referencesignatures stored in the memory unit, and to calculate a matching probability of apredeterrnined number of article identities. The latter step is preferably performed bycomparing the measured signature with all, or a subset of, the digital referencesignatures of the memory unit and subsequently delivering the article identities havingthe highest matching probabilities to a further classification algorithm, such as a BBNor a neural network.
In case of using two or more different sensors thus providing two or moredifferent measured signatures, the processor 120 is configured to independently compare each sensors measured signature with all, or a subset of, the digital reference signatures of every sensor. The identified articles and their matching probabilities fromtwo or more sensors are transmitted to the network for further analysis and to combinethe highest matching probabilities of each sensor into one combined matchingprobability.
The articles may also be identified using a pattem recognition algorithm basedon the data collected by the sensor arrangement ll0. In one embodiment the pattemrecognition algorithm is based on SVM, which preferably uses the measured signaturesand the digital reference signatures mentioned above as input to the support vectormachine model. In an altemative embodiment of the algorithm the system iscontinuously leamed by using the raw data from the sensor arrangement ll0. In thissituation, the algorithm preferably uses deep leaming methods, such as Convolutionalneural network or Fully Connected neural network.
The different identification algorithms discussed above may be used separatelyor as subsequent steps in a combined identification routine. In the latter method theoutcome of each step is combined in order to produce a unique identified article withthe highest matching probability.
The classification method may further include a step of comparing the highestmatching probability with an alarm threshold and, in case the highest matchingprobability is below the alarm threshold, the system awaits manual input from thecustomer in the store before proceeding. In one embodiment the identification includesthe step of comparing the highest matching probability with two alarm thresholdwherein, in case the highest matching probability is above the lowest alarm thresholdbut below the upper alarm threshold, the method awaits manual input from the customerin the store, and in case the highest matching probability is below the lowest alarmthreshold, the system flags the registration account for additional security check at thecheckout station, as will be described more in detail later on. In an altemativeembodiment, the attendant is not needed and the customer input is sufficient.
In one embodiment the system l00 identifies the article based on at least oneimage taken of at least a part of the customer°s hand and/or arm when it is in the article containing area and/or when the hand and/or arm is removed from the area. 26 In one embodiment, once the identification process is initiated, preferably bythe sensor arrangement 110 as discussed above, the sensor arrangement 110 Will take animage of at least a portion of a user°s hand and/or arrn being captured after the user°shand and/or arrn is removed from article containing area. The image Will thus containinformation of the article being picked from the article containing area. Thisinformation may be used, possibly together With other output from the sensorarrangement 110, in order to identify the article. Additionally, or altematively, theimage may be used to determine if the customer is holding an article or not. Hence, theinformation gathered from the image of the customer°s hand and/or arrn may not beused to directly identify the article.
Additionally, or altematively, the sensor arrangement 110 takes an image of atleast a portion of a user°s hand and/or arrn being captured When the user°s hand and/orarrn is inside the article containing area. This information may be used to determineWhere in the article containing area the hand and/or arrn, i.e. the article to be picked, islocated. This information can thus be used to gain more accurate position data.
In one embodiment the sensor arrangement 110 is activated based on one typeof sensors, such as Weight sensors, LIDAR-sensor and/or capacitive sensors. After theactivation, the sensor arrangement 110 takes an image of at least a portion of a user°shand and/or arrn being captured When the user°s hand and/or arrn is inside the articlecontaining area and one image of at least a portion of a user°s hand and/or arrn beingcaptured after the user"s hand and/or arrn is removed from article containing area. Thisinformation may be used, possibly together With other output from the sensorarrangement 110, in order to identify the article and/or to gain more accurate positiondata.
Additionally, or altematively, the sensor arrangement 110 is arranged to firsttake an image of the article containing area When a customer°s hand and/or arrn isapproaching the area. In a next step, an image is taken of the customer"s hand and/orarrn While being inside the article containing area. Finally, an image is taken of thecustomer°s hand and/or arrn When it is moving away from the article containing area,possibly holding a picked article. These images are evaluated in order to identify the article. The information gained from the images may be position data, knowing if an 27 article was removed or added to the article containing area and/or for identification ofthe article.
For the embodiments described above the picking position is used to limit thetotal number of possible articles, and the network is used to further limit the number ofpossible articles to a single, positively identified article. However, it may also bepossible to determine the picking position accurately so that a single article is positivelyidentified using only position data. As explained above, LIDAR sensors, weight sensorsand/or capacitive sensors may be used to accomplish this. In these situations, it is notnecessary to make use of the network for further identification of the article using forexample images of the customer°s hand and/or arm. Hence, step 204 and 206 in Fig. 4amay be performed merely on position data.
For all cases described above the first identification step may result in a single,positively identified article, or a list of possible articles.
The information that an article, identified or unidentified, has been taken fromits position in the article containing area A-I may be used in a second identificationprocess either to verify the identification from the first identification step or to provide acomplete identification of an article which was only partly identified in the firstidentification step. The second identification step comprises two main steps; limitationof possible article identities and identification or verification of a unique article. Thesetwo steps may be performed in sequence or simultaneously.
The second identification process may be initiated when something is movedinside the boundaries of the article carrying device 5 (seen from above when somethingis moving within the railing rectangle of the article carrying device 5), and the systemthus detects 208 that an article is being placed in the article carrying device 5. Althoughthe first identification process may have succeeded in identifying the article when beingremoved from the article containing area, the system must also determine to whichregistration account 130 said article belongs to or if this has already been deterrnined toverify that the article and the registration account 130 belongs to each other.
Once the system detects that an article is being placed in the article carryingdevice 5, the list of previously identified articles, or partly identified articles, is fetched210 and the number of articles in the list is reduced 212 based on the position of the 28 article carrying device 5. Finally, the article is identified 214 from the reduced numberof article identities in the list and the article is added 216 to the registration account.These steps will now be described more in detail. Fig. 5 illustrates the relationshipbetween an article in an article containing area A-D and the time at which the customerremoves the article from the article containing area A-D. Article containing area Acomprises articles named a, article containing area B comprises articles named b, articlecontaining area C comprises articles named c and article containing area D comprisesarticles named d. In this example, the article containing areas A-D are arranged withinthe predeterrnined distance from the article carrying device 5. All articles arranged inthe article containing areas A-D are thus possible candidates when limiting the numberof possible articles. In order to further narrow down the possible articles during thelimitation process, the time at which the article was taken from the article container isused.
All articles that are taken from the shelves A-D will be saved in a temporarylist, and will be saved thereon for a predeterrnined period of time t1 before they areremoved from the list. The memory 122 saves information of articles being taken fromeach article containing area A-D for a predefined period of time t1. The time periodallows the registration system 100 to have a limited amount of articles to choose from.Hence, the shorter period of time the less number of possible articles to choose from inthe identification process. It may take a while from the time when the customer takesthe article from the article container until he/she puts the article into the article carryingdevice 5. The customer may for example grab a couple of different articles beforeputting them all into the article carrying device 5. The time t1 needs thus be sufficientlylong, so that the article still is among the possible articles during identification even ifthe customer has been slow in putting the article into the article carrying device 5.
If the article was not identified when picked from the article container, allarticles that were within a predeterrnined distance from said article container will beavailable in the temporary list.
The registration system 100 keeps track of the position of the article carryingdevice 5, as previously described. This position information is used to limit the possible article candidates as described with reference to Fig. 5. The possible number of articles 29 may be limited by analyzing the location of the article carrying device 5. Only articlesthat are in a predeterrnined distance from the article carrying device 5 Will be consideredto be candidates When the system 100 operates to identify the chosen article. Thus in thesecond identification step the registration system 100 Will use a modified version of thelist generated When an article is taken from the article containing area. The list ofpossible article candidates available during the second identification step Will onlycomprise information relating to articles that have been removed from article containersWithin a predeterrnined distance from the article carrying device 5, and Within apredeterrnined time. The number of possible articles Which corresponds to the articlethat has been placed into the article carrying device 5 is thus greatly reduced.
Hence, the registration system 100 uses the list to identify that the identifiedarticle corresponds to the right registration account 130. If there are several possiblecandidates the registration system 100 identifies the article once the article is placed intothe article carrying device 5 in the same or similar manner as the first identificationprocess. The second identification step may be performed When the article isapproaching the article carrying device 5, such that the article is moving across a virtualboundary set at a predeterrnined distance from the article carrying device 5. The virtualboundary may coincide With the physical dimensions of the article carrying device 5.
As With the first identification step, the identification may be achieved usingdifferent identification techniques. One such technique may be an identificationalgorithm, Which uses information gathered from the sensor arrangement 110 as input.In the second identification step, the input to the algorithm is taken from sensor(s)arranged such that the sensor(s) oversees the article carrying device 5 in the store. Thecontent of the article carrying device 5 may be analyzed continuously so that it ispossible to detect once an article is placed into, or close to the article carrying device 5as explained above. The at least one sensor 112 collects data from the shopping cartbefore and after the customer approaches it and it is thus possible to detect When anarticle has been added or removed from the article carrying device 5. The sensorarrangement 110 may use different identification algorithms in order to identify the articles being arranged in the shopping basket.
In one embodiment the system continuously monitor the article carrying device5 by means of the sensor arrangement 110 and receives a first image, from the sensorarrangement 110, of at least a portion of a customer°s hand and/or arm being capturedWhen the customer°s hand and/or arm is moving across a virtual boundary set at apredeterrnined distance from the article carrying device 5. In this image the customer isholding an article Which he/she intends to put in the article carrying device. In oneembodiment, the system Will make the final identification of the article, among thereduced number of articles on the temporary article list, at least in part based on thisfirst image.
The system may, in some embodiments, further receive a second image of atleast a portion of a customer°s hand and/or arm being captured When the customer°shand and/or arm is moving away from the virtual boundary set at a predeterrnineddistance from the article carrying device 5. In this image, the customer has placed thearticle into the article carrying device 5 and is thus no longer holding the article. In oneembodiment, the system Will make the final identification of the article, among thereduced number of articles on the temporary article list, at least in part based on the firstand second image.
Additionally, or altemative, in some embodiments, the system is arranged toreceive an image of the interior of the article carrying device 5 prior to the customer"shand and/or arm moving across a virtual boundary set at a predeterrnined distance fromthe article cariying device 5 and to receive an image of the interior of the articlecarrying device 5 after the customer°s hand and/or arm is removed from the virtualboundary set. The system is further arranged to identify, based at least in part on acomparison of the tWo images, that an item has been added to, removed from orrearranged in the article carrying device 5. In some embodiments, the system is furtherarranged to identify the article, among the reduced number of articles on the temporaryarticle list, based at least in part on said comparison of the images. In otherembodiments, the system is arranged to identify the article, based at least in part onsaid comparison of the images. Hence, in some embodiments the use of a temporary article list is not needed. 31 If the registration system 100 failed to completely identify the article onceremoved from the article container, the system 100 gets another chance of identifyingthe article by analyzing the article carrying device 5 as has been described above. If thearticle Was not identified When picked from the article container, all articles that WereWithin a predeterrnined distance from said article container Will be available in the list.
As soon as the article has been detected and identified, the registration system100 may determine Whether the article Was really picked up by the customer or if it Wasput back in the article containing area A-I. This may be done by analyzing sensor datafrom the article containing area before and after the customer approached it. It is alsopossible to use sensor data for searching article specific pattems inside the customers"article carrying device 5. In one embodiment, in case a customer retums an article thatis already registered to its account, the registration system 100 recognizes that theidentified article that Was retumed and removes the article from the registration account1 3 0.
In one embodiment, the processor 120 is further configured to receive asecond image from the sensor arrangement 110 of at least a portion of a customer°shand and/or arm being captured When the customer°s hand and/or arm is moving awayfrom the virtual boundary set. The processor 120 deterrnines, based at least in part on acomparison of the first image and the second image, if the first image includes arepresentation of an article held in the customer°s hand and/or arm.
If there is an article in the first image, the processor 120 deterrnines if thesecond image includes a representation of an article. If it is deterrnined that the firstimage includes a representation of an article and that the second image includes arepresentation of a different article as of the first image, both articles are identified. Thearticle identity from the first image is added to the registration account, and the articleidentity of the second image is removed from the registration account. If it isdeterrnined, based at least in part on a comparison of the first image and the secondimage, that the first image includes a representation of an article held in the customer°shand and/or arm and that the second image includes a representation of the same articleas of the first image, the article is not added to the dynamic list of articles of the registration account. 32 If it is deterrnined, based at least in part on a comparison of the first image andthe second image, that the first image does not include a representation of an article andthat the second image includes a representation of an article held in the customer°s handand/or arm, the method further comprises the step of identifying said article at least inpart based on the second image from the article identities currently included in thedynamic list of articles of the registration account 130. The processor is then configuredto remove the identity of the article which was identified in the second identificationfrom the dynamic list of articles of the registration account 130.
Once the registration system 100 has identified the article, and possiblyVerified that it was actually collected by the customer or added to the article carrying device 5, the article identity is added to the associated registration account 130.
Article carrving device - Adding Weight priced articles to a registration account In Fig. 4b a method for adding weight priced articles to a registration accountassociated with an article carrying device 5 is schematically shown. The followingdescription is related to the case where the registration account 130 is associated with anarticle carrying device 5, with or without the use of an extemal device.
In Fig. 4b some of the method steps are identical to the method steps of Fig. 4a.Therefore, these are assigned the same reference numerals.
In step 408 the system detects that an article is being placed on an articleweighing device located in the store. The article weighing device is preferably forrningpart of the sensor arrangement 110. This event triggers the following steps. The positionof the article weighing device is deterrnined and following this, in step 410 the list ofpreviously identified articles is fetched, and the number of possible articles is reducedby a factor corresponding to the position of the article weighing device. For this, it ispossible that all articles being picked by customers are added to the same list of possiblearticles. This means that as soon as an article is picked from the article containing area,it is added to the list (optionally together with possible altematives). The articles remainon the list until an article is added to a registration account or it is added to the scale for price calculation by weighing, at which point the article is also removed from the list. 33 Any additional articles that were linked to the positively identified article are alsoremoved from the list.
However, in step 412 the number of articles on the list is reduced in order todetermine the correct article. This reduction is however not made permanent to the list,but only used as a selection for the final deterrnination step 414. The reduction ofnumber of articles is made based on the position of the article weighing device, suchthat only the articles being present on the list and norrnally belonging to a defined andlimited area surrounding the position of the article weighing device is used for the finalidentification before weighing. In step 414, the correct article is identified based on thereduced number of articles from the list. This final identification before weighing maye. g. be made by monitoring the article weighing device, and determine, by means of thesensor arrangement 110, the correct identity of the article. This step is greatly simplifiedby only considering the reduced number of possible articles. If the identification in step414 can°t determine a unique article, then the customer selects the correct article on thearticle weighing device screen. The possible articles to select from can either be thereduced articles from step 412 or all weight priced articles in the store. In step 417 is thearticle weighed by the article weighing device and the price is then calculated. In step418 is the article again added to the list of identified articles together with the calculatedprice and measured weight.
In step 208 the system detects that an article is being placed into an articlecarrying device 5. This event triggers the following steps. The position of the articlecarrying device 5 is deterrnined and following this, in step 210 the list of previouslyidentified articles is fetched including the weight priced article that was added in step418, and the number of possible articles is reduced by a factor corresponding to theposition of the article carrying device 5. For this, it is possible that all articles beingpicked by customers are added to the same list of possible articles. This means that assoon as an article is picked from the article containing area, it is added to the list(optionally together with possible altematives). The articles remain on the list until anarticle is added to a registration account, at which point the article is also removed from the list. There is also a possibility, in one embodiment, to track the customers between 34 step 418 and step 208, to simplify the identification when placing the weight pricedarticle into the article carrying device 5.
However, in step 212 the number of articles on the list is reduced in order todetermine the correct article. This reduction is however not made perrnanent to the list,but only used as a selection for the final deterrnination step 214. The reduction ofnumber of articles is made based on the position of the article carrying device 5, suchthat only the articles being present on the list and norrnally belonging to a defined andlimited area surrounding the position of the article carrying device 5 is used for the finalidentification. In step 214, the correct article is identified based on the reduced numberof articles from the list. This final identification may e. g. be made by monitoring thearticle carrying device 5, and determine, by means of the sensor arrangement 110, thecorrect identity of the article. This step is greatly simplified by only considering thereduced number of possible articles. In a final step 216 the article is added to theregistration account 130 and the customer is preferably notified that the article has beenadded to the list of his registration account 130.
If the registration system 100 fails to identify the article correctly, e.g. if amatching probability computed by the registration system 100 is not above a predefinedthreshold, in step 212 the system 100 notifies the customer to manually identify thearticle. Once the customer has manually added the article, the article identity is added tothe registration account 130.
The user interface e. g. provided by the application of the extemal device, or bya display arranged on the article carrying device 5, may also allow customers tomanually add articles to the registration account without first trying automaticidentification. This may e. g. be particularly suitable for bulk goods, whereby theregistration system 100 may automatically detect that the customer (i.e. the articlecarrying device 5) is arranged in an area of the store where bulk goods is present. Bysuch detection, the registration system 100 may prompt the customer to manually inputpicked articles, whereby these articles are stored in the registration account 130.
In one embodiment, as shown in Fig. 4c, the method of automatic registrationof an article, where the registration account 130 is associated with an article carrying device 5, with or without the use of an extemal device, does not comprise the first identification step. Hence, the identification of the article is performed solely by the useof the second identification step. The method comprises, associate 310 an articlecarrying device 5 With a specific registration account 130. The sensor arrangement 110is configured to continuously monitor 312 the article carrying device 5. The processor isconfigured to receive 314 a first image from the sensor arrangement 110 of at least aportion of a customer"s hand and/or arm being captured When the customer°s handand/or arm is moving across a virtual boundary set at a predeterrnined distance from thearticle carrying device 5. The processor is further configured to determine, based at leastin part on the first image, if the image contains an article. In other Words, the processoris configured to determine 316 if the first image includes a representation of an articleheld in the customer°s hand and/or arm. If the image includes a representation of anarticle, the article is identified 320. The identity of the identified article is added 322 tothe dynamic list of articles of the registration account 130.
In an optional step, once the system detects an arm and/or hand is movingacross a virtual boundary set, the position of the article carrying device 5 is registered.Based on the position of the article carrying device, the number of possible articlesWhich the first image represents is limited. Preferably, only article identities that arearranged on an article containing area Within a predeterrnined area are used Whenmaking the identification of article.
In one embodiment, as shown in Fig. 4d, the processor 120 is furtherconfigured to receive 323 a second image from the sensor arrangement 110 of at least aportion of a customer"s hand and/or arm being captured When the customer°s handand/or arm is moving away from the virtual boundary set. The processor 120deterrnines, based at least in part on a comparison of the first image and the secondimage, if the first image includes a representation of an article held in the customer°shand and/or arm.
If there is an article in the first image, the processor 120 deterrnines 324 ifthe second image includes a representation of the article of the first image. If it does notinclude the same article, the article of the first image is identified and added to the registration account. 36 If it is deterrnined 324, based at least in part on a comparison of the first imageand the second image, that the first image includes a representation of an article held inthe customer°s hand and/or arm and that the second image includes a representation ofthe same (in step 326) article as of the first image, the article is not added 328 to thedynamic list of articles of the registration account. In some embodiments, the article isidentified before step 324. In some embodiments, the article is identified after step 324.In yet some embodiments, Where it is deterrnined that the article in the first and secondimage are the same, the article is not identified at all.
If it is deterrnined in step 326, that the first image includes a representation ofan article and that the second image includes a representation of a different article as ofthe first image, both articles are identified 330. The article identity from the first imageis added to the registration account, and the article identity of the second image isremoved from the registration account in step 332. As should be understood, the orderof these steps may differ, for example the article in the first image may be identified andadded to the registration account before the second article is identified and removedfrom the registration account, and vice versa.
If it is deterrnined, based at least in part on a comparison of the first image andthe second image, that the first image does not include a representation of an article andthat the second image includes a representation of an article held in the customer°s handand/or arm, the method further comprises the step of identifying 334 said article at leastin part based on the second image from the article identities currently included in thedynamic list of articles of the registration account 130. The processor is then configuredto remove 336 the identity of the article Which Was identified in the second identification from the dynamic list of articles of the registration account 130.
Article carrying arrangement - being a part of the customer In one embodiment the article carrying arrangement 5 is a virtual arrangement,Where the arrangement 5 is a part of the customer°s upper body. In one embodiment,the method for automatic registration of an article in a store comprises the steps ofassociating an article carrying arrangement 5 With the specific registration account 130, Where the registration account 130 is associated With a dynamic list of articles as been 37 described in previous embodiments. The method further comprises the step ofcontinuously monitor the at least one article containing area A-I by means of the sensorarrangement 110, and detect a change in the at least one article containing area A-I and,in response to said detected change, identify one or more possible articles causing thechange in a first identification step and save said one or more possible articles to atemporary article list.
The method further comprises the step of continuously monitor the articlecarrying arrangement 5 by means of the sensor arrangement 110. Hence, if the articlecarrying arrangement 5 is a part of the customer°s upper extremity, the hands and/orarms may be monitored. The virtual boundary set at a predeterrnined distance from thearticle carrying arrangement may for example be a distance from the customer°s armsand/or hands. The virtual boundary may also be a distance in one or tWo direction fromthe center point of the customer°s upper extremity.
In this embodiment, the method further comprises the step of receiving afirst image from the sensor arrangement 110 of at least a portion of an article beingcaptured When at least a part of the article is moving across a virtual boundary set at apredeterrnined distance from the article carrying arrangement 5. In a next step,determine, based at least in part on the first image, the identity of the article beingrepresented in said first image in a second identification step from the article identitiescurrently included in the temporary article list and add the identity of the article WhichWas identified in the second identification to the dynamic list of articles of theregistration account 130.
In yet one embodiment, a method for automatic registration of an article in astore comprises the steps of associate an article carrying arrangement With a specificregistration account and continuously monitor the article carrying arrangement bymeans of the sensor arrangement. The method further comprises the step of receiving afirst image from the sensor arrangement of at least a portion of an article being capturedWhen at least a part of the article is moving across a virtual boundary set at apredeterrnined distance from the article carrying arrangement, determine, based at least in part on the first image, the identity of the article being represented in said first image, 38 and add the identity of the identified article to the dynamic list of articles of theregistration account.
Although the two above described embodiments receives an image of an articleand not a hand and/or arrn and/or other part of the upper extremity of the customer, thesame procedure using a plurality of images to determine if the article is to be placedand/or removed from the registration account as has been described with the other embodiments herein apply.
Customer - registration account In the previous, the description has been focused on the embodiments wherethe registration account 130 is associated with an article carrying device 5. Anembodiment where the customer is directly associated with a registration account 130will now be described. The connection between the registration account 130 and thecustomer 1 may be done in several ways. In one embodiment the customer 1 is trackedinside the store using image recognition software, such as person tracking system, 3Dimage tracking from area cameras or other person-tracking algorithms known per se.The sensor arrangement 110 may e.g. be used for providing input data to suchpositioning. The registration system 100 will therefore know the position of thecustomer 1 by following the customer as he or she moves around the store 10. Hence,no further position detecting unit may be needed. However, in order to be able tomanually register articles and to perform a check-in operation the customer may usesome extemal device associated with the registration account 130.
In one embodiment the customer is connected to a registration account 130 bythe use of an extemal device such as a mobile application on the smart phone or tabletof the customer 1 or any other electronic device carried by the customer.
Once the customer 1 enters the entrance area 12 of the store 10 he or she isassociated with a registration account 130. This may be done e. g. by using a fixed unitarranged in the entrance area or on an extemal device of the customer. In oneembodiment, the entrance of the store is arranged with a NFC-tag. The customer canthus be connected to the registration account 130 by using his/hers extemal device having a NFC-reader and connect with the NFC-tag arranged in the store. In yet some 39 embodiments, the store is arranged With a NFC-reader in the entrance area. Thecustomer is connected to the registration account 130 by a NFC-tag preferably arrangedin an external device of the customer. The NFC-reader of the store thus connects Withthe NFC-tag of the customer.
In some embodiments, the registration system 100 further identifies theappearance of the customer 1 in the entrance area, Which allows the system 100 to trackthe customer throughout the store.
The registration system 100 needs to determine the position of the customer.This may either be done by the extemal electronic device of the customer, if itcomprises a position detecting unit, or the position of the customer may be deterrninedby direct tracking the customer as the customer Walks through the store.
If multiple customers belongs together (such as a family) and arrive to the storeat different time it is possible to connect them to the same registration account 130 bymanually merging their accounts. Preferably, the merge operation is made by thecustomers themselves Without any interaction With store attendants.
If multiple customers belonging together arrive to the store at the same time, itis desirable if the registration system adds these multiple persons to the sameregistration account 130. Preferably, automatic recognition of multiple personsshopping together is possible in the check-in area 12. These customers Will thus beconnected to same registration account 130 during their shopping session. It is alsopossible to merge accounts inside the store (i.e. in the article container area) by use of the extemal device.
Customer - Adding articles to a registration account A method of adding an article to a registration account 130 When theregistration account is associated With the customer Will now be described Withreference to Fig. 6. In a first step 402 the sensor arrangement 110 monitors all articlecontainers A-I arranged in the store. By continuously monitoring an article containingarea, it is possible in step 404 to recognize When a change in the article containing areahas occurred. Such a change may indicate that a customer has selected and fetched an article from the containing area.
Once the registration system 100 recognizes that an article has been picked bya customer, the registration system identifies the article in step 406. The article may beidentified directly as it is removed from the article container A-I using the sensorarrangement 1 10.
If the article Was identified, in step 408, as belonging to the customer pickingthe article from the article container A-I, the article identity is in step 410 added to theregistration account 130 associated With that customer. The customer is preferablynotified that the article has been added to the list of his registration account 130.
The user interface e. g. provided by the application of the extemal device mayalso allow customers to manually add articles to the registration account Without firsttrying automatic identification. This may e.g. be particularly suitable for bulk goods,Whereby the registration system 100 may automatically detect that the customer (orassociated article carrying device 5) is arranged in an area of the store Where bulk goodsis present. By such detection, the registration system 100 may prompt the customer tomanually input picked articles, Whereby these articles are stored in the registration account.
Customer - First and second identification step The description relating to the steps of the first identification process for theembodiment Where the registration account is associated With the shopping cart areapplicable also to the embodiment Where the registration account is associated to thecustomer. More specifically, the system 100 may identify the article based on at leastone image taken of at least a part of the customer"s hand and/or arm When it is in thearticle containing area and/or When the hand and/or arm is removed from the area.
In one embodiment, once the identification process is initiated, preferably bythe sensor arrangement 110 as discussed above, the sensor arrangement 110 Will take animage of at least a portion of a user°s hand and/or arm being captured after the user°shand and/or arm is removed from article containing area. The image Will thus containinformation of the article being picked from the article containing area. Thisinformation may be used, possibly together With other output from the sensor arrangement 110, in order to identify the article. Additionally, or altematively, the 41 image may be used to determine if the customer is holding an article or not. Hence, theinforrnation gathered from the image of the customer°s hand and/or arm may not beused to directly identify the article.
Additionally, or altematively, the sensor arrangement 110 takes an image of atleast a portion of a user°s hand and/or arm being captured When the user°s hand and/orarm is inside the article containing area. This information may be used to determineWhere in the article containing area the hand and/or arm, i.e. the article to be picked, islocated. This information can thus be used to gain more accurate position data.
In one embodiment the sensor arrangement 110 is activated based on one typeof sensors, such as Weight sensors, LIDAR-sensor and/or capacitive sensors. After theactivation, the sensor arrangement 110 takes an image of at least a portion of a user°shand and/or arm being captured When the user°s hand and/or arm is inside the articlecontaining area and one image of at least a portion of a user°s hand and/or arm beingcaptured after the user°s hand and/or arm is removed from article containing area. Thisinformation may be used, possibly together With other output from the sensorarrangement 110, in order to identify the article and/or to gain more accurate positiondata.
In one embodiment, the sensor arrangement 110 is arranged to first take animage of the article containing area When a customer°s hand and/or arm is approachingthe area. In a next step, an image is taken of the customer°s hand and/or arm While beinginside the article containing area. Finally, an image is taken of the customer°s handand/or arm When it is moving away from the article containing area, possibly holding apicked article. These images are evaluated in order to identify the article. Theinformation gained from the images may be position data, knowing if an article Wasremoved or added to the article containing area and/or for identification of the article.
For the embodiments described above the picking position is used to limit thetotal number of possible articles, and the recognition system is used to further limit thenumber of possible articles to a single, positively identified article. HoWever, it mayalso be possible to determine the picking position accurately so that a single article ispositively identified using only position data. As explained above, LIDAR sensors, Weight sensors and/or capacitive sensors may be used to accomplish this. In these 42 situations, it is not necessary to make use of the recognition system for furtheridentification of the article.
When the registration account is associated With the customer, the number ofpossible articles may further be limited by analyzing the location of the customer 1.Only articles that are in a predeterrnined distance from the customer 1 Will beconsidered to be candidates When the system 100 operates to identify the chosen article7. That distance may be chosen differently depending on the size of the store and theaccuracy in deterrnining position of the customer.
In the embodiment Where the registration account is associated to the customerthe second identification step is someWhat different compared to the situation Where theaccount is associated to an article carrying device 5. If the first identification step isinsufficient to identify the article due to a matching probability computed by theregistration system 100 being below a predeterrnined threshold, different options arepossible.
According to a first option, in step 412 (see Fig. 6) the system 100 notifies thecustomer to manually identify the article by scanning a code associated With the article.Once the customer has manually identified the article, the article identity is added to theregistration account.
In a second option, the system 100 may in step 412 alert and instruct thecustomer to manually choose the correct article among a plurality of possible articles.The articles the customer is able to choose from are either the articles that are saved inthe reduced article list or a list of articles having high matching probability. The reducedarticle list is reduced based on the position of the customer, such that only the articlesbeing present on the list and norrnally belonging to a defined and limited areasurrounding the position of the customer is used for the final identification. Thecustomer chooses the correct article identity by using the extemal device, such as amobile phone or a tablet. Once the customer has manually identified the article, thearticle identity is added to the registration account.
In a third option, the customer is not prompted immediately but insteadinformation relating to any unidentified articles is stored in the registration account 130, and the customer Will be notified of such unidentified articles during checkout and 43 requested to manually identify those articles. For manual identification, terrninals may be provided inside the store for allowing the customer to manually identify articles.
The user interface The interaction between the customer and the registration system 100 ispreferably done by the use of a display unit. In one embodiment, the display unit maybe in communication With the processor 120 for displaying information regarding theregistration account 130. The display unit is preferably arranged for use by a customerfor inputting information or for approving the displayed information. The display unitmay be arranged at an article carrying device 5 or as an extemal device, such as amobile phone of the customer. The display unit may be an interactive display or apassive display having buttons arranged in connection thereto so as to be able to interactWith the display. The display may be a LCD-screen, a LED-screen or any other suitablescreen.
The information provided in the display unit may for example be informationrelating to the list of articles that already has been identified as picked by the customer,the total price of the selected articles, offers or deals for different articles in the storeand/or accessible information relating to registration customer account such as previouspurchase history. For this the customer may have the possibility to link the currentregistration account 130 With his customer account for the specific store.
The information provided in the display unit may also relate to identification ofan article. If the sensor arrangement 110 identifies the article, an image or a text may beshown in the display unit and if the user finds the displayed information matching thearticle Which the customer has chosen the customer may approve the identity by manualinput. Further information may be displayed, e. g. Weight and price, Wherein the usermay approve the displayed information if it is correct. The display unit may furthernotify the customer When an article needs manual registration. This may be the case ifthe automatic identification of the chosen article fails. The display unit may thus promptthe customer to manually register the article, for example by using a barcode reader or by scanning or identifying the article in some other Way. 44 The display unit may also be used by the customer if the registration systern100 has identified an article Which the customer does not Want to buy or When thesystem 100 has identified the article Wrongly. The customer may then manually remove the article from the article list of the registration account.
Incorrect article positioning The system 100 may also be used to ensure the correct position of the articles,for example, if a customer retums an article at an incorrect position. By continuouslymonitoring the article containing area A-I by the sensor arrangement 110, the automaticin-store registration system 100 may recognize When a customer adds an article back tothe article containing area A-I. The system 100 is thus preferably arranged todifferentiate between When an article is added or removed from the article containingarea A-I. The added article is identified by the registration system 100 and it isdeterrnined if the identif1ed article is an article belonging to that position in that articlecontaining area A-I, i.e. in the article containing area A-I that the automatic in-storeregistration system 100 is trained for.
The article may be added to a different article containing area A-I, if forexample the customer picks out an article from the article containing area A and puts itback to another article containing area D, this may for example occur if the customerrealizes that the article chosen Was in fact not something he/she intended to buy andretums it into a random article containing area D.
If the system 100 deterrnines that the added article Was placed in the Wrong,position in the correct article container A-I or if the article Was placed in the Wrongarticle container A-I, the system 100 Will transmit an alarm signal. Preferably, the alarmsignal is associated With the position of the article containing area in question Wherebyit is possible for a store attendant to correct the position of the Wrongly arranged article.The alarm signal may also be transmitted to the processor for storing and eValuating the data.
Payment Once all the articles of the customer are correctly identified by the registrationsystem 100, and When the customer enters the checkout area 16, the processor 120transmits a transaction signal to the checkout station or the checkout station can retrievethe account to allow the customer to finish the payment transaction. Once the customerenters the checkout area 16 the information stored in the registration account 130 isautomatically transmitted to the checkout station or the checkout station retrieves theaccount. The checkout station then transfers the registered articles to a receipt forpayment.
In one embodiment, the checkout station transmits a control signal to asurveillance system Which Will be used in the deterrnination Whether or not to open apassage gate. A person is in such embodiments only allowed to exit through the gate(s)18 of the store 10 if the person 1 has paid for its article(s). This is deterrnined based oninformation from the registration system 100 and information regarding the payment.The system may initiate security check based on abnormal activity of a certain registration account.
Special conditions for certain articles In one embodiment, some articles may be associated With a special condition.Such condition may for example be articles that are provided With age restrictions suchas alcoholic beverages or pharrnaceuticals. It is preferred if the system is capable toprevent articles With age restrictions to be purchased by minors in order to fulfildifferent regulations.
Some article identities may be arranged With a condition flag. Once an article isidentified as an article identity having a condition flag the registration account Will intum be provided With a flag. A registration account being provided With a flag Will needassistance from a store attendant, this process may be initiated in different Ways.
In one embodiment, the customer Will receive a notif1cation that a conditionarticle has been chosen. This may be notif1ed to the customer using the article carrying device 5 if present and/or by an extemal device. 46 In one embodiment, the customer Will then be instructed to go to a special areaof the store to get an evaluation of the store attendant if the conditions of the article isfulfilled. Such an area may for example the exit area l6 of the store.
In an altemative embodiment, the customer is notified about the articlecondition and the system Will evaluate if there is enough information about the customerstored in the registration account. If enough data is stored in connection With thecustomer and the conditions are fulfilled, the condition flag is removed automatically.This may for example be the case if the customer Was identified as being at a specificage When registration the registration account.
A special condition of an article may also include an alarm feature. Forexpensive and/or article being small in size that are easier to steel, the article may bearranged With an alarm device. The alarm device may be an alarm tag, a RFID-tag, abarcode or similar. Such an article is thus arranged With a condition flag in order tonotify the customer and/or the store attendant that the alarm device needs to be removedand/or deactivated. In one embodiment, the customer Will then be instructed to go to aspecial area of the store to get the alarm removed and/or deactivated by the store attendant. Such an area may for example the exit area l6 of the store.
Optional Weight sensor at exit In one embodiment, Where the registration account is associated With an articlecarrying device 5, such as a shopping cart, a bag, basket or the like, a Weight sensor maybe arranged in the exit area l6 of the store to act as a safeguard to the registrationsystem l00. The Weight sensor may either be provided before or after the paymentprocess. The total Weight of the article carrying device 5, containing all picked articles,are compared to the estimated Weight based on the Weight of an empty article carryingdevice 5 and the total Weight of the identified articles or it is compared to a trainedWeight Which is saved in the processor 120 for all possible article carrying devices suchas shopping cart, bag, basket or the like. If all articles inside the article carrying device 5are identified by the registration system l00, the estimated Weight and the actual Weight shall be equal, or the difference shall be Within a small perrnissible range. If the actual 47 Weight does not sufficiently correspond to the estimated Weight, such information istransmitted to the processor 120 and the customer and/or a store attendant is alerted.
If the article carrying device 5 has a Weight lower than the estimated Weight,the registration systern 100 has either Wrongly identified an article or an identifiedarticle has been removed from the article carrying device 5 Without being correctlyremoved from the registration account 130.
If the article carrying device 5 has a Weight higher than the estimated Weight,the registration system 100 has either Wrongly identified an article or an unidentif1edarticle has been placed into the article carrying device 5 but not being added to theregistration account 130. If an article has been placed in the article carrying device 5 buthas not been added to the registration account 130, the customer may then perform manual registration of that article.
System leaming The automatic in-store registration system 100 may be subject to training, orleaming, in order to improve the accuracy of the identification of articles. For example,the sensor arrangement 110 may be activated during predeterrnined training sessions, inWhich a store attendant picks articles in a consecutive order. For each picked articletype, the attendant scans at least one article for a secure identification. The system 100Will thus not only leam to identify the article correctly, but also the exact position of thearticle. For deterrnining the position of the article it is preferred to know the position ofthe scanner, either by detecting the position by means of the sensor arrangement 110itself, or by associating the article position (i.e. the shelf or similar) With a unique codeWhich is scanned together With the article code. As a third option the scanner maycomprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE),short-range radio frequency positioning and/or Wifi-based positioning systems.
System leaming may further be improved by using checkout counters, eithermanually operated, semi-automatic operated, or fully automatically operated.
The training, or leaming, may also occur during normal operation of the store,Where the store attendant adds or ref1lls articles into the article containing area. For each picked article type, the attendant scans at least one article by a handheld device. The 48 sensor arrangement is thus trained With every single article which the store attendantadds into the article container. Even if the scanner is used only for one or a few articlesof each article type, the sensor arrangement ll0 can register signatures and position ofeach article for every article type. The system l00 will thus leam to identify the articlecorrectly by using both the appearance of the article as well as the exact position of thearticle. Hence, in one embodiment the in-store registration system is trained by defininga virtual boundary set at a predeterrnined distance from the article carrying arrangement,allowing a store attendant or a customer to add or remove one or more articles to orfrom the article carrying arrangement, detecting at least a portion of the upper extremityof a person when at least a part of the person°s upper extremity is moving across thevirtual boundary set, registering one or more signatures of each article by means of asensor arrangement ll0, and adding or updating information relating to signature andthe position of each article to the in-store registration system l00.
In one embodiment the sensor arrangement ll0 may be activated duringpredeterrnined training sessions in order to leam the system to recognize differentmovements of a user°s (customer or store attendant) hand/ and or arm while being in ornear to the article containing area and/or the article carrying device. The system ispreferably leamed to differentiate between different movement directions, i.e.fron1/away from the article containing area and/or the article carrying device. This isbenef1cial when deterrnining if the article is fetching an article from the articlecontaining area and/or the article carrying device and when deterrnining if an article isremoved from the article containing area and/or the article carrying device.
In one embodiment, the automatic in-store registration system l00 is trained orleamed using synthetic training data. The system is trained on a synthetically generateddataset with the intention of transfer leaming data to real data. The use of synthetic datahas several advantages, for example once the synthetic environment is ready it is fastand cheap to produce as much data as needed, and the synthetic environment can bemodified to improve the model and training. Moreover, synthetic data can be used as asubstitute for certain real data segments that contain, e. g., sensitive information.
In a preferred embodiment, the synthetic environment is a 3D-model of the store and/or of the article carrying device and/or the article containing areas. Hence, the 49 training data may comprise of a synthetically generated 3D-model of at least a part ofthe store and/or the customer and/or the article containing areas.
The synthetic environment is used to train the system to recognize customer ofdifferent skin colors and skin Variations. Additionally, the system is trained to recognizecustomers of different sizes, such as being of different height and weight.
The synthetic data may be enhanced by using Generative Adversarial Networks(GAN). GAN is able to adapt the synthetic data so that it increases it resemblance to thereality. GAN is a deep neural net architectures comprised of two nets, pitting oneagainst the other.
GAN will now shortly be described. A neural network generates new datainstances, this neural network may be referred to as a generator. The generator takes inrandom numbers and then retums an image, this image is then feed into another neuralnetwork called the discriminator. The discriminator evaluates the data instances forauthenticity, hence this neural network decides whether each instance of data it reviewsbelongs to the actual training dataset or not. The image received from the generator istransmitted to the discriminator together with a stream of images taken from the actualdataset. The discriminator is arranged to receive both real and fake images. Based onthese images it retums probabilities in the form of a number between 0 and l. Thenumber 0 represents a fake image and with the number l represents a prediction ofauthenticity. In a next step, a double feedback loop is created. The discriminator is in afeedback loop with the ground truth of the images, while the generator is in a feedbackloop with the discriminator. During the training phase the generator will continuouslyimprove and eventually be able to generate images that closely resembles real data.
It should be appreciated that even though numerous characteristics andadvantages of the present invention have been set forth in the foregoing description,together with details of the structure and function of the invention, the description isonly illustrative and changes may be made in detail, especially in matters of shape, sizeand arrangement of parts within the scope of the invention to the full extent indicated by the appended claims.
Claims (40)
1. A method for automatic registration of an article in a store having at leastone article containing area (A-I) and an associated automatic in-store registrationsystem (100), Wherein the method comprises: associate an article carrying arrangement (5) With a specific registrationaccount (130), Wherein the registration account (130) is associated With a dynamic listof articles; continuously monitor the at least one article containing area (A-I) by means ofa sensor arrangement (110), and detect a change in the at least one article containingarea (A-I) and, in response to said detected change, identify one or more possiblearticles causing the change in a first identification step and save said one or morepossible articles to a temporary article list; continuously monitor the article carrying arrangement (5) by means of thesensor arrangement (110); receive a first image from the sensor arrangement (110) of at least a portion ofa customer°s upper extremity being captured When at least a part of the customer°supper extremity is moving across a virtual boundary set at a predeterrnined distancefrom the article carrying arrangement (5); determine, based at least in part on the first image, if the first image includes arepresentation of an article, and if so, identify said article in a second identification stepfrom the article identities currently included in the temporary article list; and add the identity of the article Which Was identified in the second identification to the dynamic list of articles of the registration account (130).
2. The method according to claim 1, Wherein the method fiarther comprisesthe step of:receive a second image from the sensor arrangement (110) of at least a portionof a customer°s upper extremity being captured When the at least a part of the customer”s upper extremity is moving away from the virtual boundary set; determine, based at least in part on a comparison of the first image and thesecond image, if the first image includes a representation of an article and if sodetermine if the second image does not include a representation of the article of the firstimage, and if so, identify the article of the first image in a second identification stepfrom the article identities currently included in the temporary article list, and add the identity of the article Which Was identified in the second identification to the dynamic list of articles of the registration account (130).
3. The method according to claim 2, Wherein if it is deterrnined that the firstimage includes a representation of an article and that the second image include arepresentation of the article of the first image, the method further comprises the step of:determine if the first image includes a representation of an article and that thesecond image includes a representation of the same article as of the first image,and if the first and second images includes representation of different articles:identify the article of the first image and add the identity of the article tothe dynamic list of articles of the registration account (130); andidentify the article of the second image and remove the identity of the article from the dynamic list of articles of the registration account (130).
4. The method according to claim 3, Wherein if it is deterrnined that the firstimage includes a representation of an article and that the second image includes arepresentation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account (130).
5. The method according to any one of claims 2 to 4, Wherein if it isdeterrnined, based at least in part on a comparison of the first image and the secondimage, that the first image does not include a representation of an article and that thesecond image includes a representation of an article, the method further comprises the step of: identifying said article from the second image in a second identification stepfrom the article identities currently included in the dynamic list of articles of theregistration account (130), and remove the identity of the article Which Was identified in the second identification from the dynamic list of articles of the registration account (130).
6. The method according to any one of the preceding claims, Wherein themethod further comprises the step of: receive a third image from the sensor arrangement (110) of the virtualboundary set prior to at least a part of the customer°s upper extremity is moving acrossthe virtual boundary set; receive a fourth image from the sensor arrangement (110) of the virtualboundary set after to at least a part of the customer°s upper extremity is moving awayfrom the virtual boundary set; determine, based at least in part on a comparison of the first, third and fourthimage, if the article has been added to or removed from the article carrying arrangement(5), identify said article in a second identification step from the article identitiescurrently included in the temporary article list, Wherein if is deterrnined that the article has been added to the article carryingarrangement (5), add the identity of the article Which Was identified in the secondidentification to the dynamic list of articles of the registration account (130) andWherein if is deterrnined that the article has been removed from the article carryingarrangement (5) remove the identity of the article Which Was identified in the second identification from the dynamic list of articles of the registration account (130).
7. The method according to any one of the preceding claims, Wherein the firstidentification step is performed by identifying an article by at least using informationrelating to the position of the change in the article containing area (A-I), Wherein the position of each one of the plurality of articles arranged in the article containing area (A-I) is predeterrnined and accessible for the automatic in-store registration system (100).
8. The method according to claim 7, Wherein the first identification step isperformed by reducing the total number of possible article identities using the position information of the change in the article containing area (A-I).
9. The method according to claim 7 or 8, Wherein the article containing area(A-I) is arranged With a plurality of segments, Wherein the position of each segment is known.
10. The method according to claim 9, Wherein each segment of the articlecontaining area comprises at least one capacitive sensor arranged to detect a change in the article containing area (A-I).
11. The method according to any one of the preceding claims, Wherein thesecond identification step is performed by reducing the total number of possible article identities using the information saved in the temporary article list.
12. The method according to claim 11, Wherein the second identification step isperformed by accessing the article identities saved in the temporary article list andidentifying the article as one of the accessed article identities originally positionedWithin the Virtual boundary set at a predeterrnined distance from the article carrying arrangement (5) and added to the temporary article list Within a predeterrnined time.
13. The method according to any of the preceding claims, Wherein the sensorarrangement (110) comprises at least one sensor (112) being selected from the groupcomprising a 2D-camera, a 3D-camera, an IR camera, a scale, a LIDAR sensor, a capacitive sensor and a spectrometer.
14. The method according to any of the preceding claims, further comprising deterrnining a current position of said article carrying arrangement (5).
15. The method according to any of the preceding claims, Wherein step ofassociating the article carrying arrangement (5) With a specific registration account (130) is perforrned using Near Field Communication.
16. An automatic in-store registration system (100) in a store having at leastone article containing area (A-I), the system comprising: at least one processor (120); at least one registration account (130) being associated With an article carryingarrangement (5); and a sensor arrangement (110) comprising a plurality of sensors (112), the sensorarrangement (110) being configured to, in a first identification step, continuouslymonitor the article containing area (A-I) and to detect a change in the article containingarea (A-I) and, in response to said detected change, to transmit a first identificationsignal comprising information relating to one or more possible articles to the processor(120), and Wherein the sensor arrangement (110) is further configured to, in a secondidentification step, continuously monitor the article carrying arrangement (5) and todetect a change in or near the article carrying arrangement (5) and, in response to saiddetected change, to transmit a second identification signal comprising a first image tothe processor (120); Wherein the processor (120) is configured to identify one or more possiblearticles from the first identification signal and to save said articles to a temporary articlelist, Wherein the processor (120) is further configured to identify one or more possiblearticles from the second identification signal by: receive a first image of at least a portion of a customer°s upperextremity being captured When at least a part of the customer°s upper extremityis moving across a virtual boundary set at a predeterrnined distance from the article carrying arrangement (5); determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list; andWherein the processor (120) is further configured to determine a final article bycomparing the one or more articles identified from the second identification signal Withthe articles stored in the dynamic article list, and to add said final article to the registration account (130).
17. A method for training an in-store registration system (100) configured toidentify and register articles in a store having at least one article containing area (A-I),said method comprising performing a training sequence in order to improve theaccuracy of the identification of articles by: using training data to recognize different movements of a user°s upperextremity While being in or near the article containing area and/or the article carryingarrangement, and to detect at least a portion of a upper extremity of a person When atleast a part of the person°s upper extremity is moving in or near the article containingarea and/or the article carrying arrangement, registering one or more signatures of each article by means of a sensorarrangement (110), and adding or updating information relating to signature and/or the position of each article to the in-store registration system (100).
18. The training method according to claim 17, Wherein the training datacomprises of a synthetically generated 3D-model of at least a part of the store and/or the customer.
19. The training method according to claim 17 or 18, Wherein the synthetic data is enhanced using Generative Adversarial Networks (GAN).
20. A method for automatic registration of an article in a store having at leastone article containing area (A-I) and an associated automatic in-store registrationsystem (100), Wherein the method comprises: associate an article carrying arrangement (5) With a specific registrationaccount (130), Wherein the registration account (130) is associated With a dynamic listof articles; continuously monitor the article carrying arrangement (5) by means of thesensor arrangement (110); receive a first image from the sensor arrangement (110) of at least a portion ofa customer°s upper extremity being captured When at least a part of the customer°supper extremity is moving across a virtual boundary set at a predeterrnined distancefrom the article carrying arrangement (5); determine, based at least in part on the first image, if the first image includes arepresentation of an article, and if so identify said article, and add the identity of the identified article to the dynamic list of articles of the registration account (130).
21. The method according to claim 20, Wherein the method furthercomprises the step of: receive a second image from the sensor arrangement (110) of at least aportion of a customer°s upper extremity being captured When at least a part thecustomer”s upper extremity is moving away from the virtual boundary set; determine, based at least in part on a comparison of the first image and thesecond image, if the first image includes a representation of an article and if sodetermine if the second image does not include a representation of the article of the firstimage, and if so, identify the article of the first image.
22. The method according to claim 21, Wherein if it is deterrnined that thefirst image includes a representation of an article and that the second image include a representation of the article of the first image, the method further comprises the step of: determine if the first image includes a representation of an article and that thesecond image includes a representation of the same article as of the first image,and if the first and second images includes representation of different articles:identify the article of the first image and add the identity of the article tothe dynamic list of articles of the registration account (130); andidentify the article of the second image and remove the identity of the article from the dynamic list of articles of the registration account (130).
23. The method according to claim 22, Wherein if it is deterrnined, based atleast in part on a comparison of the first image and the second image, that the firstimage includes a representation of an article and that the second image includes arepresentation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account (130).
24. The method according to any one of claims 21 to 23, Wherein if it isdeterrnined, based at least in part on a comparison of the first image and the secondimage, that the first image does not include a representation of an article and that thesecond image includes a representation of an article, the method further comprises thestep of: identify said article at least in part based on the second image from the articleidentities currently included in the dynamic list of articles of the registration account(130), and remove the identity of the article Which Was identified in the second identification from the dynamic list of articles of the registration account (130).
25. The method according to any one of claims 20-24, further comprising deterrnining a current position of said article carrying arrangement (5).
26. An automatic in-store registration system (100) in a store having at leastone article containing area (A-I), the system comprising: at least one processor (120); at least one registration account (130) being associated With an article carryingarrangement (5); and a sensor arrangement (110) comprising a plurality of sensors (112), the sensorarrangement (110) being configured to continuously monitor the article carryingarrangement (5); Wherein the processor (120) is configured to: receive, from the sensor arrangement (110), a first image of at least aportion of a customer°s upper extremity being captured When at least a part of thecustomer”s upper extremity is moving across a virtual boundary set at a predeterrnineddistance from the article carrying arrangement (5);determine, based at least in part on the first image, if the first image includesa representation of an article, and if so, identify said article; andWherein the processor (120) further is configured to add said final article to the registration account (13 0).
27. A method for training an in-store registration system (100) configured toidentify and register articles in a store, said method comprising performing a trainingsequence in order to improve the accuracy of the identification of articles by: defining a virtual boundary set at a predeterrnined distance from the articlecarrying arrangement (5); using training data to recognize different movements of a user°s upperextremity While being in or near the virtual boundary set, and to detect at least a portionof a upper extremity of a person When at least a part of the person°s upper extremity ismoving across the virtual boundary set; registering one or more signatures of each article by means of a sensorarrangement (110), and adding or updating information relating to signature and the position of each article to the in-store registration system (100).
28. The training method according to claim 27, Wherein the training datacomprises of a synthetically generated 3D-model of at least a part of the store and/or the customer.
29. The training method according to 27 or 28, Wherein the synthetic data is enhanced using Generative Adversarial Networks (GAN).
30. A method for registration of an article in a store having at least one articlecontaining area (A-I) and an associated automatic in-store registration system (100),Wherein the method comprises: associate a customer With a registration account (130); monitor the movement of the customer through the store, continuously monitor the at least one article containing area (A-I) by means ofa sensor arrangement (110), and detect a change in the at least one article containingarea (A-I) and, in response to said detected change, identify one or more possiblearticles causing the change in a first identification step and save said one or morepossible articles to a temporary article list, Wherein, if said temporary article listincludes two or more articles, positively identifying the article Which caused the changein a second identification step from the articles currently included in the temporaryarticle list by receiving a first image from the sensor arrangement (110) of at least aportion of a user°s upper extremity being captured after at least a part of the user”s upperextremity is removed from the article containing area (A-I), Wherein the first imageincludes a representation of an article; and add the identity of the article Which Was positively identified in the first or second identification to the registration account (130).
31. The method according to claim 30, Wherein the method further comprisesthe step of receiving a second image from the sensor arrangement (110) of at least aportion of a user°s upper extremity being captured When at least a part of the user”supper extremity is inside the article containing area (A-I), Wherein the first and second images are used to identifying the article in the second identification step.
32. The method according to claim 30 or 31, Wherein the second identificationstep further comprises accessing the article identities saved in the temporary article listand identifying the article as one of the accessed article identities originally positionedWithin a predeterrnined distance from the customer and added to the temporary article list Within a predeterrnined time.
33. The method according to any of claims 30 to 32, further comprisingidentifying a plurality of customers, and associating two or more customers to the same registration account (130).
34. The method according to any one of claims 30 to 33, Wherein the sensorarrangement (110) comprises at least one sensor (112) being selected from the groupcomprising a 2D-camera, a 3D-camera, an IR camera, a scale, a LIDAR sensor, a capacitive sensor and a spectrometer.
35. The method according to any one of claims 30 to 33, Wherein the firstidentification step is performed by identifying an article by at least using informationrelating to the position of the change in the article containing area (A-I), Wherein theposition of each one of the plurality of articles arranged in the article containing area(A-I) is predeterrnined and accessible for the automatic in-store registration system (100).
36. The method according to claim 35, Wherein the first identification step isperformed by reducing the total number of possible article identities using the position information of the change in the article containing area (A-I).
37. The method according to claim 35 or 36, Wherein the article containingarea (A-I) is arranged With a plurality of segments, Wherein the position of each segment is known. 61
38. The method according to claim 37, Wherein each segment of the articlecontaining area comprises at least one capacitive sensor arranged to detect a change in the article containing area (A-I).
39. The method according to any one of claims 30 to 38, Wherein step ofassociating the customer With the registration account (130) is perforrned using Near Field Communication.
40. An automatic in-store registration system (100) in a store having at leastone article containing area (A-I), the system comprising: at least one processor (l20); at least one registration account (130) being associated With a customer in theassociated store; and a sensor arrangement (110) comprising a plurality of sensors (112), the sensorarrangement (110) being configured to, in a first identification step, continuouslymonitor the article containing area (A-I) and to detect a change in the article containingarea (A-I) and, in response to said detected change, to transmit a first identificationsignal comprising information relating to one or more possible articles to the processor(120), Wherein the sensor arrangement (110) further is configured to continuouslymonitor the customer; and Wherein the processor (120) is configured to identify one or more articles fromsaid first identification signal and to save the identified one or more articles to atemporary article list, Wherein if the temporary article list includes tWo or more articles,the processor (120) is configured to positively identify the article Which caused thechange in a second identification step from the articles currently included in thetemporary article list by receiving a first image from the sensor arrangement (110) of atleast a portion of a user°s upper extremity being captured after at least a part of theuser°s upper extremity is removed from the article containing area (A-I), Wherein thefirst image includes a representation of an article, and to add the article being positively identified in the first or second identification step to the registration account (130).
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| SE1850757A SE543102C2 (en) | 2018-06-20 | 2018-06-20 | An automatic in-store registration system |
| PCT/SE2019/050589 WO2019245441A1 (en) | 2018-06-20 | 2019-06-19 | An automatic in-store registration system |
| EP19821797.8A EP3811315A4 (en) | 2018-06-20 | 2019-06-19 | An automatic in-store registration system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| SE1850757A SE543102C2 (en) | 2018-06-20 | 2018-06-20 | An automatic in-store registration system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| SE1850757A1 true SE1850757A1 (en) | 2019-12-21 |
| SE543102C2 SE543102C2 (en) | 2020-10-06 |
Family
ID=68983408
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| SE1850757A SE543102C2 (en) | 2018-06-20 | 2018-06-20 | An automatic in-store registration system |
Country Status (3)
| Country | Link |
|---|---|
| EP (1) | EP3811315A4 (en) |
| SE (1) | SE543102C2 (en) |
| WO (1) | WO2019245441A1 (en) |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001128814A (en) * | 1999-11-01 | 2001-05-15 | Atr Media Integration & Communications Res Lab | Article take-in/take-out detector |
| US7693758B1 (en) * | 2007-06-13 | 2010-04-06 | Amazon Technologies, Inc. | Activity verification |
| US10176456B2 (en) * | 2013-06-26 | 2019-01-08 | Amazon Technologies, Inc. | Transitioning items from a materials handling facility |
| US10268983B2 (en) * | 2013-06-26 | 2019-04-23 | Amazon Technologies, Inc. | Detecting item interaction and movement |
| US10290031B2 (en) * | 2013-07-24 | 2019-05-14 | Gregorio Reid | Method and system for automated retail checkout using context recognition |
| US9473747B2 (en) * | 2013-07-25 | 2016-10-18 | Ncr Corporation | Whole store scanner |
| US20150206188A1 (en) * | 2014-01-17 | 2015-07-23 | Panasonic Intellectual Property Corporation Of America | Item presentation method, and information display method |
| US9174800B1 (en) * | 2014-03-25 | 2015-11-03 | Amazon Technologies, Inc. | Verifying bin content in a materials handling facility |
| JP6968399B2 (en) * | 2016-02-29 | 2021-11-17 | サインポスト株式会社 | Information processing system |
| JP7009389B2 (en) * | 2016-05-09 | 2022-01-25 | グラバンゴ コーポレイション | Systems and methods for computer vision driven applications in the environment |
| SE542438C2 (en) * | 2016-12-21 | 2020-05-05 | Itab Scanflow Ab | An automatic in-store registration system |
-
2018
- 2018-06-20 SE SE1850757A patent/SE543102C2/en unknown
-
2019
- 2019-06-19 WO PCT/SE2019/050589 patent/WO2019245441A1/en unknown
- 2019-06-19 EP EP19821797.8A patent/EP3811315A4/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| SE543102C2 (en) | 2020-10-06 |
| EP3811315A4 (en) | 2022-03-23 |
| WO2019245441A1 (en) | 2019-12-26 |
| EP3811315A1 (en) | 2021-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| SE1651711A1 (en) | An automatic in-store registration system | |
| RU2727084C1 (en) | Device and method for determining order information | |
| CN111626681B (en) | Image recognition system for inventory management | |
| US10853702B2 (en) | Method and apparatus for checkout based on image identification technique of convolutional neural network | |
| US10290031B2 (en) | Method and system for automated retail checkout using context recognition | |
| US11908288B2 (en) | Imaging used to reconcile cart weight discrepancy | |
| US7275690B1 (en) | System and method of determining unprocessed items | |
| US10990945B2 (en) | Self-checkout system | |
| US10628695B2 (en) | Fast item identification for checkout counter | |
| US11562614B2 (en) | Method, a device and a system for checkout | |
| US20230037427A1 (en) | Identifying barcode-to-product mismatches using point of sale devices and overhead cameras | |
| CN109166000A (en) | A kind of Intelligent cargo cabinet and the method for creating order | |
| CN109934569B (en) | Settlement method, device and system | |
| SE1751401A1 (en) | Training of an automatic in-store registration system | |
| CN209328138U (en) | A kind of Intelligent cargo cabinet of application image identification device and gravity sensor | |
| CN108764129B (en) | Goods shelf and shopping system | |
| SE1850757A1 (en) | An automatic in-store registration system | |
| US20220270061A1 (en) | System and method for indicating payment method availability on a smart shopping bin |