SE542415C2 - Training of an automatic in-store registration system - Google Patents

Abstract

A method for training an in-store registration system (100) is provided. The method is configured to identify and register articles in a store having at least one article containing area (A-I), said method comprising performing a training sequence in order to improve the accuracy of the identification of articles by allowing a store attendant to add or remove one or more articles to or from the article containing area (A-I), scanning at least one article by a scanning device for identifying the article, registering one or more signatures of each article by means of a sensor arrangement (110), and adding or updating information relating to signature and the position of each article to the in-store registration system (100).

Description

TRAINING OF AN AUTOMATIC IN-STORE REGISTRATION SYSTEM TECHNICAL FIELDThe present invention relates to training of an automatic in-store registration system being configured to automatically registering articles in a store.

BACKGROUND In today”s stores many different articles, such as food products, hygienearticles, clothing, etc. may be purchased Which all have different sizes and shapes, asWell as visual appearance. Norrnally, a checkout operator handles each article manuallyand makes sure that the article is associated With the correct pricing at checkout forcorrect payment by the customer. This is traditionally done either by manually inputtingthe price, by scanning a barcode attached to the article, or a combination of those.However, semi- or fully automatic checkout counters are becoming an interestingaltemative for retail stores and supermarkets. Such checkout counters provides robustand easy identification and handling of articles and they norrnally include differentcomponents for identifying the article accurately.

However, in prior art systems the customer still needs to perform the criticalstep of feeding the articles through a checkout counter in order for the articles to beidentified. The customer is required to queue in checkout lines and pass each productfrom the shopping cart or basket to a scanning system. Therefore, there is a need for animproved registration system that is fast, reliable, and robust and thus has an increased security level While at the same time being easy and convenient for the customer.

SUMMARY An object of the present invention is therefore to provide an improved methodfor automatic registration of articles in a store.

According to a first aspect a method for training an in-store registration systemconfigured to identify and register articles in a store having at least one articlecontaining area is provided. The method comprises performing a training sequence in order to improve the accuracy of the identification of articles by: allowing a store attendant to add or remove one or more articles to or from the article containing area,scanning at least one article by a scanning device for identifying the article, registeringone or more signatures of each article by means of a sensor arrangement, and adding orupdating information relating to signature and the position of each article to the in-storeregistration system.

The sensor arrangement may comprise at least one sensor being selected fromthe group comprising a 2D-camera, a 3D-camera, an IR camera, a scale, a LIDARsensor, and a spectrometer.

The method may further comprise deterrnining the position of the article bymeans of the sensor arrangement itself, by associating the article position with a uniquecode which is scanned together with the article code, and/or by means of a positiondetecting unit provided at the scanning device.

The steps of scanning and registering one or more signatures may be performedusing checkout counters, either manually operated, semi-automatic operated, or fully automatically operated.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention will be described in the following; referencebeing made to the appended drawings which illustrate non-limiting examples of how theinventive concept can be reduced into practice.

Fig. l schematically shows a top view of a store having a registration systemaccording to an embodiment of the present invention; Fig. 2 shows an isometric view of parts of a store having a registration systemaccording to an embodiment of the present invention; Fig. 3 shows a schematic view of a registration system according to anembodiment of the present invention; Fig. 4a shows a schematic method for automatic registration according to anembodiment of the present invention; Fig. 4b shows a schematic method for automatic registration according to an embodiment of the present invention; Fig. 5 shows a schematic view of the limitation process of a registrationmethod according to an embodiment of the present invention; andFig. 6 shows a schematic method for automatic registration according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS Fig. 1 shows an illustrative view of a designated area, such as a retail orgrocery store 10. The store 10 comprises a plurality of article containing areas A-I, suchas shelves, displaying articles available for purchase. The store 10 may also include anentrance or check-in area 12, an entrance gate 14, an exit or check-out area 16, an exitgate 18 and at least one checkout station 20. The entrance gate 14 and the exit gate 18may be the same gate. Furtherrnore, the entrance area 12 and the exit area 16 may be thesame or they may overlap.

A customer enters the store 10 by passing through the entrance gate 14 and intothe entrance area 12, whereby the customer walks through the aisles between the articlecontaining areas A-I and collects the articles to be purchased. The article will be, at leastin part, identified by the automatic in-store registration system 100 described hereinonce the article is removed from its original position at the article containing areas A-I.Providing that the identification by the registration system 100 is successful, thecustomer does not need to scan the article, or provide any other manual input. Once thecustomer is finished fetching the articles to be purchased, the customer pays the amountcorresponding to the chosen articles in the checkout station 20 and exits through the exitarea 16 out of the exit gate 18. The total amount to be paid is automatically deterrninedby the registration system 100. The automatic in-store registration system 100 may alsouse information relating to discounts, coupons, sales-items etc. in the deterrnination ofthe total amount.

Fig. 2 shows an embodiment of the registration system 100 monitoring onearticle containing area A, here in the form of a shelf, comprising a plurality of differentarticles. The registration system 100 comprises a sensor arrangement 110 comprising aplurality of sensors 112 which at least monitors the shelf A per se and/or the articles arranged on the shelf A. In some embodiments, the sensors 112 also monitor a customer 1 and/or a article carrying device 5 used by the customer 1, as will be described infurther details below. Furthermore, the registration systern 100 comprises a registrationaccount 130, which together with the sensor arrangement 110 will be described more indetail with reference with Fig. 3.

Although the article containing area A shown in Fig. 2 is the forrn of a shelf,the article containing area may also take the form of a basket, tray, freezer, fridge and/orother similar arrangements which can contain and display articles to a customer. Thearticles may be of different dimensions, structure and colors as illustrated in Fig. 2.Preferably, all articles being the same type and/or brand are arranged next to each otherto facilitate selection and retrieving by the customer 1.

The article carrying device 5 in Fig. 2 is in the form of a shopping cart, but itshould be understood that the article carrying device 5 also may be a basket, a bag orsimilar container which the customer 1 may use to collect and transports its chosenarticles. In another embodiment, the article carrying device 5 may be a stationary, butpre-defined area.

In Fig. 2 the customer 1 has fetched an article 7 from the shelf A and has theintention to put down the article into the article carrying device 5. This selected article 7thus needs to be identified in order to be added to the registration account 130 so thatthe customer can pay for the article 7. For this purpose the sensors 112 monitor thearticle containing area A and all the articles arranged thereon, at least the articlesarranged at the front. By continuously monitoring the article containing area A, it ispossible to recognize when a change in the article containing A has occurred. Such achange may indicate that a customer 1 has picked an article 7 from the article containingarea A. The registration system 100 is configured to have access to information relatingto which types of articles being present on each article containing A-I and theregistration system 100 is configured to use such position information together withanalyzing data from the sensors 112 in order to identify the identity of the article 7 being selected and picked by the customer 1.

TllC SCIISOI' aITaIIgCIIICIIÉ As explained above the sensor arrangement 110 is provided in order toautomatically detect the articles 7 being picked by the customer 1 throughout the store10. The sensor arrangement 110 thus forrns part of the registration system 100, Whichalso, as indicated in Fig. 3, comprises a processor 120, being connected to a memory122, and a registration account 130 being in operative communication With theprocessor 120. The sensor arrangement 110 is configured to be in communication Withthe processor 120 as Will be explained further below.

The sensor arrangement 110 comprises at least one sensor 112. However, asindicated by the dotted lines in Fig. 3, the sensor arrangement 110 may comprise aplurality of sensors 112. Preferably, the sensor arrangement 110 comprises a plurality ofsensors 112 to create a full coverage of the articles 7 arranged in the store. Furthermore,a plurality of sensors 112 may be arranged to cover different views of the article, so thatfor example an upper, lower, left, right, front, and back side of the article 7 could bemonitored by the sensors 112. The sensors 112 may be arranged on the article containeritself, on the Walls surrounding the article containers and/or on the ceiling of the store.

Additionally or altematively, one or a plurality of sensors may be arranged onan article carrying device 5, such as on a shopping cart. If at least one sensor is arrangedon a movable article carrying device 5, the sensor can be used to continuously generatedata and thus generate a geometry of the store.

The sensor 112 may be conf1gured to provide information corresponding to thedimensions and outer appearance of the article, as Well as the position of Where thearticle Was picked in the article containing area A-I . The sensor 112 may furtherprovide information relating to size, color, shape, contour, marking With barcode and/ortext and/or figure and/or pattem and/or reflection spectrums. The sensor 112 may alsoprovide information relating to Weight, movement, different triggers, presence,sWitching, acceleration, vibration, temperature and/or the picking position in the articlecontainer.

The sensor arrangement 110 may preferably comprise different types ofsensors 112. The sensors 112 included in the sensor arrangement 110 may for examplebe any of: a camera, a spectroscopy sensor, a RFID sensor, a contour sensor, a Weight sensor (i.e. a scale), a symbol or text recognizing sensor, a color texture sensor, a color histogram sensor, microwave sensor, OCR, 3D-sensor or camera, time of flight sensor,presence sensor, switch sensor, accelerometer, movement sensor, temperature sensorand/or an object sensor and/or microwave sensor and/or a light curtain, an IR camera,and/or LIDAR sensor. The contour sensor, the symbol or text recognizing sensor, thecolor texture sensor, the color histogram sensor, and the object sensor may all beimplemented by one or several cameras which are connected to the processor 120,which processor l20 is configured to execute a specific image processing algorithm forextracting the required information from the captured images.

In one embodiment at least one sensor ll2 in the sensor arrangement ll0 is acamera. The camera may be a 2D and/or 3D camera. The camera may have associatedimage processing software, or the required processing software may be provided by theprocessor l20.

The at least one sensor ll2 will during operation detect a sensor signal, such asan measured signature, from an article, whereby the processor l20 is configured tomatch the measured signature with a database of reference signatures. Each referencesignature is associated with a unique article identity. Hence, different articles willprovide a unique signature whereby an accurate identification may be achieved. Thesensor must thus be capable of detecting article signatures for a wide variety of articleshaving different dimensions. For example, such articles may have small dimensions, aswell as large dimensions. The memory unit l22 of the processor l20 may comprises oneor more digital reference signatures created by the at least one sensor ll2, wherein eachone of said reference signature is associated with a corresponding article identity. Theidentity of the article may be deterrnined by matching a measured signature to adatabase comprising properties of a plurality of articles. The database may be stored inthe memory unit l22 and examples of properties are weight, size, color, shape, contour,pattem, barcode pattems and/or text and/or figure and/or reflection spectrum. In oneembodiment the sensor arrangement ll0 comprises at least one sensor in the form of acamera. The camera detects an image from an article, whereby the processor l20 isconfigured to match the detected image, or a processed digital representation of theimage, with a database of reference images, whereby each reference image is associated with a unique article identity. Hence, different articles will provide a unique digital representation of the image Whereby an accurate identification may be achieved. Thememory unit 122 of the processor 120 may comprises one or more reference images (orreference processed digital representations of the images) created by the at least onecamera, Wherein each one of said reference images/processed digital representations isassociated With a corresponding article identity. The identity of the article may bedeterrnined by matching a deterrnined image or a processed digital representation of theimage to a database comprising properties of a plurality of articles. The database may bestored in the memory unit 122 and examples of properties are size, color, shape,contour, barcode pattems and/or text and/or figure.

A number of different sensors 1 12 may be used to provide correctidentification of an article. For example, a shelf A is filled With one hundred differentarticle types. Each type of article is represented by a certain number of individualarticles, Which means that the shelf A stores several hundred, or even thousands ofindividual articles. There is thus a need for a reliable identification process.

The shelf A may e.g. be equipped With one or more scales forrning part of thesensor arrangement 110. When a customer 1 picks an article from the shelf A the totalWeight of the shelf A Will be reduced, and the information relating to the change inWeight Will be transmitted to the processor 120. If the Weight reduction is deterrnined tobe 250g, the processor 120 Will determine that the article being picked must belong toone of those article identities not Weighing more than 25 0g. Preferably, the sensorarrangement 110 may comprise a plurality of scales being positioned at a common shelfA. For example, if a shelf A stores ten different article types, ten individual scales maybe provided at the shelf A. Each scale may then be configured to measure the Weight ofa separate part of the shelf A such that the article type may be identified by detecting achange in Weight of the part of the shelf A storing the particular article type. Yet further,the data measured by the scales may further be used to identify the number of articlesbeing picked from the shelf A.

The sensor arrangement may also include one or more LIDAR sensors, Whichmay be attached to a shelf or in the ceiling of the article containing area. The LIDARsensor is preferably mounted so that it creates a vertical plane in front of an article container, e. g. a shelf A. When a customer or attendant reaches for an article in the shelf A the LIDAR sensor will detect an exact x-y coordinate of where the plane wasobstructed. The system may be configured to have information of every unique article°sexact position in the shelf A, thus by using the coordinate information from one or moreLIDAR sensor the article identity can be accurately deterrnined. The information fromthe LIDAR sensors may also be used as leaming data to the processor 120 when articlesare put into the shelf, e. g. as performed when attendants are unpacking new articles fordisplay.

The sensor arrangement 130 may also include one or more IR cameras whichmay be attached to a shelf or in the ceiling of the article containing area A-I. IR camerasmay be specif1cally advantageous for identifying cold and/or hot articles.

As is evident from the foregoing description, the above example onlyrepresents some possible uses of different sensors 112. The above-mentioneddescription could thus also be implemented using another type of sensor, such as a 3D- camera, a spectrometer, etc, The communication interface The sensor arrangement 110, and thus the individual sensors 112,communicates with the processor 120 by the use of a communication interface. Thecommunication interface is adapted to allow the sensor arrangement 110 tocommunicate with the processor 120, through the use of different communicationtechnologies. Such communication technologies may be Wired or wireless. Examples ofsuch Wired technologies are Universal Serial Bus (USB) and Ethemet to name a few. Itshould be noted that other technologies exist and are taken to be an obvious equivalentfor such wired communication interfaces. Examples of such wireless technologies areIEEE 802.11, IEEE 802.15, ZigBee, WirelessHART, WIFI, Bluetooth®, W-CDMA/HSPA, GSM, UTRAN and LTE to name a few. It should be noted that othertechnologies exist and are taken to be an obvious equivalent for such wireless communication interfaces.

The processor The processor 120, or a plurality of processors 120 arranged in communicationWith each other, is connected to the sensor arrangement 110. The at least one processor120 is configured to deterrnine the identity of an article picked by the customer 1. Theprocessor 120 is responsible for the overall operation of the sensor arrangement 120 andthe registration account 130 and is preferably implemented by any commerciallyavailable CPU ("Central Processing Unit"), DSP ("Digital Signal Processor") or anyother electronic programmable logic device. The processor 120 is configured to readinstructions from a memory 122 and to execute these instructions to control theoperation of the sensor arrangement 110 and/or the registration account 130. Thememory 122 may be implemented using any commonly known technology forcomputer-readable memories such as ROM, RAM, SRAM, DRAM, CMOS, FLASH,DDR, SDRAM or some other memory technology. The memory 122 is used for Variouspurposes by the processor 120, such as for storing application data and programinstructions, as Well as storing reference data, such as reference signatures, for thearticles of the store 10.

The processor 120 is further connected to the communication interface in orderto communicate With the sensor arrangement 120, the registration account 130 and/or anextemal electronic device (such as a mobile phone, tablet or any other computingdevice). Such communication technologies may be Wired or Wireless, as described in relation to the communication interface arranged in the sensor arrangement 110.

The registration account In some embodiments the registration account 130 is created and connected tothe customer in a direct Way, i.e. by tracking a customer inside a store, and in someembodiments the registration account 130 is created and connected to an accompanyingunit such as an article carrying device 5. The respective embodiments Will now bedescribed together With preferred methods of adding identified articles to the registration account.

Article carrying device - registration account 1st embodiment In some embodiments the registration account 130 is created and connected toan accompanying unit such as an article carrying device. The article carrying device 5may for example be a shopping cart, basket or a bag.

The registration account 130 keeps a list of the articles that has been collectedin the store 10. The registration account 130 is in communication with the processor 120and during checkout the list of articles is automatically transferred from the registrationaccount 130 to an associated payment system. The registration account 130 comprises alink to a article carrying device 5, represented by means of a unique ID or similar, and adynamic article list of the items being picked from the article containing areas A-I.

As soon as the person pulls out the article carrying device 5 form its idlelocation, the article carrying device 5 will be associated with a unique registrationaccount 130. Altematively, the customer may e.g. press an activation button arranged onthe article carrying device 5 or at another position at the entrance area for allowing thearticle carrying device 5 to be associated with the registration account 130. Associationof the article carrying device 5 and the registration account 130 is required in order tomake use of the registration system 100. The registration account 130 may either bemanaged using the article carrying device 5 itself or by connecting the article carryingdevice 5 to a mobile electronic device, as will be described further below.

In one embodiment, the article carrying device 5 in the form of a shopping cart5 is used to manage the registration account 130. The shopping cart 5 thus needs to havenecessary hardware for allowing the customer to interact with the registration account130. In one embodiment, the article carrying device 5 may comprise an interactivedisplay and equipment for allowing for manual registration of articles if needed. Sucharticles may e.g. by bulk articles or similar. The communication interface of the articlecarrying device 5 may also allow the customer to review and/or mange the registrationaccount 130. It is beneficial if the in-store registration system 100 can gain informationof the position of the article carrying device 5. This may be done in several ways. In oneembodiment the article carrying device 5 comprises a unique identification tag. Thisidentification tag is read by the sensor arrangement 110, wherein the positions of theindividual sensors are known, in order to determine a current position of the article carrying device 5. The identification tag may for example be a QR-code or another ll unique pattern Which can be used for the sensor arrangement 110 to identify the articlecarrying device 5. In an alternative embodiment the article carrying device 5 maycomprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE),short-range radio frequency positioning and/or Wifi-based positioning systems.

The use of a smart article carrying device 5 alloWs the registration account 130to be associated With a current position in the store Without tracking the individualcustomer itself. It can be assumed that the customer picks an article in an articlecontaining area A-I Which is Within a specified distance from the article carrying device5. Such presumption reduces the number of possible articles to choose from Whenidentifying a fetched article. This information is used in the identification process of thearticle, as Will be described more in detail With reference to Fig. 5.

Here the customer does not need any other physical device than the articlecarrying device 5 in order to properly use the registration system 100. Additionally,there is no need for the customer 1 to manually check in When entering the store andmultiple customers may use the same article carrying device 5 and thus the sameaccount 130 Without requiring adaptation of the registration account 130. Having theregistration account 130 associated With an article carrying device 5 has the benefit thatthe account is not directly associated With the customer and his/her identity.

The embodiment Where the registration account 130 is managed using thearticle carrying device 5 per se is preferably used When the probability of correct identification of an article is very high, so that manual registration is rarely needed.

Article carrying device - registration account. 2nd embodiment An embodiment in Which the registration account 130 is associated With aarticle carrying device 5 Will now be described. In some embodiments, the registrationaccount 130 is managed by connecting the article carrying device 5 to a separatephysical device such as an extemal electronic device. The extemal device may be amobile device, such as an mobile phone or other suitable devices for allowing thecustomer to interact With the registration account 130 In either case, the article carrying device 5 does not need to have any special hardware, display or position detection unit. 12 As soon as the person pulls out the article carrying device 5 form its idlelocation, the article carrying device 5 will be associated with a unique registrationaccount 130. Altematively, the customer may e. g. press an activation button arranged atthe entrance area for allowing the article carrying device 5 to be associated with theregistration account 130, or use the external device to make the association between theregistration account 130 and the article carrying device 5. Association of the articlecarrying device 5 and the registration account 130 is required in order to make use ofthe registration system 100.

Preferably, the article carrying device 5 comprises a unique identification tagwhich the sensors arrangement 120 reads in order to gain the position of the articlecarrying device 5. Since the positions of the individual sensors in the sensorarrangement are known, it is possible to determine the position of the article carryingdevice 5. The identification tag may for example be a QR-code or another uniquepattem which can be used for the sensor arrangement 110 to identify the article carryingdevice 5.

When an extemal device, such as a mobile phone or similar, is used formanaging the registration account 130 the article carrying device 5 may be providedwith a unique code arranged to be scanned by an application on the external device.Such a code may be in the form of a QR-code, EAN, UPC code or an RFID tag. Byscanning the code arranged on the article carrying device 5, the registration account 130can e. g. be managed using the application of the extemal device. This allows thecustomer to manually scan articles using the external device when the registrationsystem 100 does not manage to correctly identify an article, and to manually removeidentified articles which for some reason were decided not to be bought. The applicationof the extemal device must thus preferably be able to scan codes of an article. Theextemal device may also allow the customer to review and/or mange its registrationaccount 130. For this purpose the extemal device may e.g. be a dedicated portable codescanning unit.

The tags used by the customer when linking the article carrying device 5 to theregistration account 130, and the tag used for identifying the article carrying device 5 by means of the sensor arrangement 1 10 may not necessarily be the same. In some 13 embodiments, the position of the article carrying device 5 may be deterrnined using apositioning unit on the external device.

The embodiment Where the registration account 130 is managed using anexternal device connected to the article carrying device 5 is preferably used when theprobability of correct identification of an article is very high, so that manual registration is rarely needed.

Article carrying device - First identification step: Limitation The method of identifying an article 7 once it has been removed from an articlecontaining area A-I will now be described, although already being briefly discussed inthe foregoing. The following description is related to the case where the registrationaccount 130 is associated with an article carrying device 5, with or without the use of anexternal device. As been mentioned above, the sensor arrangement 110 monitors allarticle containing areas A-I arranged in the store. By continuously monitoring an articlecontaining area A-I, it is possible to recognize when a change in the article containingarea A-I has occurred. This may be done by comparing data streams generated by thesensor arrangement 110 and by comparing the data frame by frame.

A change in article containing area A-I may indicate that a customer 1 hasselected and removed an article 7 from the containing area A-I. This article 7 needs tobe identified in order to be added to the registration account 130 associated with thearticle carrying device 5. In one embodiment, the first identification step is performedonce the system 100 recognizes a change in the article containing area A-I and a secondidentification step is performed once the customer puts an article into the article carrying device 5.

Article carrying device - First identification step: Limitation + identification The first identification step comprises two main steps; limitation of possiblearticle identities and identification of a unique article, which steps will now be describedmore in detail. These two steps may be performed in sequence or simultaneously. Thelimitation process is based on the fact that the article containing area A-I has a known position in the store and a known range of articles and the limitation process uses this 14 information in order to at least partly identify the article. If an article is removed fromone article containing area A, it is a very high probability that said article is one of thearticles Which usually is accommodated in that article containing area A. This limitationprocess based on position decreases the number of possible articles.

Once the number of possible articles is reduced, the system 100 needs tonarrow down the possible articles to only one unique article. This identification may beachieved using different identification techniques. One such technique may be anidentification algorithm, Which uses information gathered from the sensor arrangement110 as an input. The sensor arrangement 110 comprises at least one sensor 112 Which isconfigured to determine a measured signature of the article being taken from the articlecontainer. The sensors 112 may be sWitched on, i.e. activated, in sequences to be able tofind beneficial combinations or a partial set of or all the sensors may be active until oneof the combinations provides a positive decision, Wherein one or more of the redundantsensors may be disconnected. A positive decision is here When all sensors in thecombination have detected and identified a predeterrnined property of an article, Wherethe properties in combination together give the article an identity. The identity may bedeterrnined through control against a database comprising properties, or signatures, ofan amount of articles. The database may be stored in the memory unit according to Whathas previously been described. Examples of properties are size, color, shape, contour,marking With barcode and/or text and/or figure and/or pattem. In order to provide asuccessful classification, an activated sensor deterrnines a measured signature of anarticle. The measured signature is associated With the sensed signal, and may thus be adigital representation of a number of different article properties. The processor 120 isfor this purpose configured to compare the measured signature With the digital referencesignatures stored in the memory unit, and to calculate a matching probability of apredeterrnined number of article identities. The latter step is preferably performed bycomparing the measured signature With all, or a subset of, the digital referencesignatures of the memory unit and subsequently delivering the article identities havingthe highest matching probabilities to a further classification algorithm, such as a BBN or a neural network.

In case of using two or more different sensors thus providing two or moredifferent measured signatures, the processor 120 is configured to independentlycompare each sensors measured signature with all, or a subset of, the digital referencesignatures of every sensor. The identified articles and their matching probabilities fromtwo or more sensors are transmitted to the network for further analysis and to combinethe highest matching probabilities of each sensor into one combined matchingprobability.

The articles may also be identified using a pattem recognition algorithm basedon the data collected by the sensor arrangement 110. In one embodiment the pattemrecognition algorithm is based on SVM, which preferably uses the measured signaturesand the digital reference signatures mentioned above as input to the support vectormachine model. In an altemative embodiment of the algorithm the system iscontinuously leamed by using the raw data from the sensor arrangement 110. In thissituation, the algorithm preferably uses deep leaming methods, such as Convolutionalneural network or Fully Connected neural network.

The different identification algorithms discussed above may be used separatelyor as subsequent steps in a combined identification routine. In the latter method theoutcome of each step is combined in order to produce a unique identified article withthe highest matching probability.

The classification method may further include a step of comparing the highestmatching probability with an alarm threshold and, in case the highest matchingprobability is below the alarm threshold, the system awaits manual input from thecustomer in the store before proceeding. In one embodiment the identification includesthe step of comparing the highest matching probability with two alarm thresholdwherein, in case the highest matching probability is above the lowest alarm thresholdbut below the upper alarm threshold, the method awaits manual input from the customerin the store, and in case the highest matching probability is below the lowest alarmthreshold, the system flags the registration account for additional security check at thecheckout station. In an altemative embodiment, the attendant is not needed and the customer input is sufficient. 16 For the embodiments described above the picking position is used to limit thetotal number of possible articles, and the network is used to further limit the number ofpossible articles to a single, positively identified article. However, it may also bepossible to deterrnine the picking position accurately so that a single article is positivelyidentified using only position data. As explained above, LIDAR sensors may be used toaccomplish this. In these situations, it is not necessary to make use of the network forfurther identification of the article.

For all cases described above the first identification step may result in a single, positively identified article, or a list of possible articles.

Article carrying device - Second identification step The information that an article, identified or unidentified, has been taken fromits position in the article containing area A-I may be used in a second identificationprocess either to verify the identification from the first identification step or to provide acomplete identification of an article which was only partly identified in the firstidentification step. The second identification step comprises two main steps; limitationof possible article identities and identification or verification of a unique article. Thesetwo steps may be performed in sequence or simultaneously.

The second identification process may be initiated when something is movedinside the boundaries of the article carrying device 5 (seen from above when somethingis moving within the railing rectangle of the article carrying device 5). Although thefirst identification process may have succeeded in identifying the article when beingremoved from the article containing area, the system must also deterrnine to whichregistration account 130 said article belongs to or if this has already been deterrnined toverify that the article and the registration account 130 belongs to each other.

The limitation process of the second identification step will now be described..Fig. 5 illustrates the relationship between an article in an article containing area A-Dand the time at which the customer removes the article from the article containing areaA-D. Article containing area A comprises articles named a, article containing area Bcomprises articles named b, article containing area C comprises articles named c and article containing area D comprises articles named d. In this example, the article 17 containing areas A-D are arranged Within the predeterrnined distance from the articlecarrying device 5. All articles arranged in the article containing areas A-D are thuspossible candidates When limiting the number of possible articles. In order to furthernarroW down the possible articles during the limitation process the time at Which thearticle Was taken from the article container is used.

All articles that are taken from the shelves A-D Will be saved in a temporarylist, and Will be saved thereon for a predeterrnined period of time t1 before they areremoved from the list. The memory 122 saves information of articles being taken fromeach article containing area A-D for a predefined period of time t1. The time periodalloWs the registration system 100 to have a limited amount of articles to choose from.Hence, the shorter period of time the less number of possible articles to choose from inthe identification process. It may take a While from the time When the customer takesthe article from the article container until he/she puts the article into the article carryingdevice 5. The customer may for example grab a couple of different articles beforeputting them all into the article carrying device 5. The time t1 needs thus be sufficientlylong, so that the article still is among the possible articles during identification even ifthe customer has been slow in putting the article into the article carrying device 5.

If the article Was not identified When picked from the article container, allarticles that Were Within a predeterrnined distance from said article container Will beavailable in the temporary list.

The registration system 100 keeps track of the position of the article carryingdevice 5, as previously described. This position information is used to limit the possiblearticle candidates as described With reference to Fig. 5. The possible number of articlesmay be limited by analyzing the location of the article carrying device 5. Only articlesthat are in a predeterrnined distance from the article carrying device 5 Will be consideredto be candidates When the system 100 operates to identify the chosen article. Thus in thesecond identification step the registration system 100 Will use a modified version of thelist generated When an article is taken from the article containing area. The list ofpossible article candidates available during the second identification step Will onlycomprise information relating to articles that has been removed from article containers Within a predeterrnined distance from the article carrying device 5, and Within a 18 predeterrnined time. The number of possible articles Which corresponds to the articlethat has been placed into the article carrying device 5 is thus greatly reduced.

Hence, the registration system l00 uses the list to identify that the identifiedarticle corresponds to the right registration account 130. If there are several possiblecandidates the registration system l00 identifies the article once the article is placed intothe article carrying device 5 in the same or similar manner as the first identificationprocess. The second identification step may be performed When the article isapproaching the article carrying device 5, such that the article is moving across a virtualboundary set at a predeterrnined distance from the article carrying device 5. The virtualboundary may coincide With the physical dimensions of the article carrying device 5.

As With the first identification step, the identification may be achieved usingdifferent identification techniques. One such technique may be an identificationalgorithm, Which uses information gathered from the sensor arrangement ll0 as input.In the second identification step, the input to the algorithm is taken from sensor(s)arranged such that the sensor(s) oversees the article carrying device 5 in the store. Thecontent of the article carrying device 5 may be analyzed continuously so that it ispossible to detect once an article is placed into, or close to the article carrying device 5as explained above. The at least one sensor ll2 collects data from the shopping cartbefore and after the customer approaches it and it is thus possible to detect When anarticle has been added or removed from the article carrying device 5. The sensorarrangement ll0 may use different identification algorithms in order to identify thearticles being arranged in the shopping basket.

If the registration system l00 failed to completely identify the article onceremoved from the article container, the system l00 gets another chance of identifyingthe article by analyzing the article carrying device 5 as has been described above. If thearticle Was not identified When picked from the article container, all articles that WereWithin a predeterrnined distance from said article container Will be available in the list.

As soon as the article has been detected and identified, the registration systeml00 may determine Whether the article Was really picked up by the customer or if it Wasput back in the article containing area A-I. This may be done by analyzing sensor data from the article containing area before and after the customer approached it. It is also 19 possible to use sensor data for searching article specific patterns inside the customersarticle carrying device 5. In one embodiment, in case a customer returns an article thatis already registered to its account, the registration system 100 recognizes that theidentified article that was retumed and removes the article from the registration account1 3 0.Once the registration system 100 has identified the article, and possibly verified that it was actually collected by the customer or added to the article carrying device 5, the article identity is added to the associated registration account 130.

Article carrving device - Adding Weight priced articles to a registration account In Fig. 4b a method for adding weight priced articles to a registration accountassociated with a article carrying device 5 is schematically shown. The followingdescription is related to the case where the registration account 130 is associated with anarticle carrying device 5, with or without the use of an extemal device.

In Fig. 4b some of the method steps are identical to the method steps of Fig. 4a.Therefore these are assigned the same reference numerals.

In a first step 202 the sensor arrangement 110 monitors all article containers A-I arranged in the store. By continuously monitoring an article containing area, it ispossible in step 204 to recognize when a change in any of the article containing areashas occurred. Such a change may indicate that a customer has selected and picked anarticle from the article containing area.

Once the registration system 100 recognizes that an article has been picked bya customer, the registration system identif1es the article in step 206. The article maypreferably be identified directly as it is removed from the article container A-I, althougha 100% proof identification is not required. Instead, this identification step may result ina list of possible articles.

In step 408 the system detects that an article is being placed on an articleweighing device located in the store. The article weighing device is preferably forrningpart of the sensor arrangement 110. This event triggers the following steps. The positionof the article weighing device is deterrnined and following this, in step 410 the list of previously identified articles is fetched, and the number of possible articles is reduced by a factor corresponding to the position of the article weighing device. For this, it ispossible that all articles being picked by customers are added to the same list of possiblearticles. This means that as soon as an article is picked from the article containing area,it is added to the list (optionally together with possible altematives). The articles remainon the list until an article is added to a registration account or it is added to the scale forprice calculation by weighing, at which point the article is also removed from the list.Any additional articles which were linked to the positively identified article are alsoremoved from the list.

However, in step 412 the number of articles on the list is reduced in order todetermine the correct article. This reduction is however not made permanent to the list,but only used as a selection for the final deterrnination step 414. The reduction ofnumber of articles is made based on the position of the article weighing device, suchthat only the articles being present on the list and norrnally belonging to a defined andlimited area surrounding the position of the article weighing device is used for the finalidentification before weighing. In step 414, the correct article is identified based on thereduced number of articles from the list. This final identification before weighing maye. g. be made by monitoring the article weighing device, and determine, by means of thesensor arrangement 110, the correct identity of the article. This step is greatly simplifiedby only considering the reduced number of possible articles. If the identification in step414 can°t determine a unique article, then the customer selects the correct article on thearticle weighing device screen. The possible articles to select from can either be thereduced articles from step 412 or all weight priced articles in the store. In step 417 is thearticle weighed by the article weighing device and the price is then calculated. In step418 is the article again added to the list of identified articles together with the calculatedprice and measured weight.

In step 208 the system detects that an article is being placed into an articlecariying device 5. This event triggers the following steps. The position of the articlecarrying device 5 is deterrnined and following this, in step 210 the list of previouslyidentified articles is fetched including the weight priced article that was added in step418, and the number of possible articles is reduced by a factor corresponding to the position of the article carrying device 5. For this, it is possible that all articles being 21 picked by customers are added to the same list of possible articles. This means that assoon as an article is picked from the article containing area, it is added to the list(optionally together with possible altematives). The articles remain on the list until anarticle is added to a registration account, at which point the article is also removed fromthe list. There is also a possibility, in one embodiment, to track the customers betweenstep 418 and step 208, to simplify the identification when placing the weight pricedarticle into the article carrying device 5.

However, in step 212 the number of articles on the list is reduced in order todetermine the correct article. This reduction is however not made permanent to the list,but only used as a selection for the final deterrnination step 214. The reduction ofnumber of articles is made based on the position of the article carrying device 5, suchthat only the articles being present on the list and norrnally belonging to a defined andlimited area surrounding the position of the article carrying device 5 is used for the finalidentification. In step 214, the correct article is identified based on the reduced numberof articles from the list. This final identification may e.g. be made by monitoring thearticle carrying device 5, and determine, by means of the sensor arrangement 110, thecorrect identity of the article. This step is greatly simplified by only considering thereduced number of possible articles. In a final step 216 the article is added to theregistration account 130 and the customer is preferably notified that the article has beenadded to the list of his registration account 130.

If the registration system 100 fails to identify the article correctly, e.g. if amatching probability computed by the registration system 100 is not above a predefinedthreshold, in step 212 the system 100 notifies the customer to manually identify thearticle. Once the customer has manually added the article, the article identity is added tothe registration account 130.

The user interface e.g. provided by the application of the extemal device, or bya display arranged on the article carrying device 5, may also allow customers tomanually add articles to the registration account without first trying automaticidentification. This may e.g. be particularly suitable for bulk goods, whereby theregistration system 100 may automatically detect that the customer (i.e. the article carrying device 5) is arranged in an area of the store where bulk goods is present. By 22 such detection, the registration system 100 may prompt the customer to manually input picked articles, whereby these articles are stored in the registration account 130.

Customer - registration account In the previous the description has been focused on the embodiments where theregistration account 130 is associated with an article carrying device 5. An embodimentwhere the customer is directly associated with a registration account 130 will now bedescribed. The connection between the registration account 130 and the customer 1 maybe done in several ways. In one embodiment the customer 1 is tracked inside the storeusing image recognition software, such as person tracking system, 3D image trackingfrom area cameras or other person-tracking algorithms known per se. The sensorarrangement 110 may e. g. be used for providing input data to such positioning. Theregistration system 100 will therefore know the position of the customer 1 by followingthe customer as he or she moves around the store 10. Hence, no further positiondetecting unit may be needed. However, in order to be able to manually register articlesand to perform a check-in operation the customer may use some extemal deviceassociated with the registration account 130.

In one embodiment the customer is connected to a registration account 130 bythe use of an extemal device such as a mobile application on the smart phone or tabletof the customer 1 or any other electronic device carried by the customer.

Once the customer 1 enters the entrance area 12 of the store 10 he or she isassociated with a registration account 130. This may be done e. g. by using a fixed unitarranged in the entrance area or on an extemal device of the customer. In someembodiments, the registration system 100 further identif1es the appearance of thecustomer 1 in the entrance area, which allows the system 100 to track the customerthroughout the store.

The registration system 100 needs to determine the position of the customer.This may either be done by the extemal electronic device of the customer, if itcomprises a position detecting unit, or the position of the customer may be deterrnined by direct tracking the customer as the customer walks through the store. 23 If multiple customers belongs together (such as a family) and arrive to the storeat different time it is possible to connect them to the same registration account 130 bymanually merging their accounts. Preferably, the merge operation is made by thecustomers themselves Without any interaction With store attendants.

If multiple customers belonging together arrive to the store at the same time itis desirable if the registration system adds these multiple persons to the sameregistration account 130. Preferably, automatic recognition of multiple personsshopping together is possible in the check-in area 12. These customers Will thus beconnected to same registration account 130 during their shopping session. It is alsopossible to merge accounts inside the store (i.e. in the article container area) by use of the extemal device.

Customer - Adding articles to a registration account A method of adding an article to a registration account 130 When theregistration account is associated With the customer Will now be described Withreference to Fig. 6. In a first step 302 the sensor arrangement 110 monitors all articlecontainers A-I arranged in the store. By continuously monitoring an article containingarea, it is possible in step 304 to recognize When a change in the article containing areahas occurred. Such a change may indicate that a customer has selected and fetched anarticle from the containing area.

Once the registration system 100 recognizes that an article has been picked bya customer, the registration system identifies the article in step 306. The article may beidentified directly as it is removed from the article container A-I using the sensorarrangement 1 10.

If the article Was identified, in step 308, as belonging to the customer pickingthe article from the article container A-I, the article identity is in step 310 added to theregistration account 130 associated With that customer. The customer is preferablynotified that the article has been added to the list of his registration account 130.

The user interface e.g. provided by the application of the extemal device mayalso allow customers to manually add articles to the registration account Without first trying automatic identification. This may e. g. be particularly suitable for bulk goods, 24 Whereby the registration system 100 may automatically detect that the customer (orassociated article carrying device 5) is arranged in an area of the store Where bulk goodsis present. By such detection, the registration system 100 may prompt the customer tomanually input picked articles, Whereby these articles are stored in the registration account.

Customer - First and second identification step The description relating to the steps of the first identification process for theembodiment Where the registration account is associated With the shopping cart areapplicable also to the embodiment Where the registration account is associated to thecustomer. However, the number of possible articles may further be limited by analyzingthe location of the customer 1. Only articles that are in a predeterrnined distance fromthe customer 1 Will be considered to be candidates When the system 100 operates toidentify the chosen article 7. That distance may be chosen differently depending on thesize of the store and the accuracy in deterrnining position of the customer.

In the embodiment Where the registration account is associated to the customer thesecond identification step is someWhat different compared to the situation Where theaccount is associated to an article carrying device 5. If the first identification step isinsufficient to identify the article due to a matching probability computed by theregistration system 100 being below a predeterrnined threshold, different options arepossible.

According to a first option, in step 312 (see Fig. 6) the system 100 notifies thecustomer to manually identify the article by scanning a code associated With the article.Once the customer has manually identified the article, the article identity is added to theregistration account.

In a second option, the system 100 may in step 312 alert and instruct thecustomer to manually choose the correct article among a plurality of possible articles.The articles the customer is able to choose from are either the articles that are saved inthe reduced article list or a list of articles having high matching probability. The reducedarticle list is reduced based on the position of the customer, such that only the articles being present on the list and norrnally belonging to a defined and limited area surrounding the position of the customer is used for the final identification. Thecustomer chooses the correct article identity by using the external device, such as amobile phone or a tablet. Once the customer has manually identified the article, thearticle identity is added to the registration account.

In a third option, the customer is not prompted immediately but insteadinformation relating to any unidentified articles is stored in the registration account 130,and the customer Will be notified of such unidentified articles during checkout andrequested to manually identify those articles. For manual identification, terrninals may be provided inside the store for allowing the customer to manually identify articles.

The user interface The interaction between the customer and the registration system 100 ispreferably done by the use of a display unit. In one embodiment, the display unit maybe in communication With the processor 120 for displaying information regarding theregistration account 130. The display unit is preferably arranged for use by a customerfor inputting information or for approving the displayed information. The display unitmay be arranged at an article carrying device 5 or as an extemal device, such as amobile phone of the customer. The display unit may be an interactive display or apassive display having buttons arranged in connection thereto so as to be able to interactWith the display. The display may be a LCD-screen, a LED-screen or any other suitablescreen.

The information provided in the display unit may for example be informationrelating to the list of articles that already has been identified as picked by the customer,the total price of the selected articles, offers or deals for different articles in the storeand/or accessible information relating to registration customer account such as previouspurchase history. For this the customer may have the possibility to link the currentregistration account 130 With his customer account for the specific store.

The information provided in the display unit may also relate to identification ofan article. If the sensor arrangement 110 identifies the article, an image or a text may beshown in the display unit and if the user finds the displayed information matching the article Which the customer has chosen the customer may approve the identity by manual 26 input. Further information may be displayed, e. g. Weight and price, Wherein the usermay approve the displayed information if it is correct. The display unit may furthernotify the customer When an article needs manual registration. This may be the case ifthe automatic identification of the chosen article fails. The display unit may thus promptthe customer to manually register the article, for example by using a barcode reader orby scanning or identifying the article in some other Way.

The display unit may also be used by the customer if the registration systeml00 has identified an article Which the customer does not Want to buy or When thesystem l00 has identified the article Wrongly. The customer may then manually remove the article from the article list of the registration account.

Incorrect article positioning The system l00 may also be used to ensure the correct position of the articles,for example, if a customer returns an article at an incorrect position. By continuouslymonitoring the article containing area A-I by the sensor arrangement ll0, the automaticin-store registration system l00 may recognize When a customer adds an article back tothe article containing area A-I. The system l00 is thus preferably arranged todifferentiate between When an article is added or removed from the article containingarea A-I. The added article is identified by the registration system l00 and it isdeterrnined if the identified article is an article belonging to that position in that articlecontaining area A-I, i.e. in the article containing area A-I that the automatic in-storeregistration system l00 is trained for.

The article may be added to a different article containing area A-I, if forexample the customer picks out an article from the article containing area A and puts itback to another article containing area D, this may for example occur if the customerrealizes that the article chosen Was in fact not something he/ she intended to buy andreturns it into a random article containing area D.

If the system l00 deterrnines that the added article Was placed in the Wrong,position in the correct article container A-I or if the article Was placed in the Wrongarticle container A-I, the system l00 Will transmit an alarm signal. Preferably, the alarm signal is associated With the position of the article containing area in question Whereby 27 it is possible for an store attendant to correct the position of the Wrongly arrangedarticle. The alarm signal may also be transmitted to the processor for storing and evaluating the data.

Payment Once all the articles of the customer are correctly identified by the registrationsystem 100, and When the customer enters the checkout area 16, the processor 120transmits a transaction signal to the checkout station or the checkout station can retrievethe account to allow the customer to finish the payment transaction. Once the customerenters the checkout area 16 the information stored in the registration account 130 isautomatically transmitted to the checkout station or the checkout station retrieves theaccount. The checkout station then transfers the registered articles to a receipt forpayment.

In one embodiment, the checkout station transmits a control signal to asurveillance system Which Will be used in the deterrnination Whether or not to open apassage gate. A person is in such embodiments only allowed to exit through the gate(s)18 of the store 10 if the person 1 has paid for its article(s). This is deterrnined based oninformation from the registration system 100 and information regarding the payment.The system may initiate security check based on abnormal activity of a certain registration account.

Optional Weight sensor at exit In one embodiment, Where the registration account is associated With an articlecarrying device 5, such as a shopping cart , a bag, basket or the like, a Weight sensormay be arranged in the exit area 16 of the store to act as a safeguard to the registrationsystem 100. The Weight sensor may either be provided before or after the paymentprocess. The total Weight of the article carrying device 5, containing all picked articles,are compared to the estimated Weight based on the Weight of an empty article carryingdevice 5 and the total Weight of the identified articles or it is compared to a trainedWeight Which is saved in the processor 120 for all possible article carrying devices such as shopping cart, bag, basket or the like. If all articles inside the article cariying device 5 28 are identified by the registration system 100, the estimated Weight and the actual Weightshall be equal, or the difference shall be Within a small perrnissible range. If the actualWeight does not sufficiently correspond to the estimated Weight, such information istransmitted to the processor 120 and the customer and/or a store attendant is alerted.

If the article carrying device 5 has a Weight lower than the estimated Weight,the registration system 100 has either Wrongly identified an article or an identifiedarticle has been removed from the article carrying device 5 Without being correctlyremoved from the registration account 130.

If the article carrying device 5 has a Weight higher than the estimated Weight,the registration system 100 has either Wrongly identified an article or an unidentifiedarticle has been placed into the article carrying device 5 but not being added to theregistration account 130. If an article has been placed in the article carrying device 5 buthave not been added to the registration account 130, the customer may then perform manual registration of that article.

System leaming The automatic in-store registration system 100 may be subject to training, orleaming, in order to improve the accuracy of the identification of articles. For example,the sensor arrangement 110 may be activated during predeterrnined training sessions, inWhich a store attendant picks articles in a consecutive order. For each picked articletype, the attendant scans at least one article for a secure identification. The system 100Will thus not only leam to identify the article correctly, but also the exact position of thearticle. For deterrnining the position of the article it is preferred to know the position ofthe scanner, either by detecting the position by means of the sensor arrangement 110itself, or by associating the article position (i.e. the shelf or similar) With a unique codeWhich is scanned together With the article code. As a third option the scanner maycomprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE),short-range radio frequency positioning and/or Wifi-based positioning systems.

System leaming may further be improved by using checkout counters, either manually operated, semi-automatic operated, or fully automatically operated. 29 The training, or leaming, may also occur during norrnal operation of the store,Where the store attendant adds or ref1lls articles into the article containing area. For eachpicked article type, the attendant scans at least one article by a handheld device. Thesensor arrangement is thus trained With every single article Which the store attendantadds into the article container. Even if the scanner is used only for one or a few articlesof each article type, the sensor arrangement 110 can register signatures and position ofeach article for every article type. The system 100 Will thus leam to identify the articlecorrectly by using both the appearance of the article as Well as the exact position of thearticle.

It should be appreciated that even though numerous characteristics andadvantages of the present invention have been set forth in the foregoing description,together With details of the structure and function of the invention, the description isonly illustrative and changes may be made in detail, especially in matters of shape, sizeand arrangement of parts Within the scope of the invention to the full extent indicated by the appended claims.

Claims (7)

1. A method for training an in-store registration system (100) conf1gured toidentify and register articles in a store having at least one article containing area (A-I),said method comprising perforrning a training sequence in order to improve theaccuracy of the identification of articles by:allowing a store attendant to add or remove one or more articles to or from thearticle containing area (A-I),scanning them at least one added/rerrifßeefš article by a scanning device foridentifying the article,registering one or more signatures of each ysgf_rg_ft_g'_l_yt_i_gf_çi__article by means of a sensorarrangement (110), andadding or updating information relating to signature and the position of each article to the in-store registration system (100).

2. The method according to claim 1, Wherein the step of scanning ägg at leastone add-edfírfxrnoxfed. article by a scanning device further is for deterrnining the position of the scannreai. article.

3. The method according to claim 1 or 2, Wherein the sensor arrangement (110)comprises at least one sensor (112) being selected from the group comprising a 2D- camera, a 3D-camera, an IR camera, a scale, a LIDAR sensor, and a spectrometer.

4. The method according to any preceding claims, further comprisingdeterrnining the position of the scanned article by means of the sensor arrangement (1 10) itself.~

5. The method according to any preceding claims, further comprisingdeterrnining the position of the sarzinried. article by associating the article position With aunique code Which is scanned together With the article code, and/or by means of a position detecting unit provided at the scanning device.

6. The method according to any preceding claims, Wherein the steps ofscanning and registering one or more signatures is perforrned using checkout counters, either manually operated, semi-automatic operated, or fully automatically operated.

7. The method according to any -preceding claims, Wherein the scanning devicecomprises a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE), short-range radio frequency positioning and/or Wifi-based positioning systems.