WO2019245441A1

WO2019245441A1 - An automatic in-store registration system

Info

Publication number: WO2019245441A1
Application number: PCT/SE2019/050589
Authority: WO
Inventors: Martin ANGENFELT; Johan MÖLLER; Tobias PETTERSSON
Original assignee: Itab Scanflow Ab
Priority date: 2018-06-20
Filing date: 2019-06-19
Publication date: 2019-12-26
Also published as: EP3811315A1; SE1850757A1; SE543102C2; EP3811315A4

Abstract

A method for automatic registration of an article in a store having at least one article containing area (A-I) and an associated automatic in-store registration system (100) is provided. The method comprises associate an article carrying arrangement (5) with a specific registration account (130), wherein the registration account (130) is associated with a dynamic list of articles, continuously monitor the at least one article containing area (A-I) by means of a sensor arrangement (110), and detect a change in the at least one article containing area (A-I) and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list. The method further comprises continuously monitor the article carrying arrangement (5) by means of the sensor arrangement (110), receive a first image from the sensor arrangement (110) of at least a portion of a customer's upper extremity being captured when at least a part of the customer's upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement (5), determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list and add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account (130).

Description

AN AUTOMATIC IN-STORE REGISTRATION SYSTEM

TECHNICAL FIELD

The present invention relates to an automatic in-store registration system being configured to automatically registering articles in a store.

BACKGROUND

In today’s stores many different articles, such as food products, hygiene articles, clothing, etc. may be purchased which all have different sizes and shapes, as well as visual appearance. Normally, a checkout operator handles each article manually and makes sure that the article is associated with the correct pricing at checkout for correct payment by the customer. This is traditionally done either by manually inputting the price, by scanning a barcode attached to the article, or a combination of those. However, semi- or fully automatic checkout counters are becoming an interesting alternative for retail stores and supermarkets. Such checkout counters provide robust and easy identification and handling of articles and they normally include different components for identifying the article accurately.

However, in prior art systems the customer still needs to perform the critical step of feeding the articles through a checkout counter in order for the articles to be identified. The customer is required to queue in checkout lines and pass each product from the shopping cart or basket to a scanning system. Therefore, there is a need for an improved registration system that is fast, reliable, and robust and thus has an increased security level while at the same time being easy and convenient for the customer.

SUMMARY

An object of the present invention is therefore to provide an improved method and system for automatic registration of articles in a store.

In a first aspect, a method for automatic registration of an article in a store having at least one article containing area and an associated automatic in-store registration system is provided. The method comprises associate an article carrying arrangement with a specific registration account, wherein the registration account is associated with a dynamic list of articles, continuously monitor the at least one article containing area by means of a sensor arrangement, and detect a change in the at least one article containing area and, in response to said detected change, and identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list. The method further comprises continuously monitor the article carrying arrangement by means of the sensor arrangement, receive a first image from the sensor arrangement of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement, determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list, and add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account.

In one embodiment the method further comprises the step of receiving a second image from the sensor arrangement of at least a portion of a customer’s upper extremity being captured when the at least a part of the customer’s upper extremity is moving away from the virtual boundary set and determine, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article and if so determine if the second image does not include a representation of the article of the first image. If so, identify the article of the first image in a second identification step from the article identities currently included in the temporary article list, and add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account.

In one embodiment, if it is determined that the first image includes a representation of an article and that the second image include a representation of the article of the first image, the method further comprises the step of determine if the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image. And if the first and second images includes representation of different articles: identify the article of the first image and add the identity of the article to the dynamic list of articles of the registration account, and identify the article of the second image and remove the identity of the article from the dynamic list of articles of the registration account.

In one embodiment, if it is determined that the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account.

In one embodiment, if it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article, the method further comprises the step of identifying said article from the second image in a second identification step from the article identities currently included in the dynamic list of articles of the registration account, and removing the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account.

In one embodiment, the article carrying arrangement is either physical or virtual arrangement, or a part of the customer.

The article carrying arrangement may be a shopping cart, basket, a bag or the upper extremity of the customer. If the article carrying arrangement is an upper extremity, it may more specifically be the hands and/or arms of the customer.

The method may further comprises the step of receive a third image from the sensor arrangement of the virtual boundary set prior to at least a part of the customer’s upper extremity is moving across the virtual boundary set, receive a fourth image from the sensor arrangement of the virtual boundary set after to at least a part of the customer’s upper extremity is moving away from the virtual boundary set, determine, based at least in part on a comparison of the first, third and fourth image, if the article has been added to or removed from the article carrying arrangement, identify said article in a second identification step from the article identities currently included in the temporary article list, and wherein if is determined that the article has been added to the article carrying arrangement add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account and wherein if is determined that the article has been removed from the article carrying arrangement remove the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account.

In one embodiment, the first identification step is performed by identifying an article by at least using information relating to the position of the change in the article containing area, wherein the position of each one of the plurality of articles arranged in the article containing area is predetermined and accessible for the automatic in-store registration system.

The first identification step may be performed by reducing the total number of possible article identities using the position information of the change in the article containing area. The article containing area may be arranged with a plurality of segments, wherein the position of each segment is known. Each segment of the article containing area may comprise at least one capacitive sensor arranged to detect a change in the article containing area.

In one embodiment, the step of associating the article carrying arrangement with a specific registration account is performed using Near Field Communication.

In a second aspect, an automatic in-store registration system in a store having at least one article containing area is provided. The system comprises at least one processor, at least one registration account being associated with an article carrying arrangement, and a sensor arrangement comprising a plurality of sensors, the sensor arrangement being configured to, in a first identification step, continuously monitor the article containing area and to detect a change in the article containing area and, in response to said detected change, to transmit a first identification signal comprising information relating to one or more possible articles to the processor, and wherein the sensor arrangement is further configured to, in a second identification step, continuously monitor the article carrying arrangement and to detect a change in or near the article carrying arrangement and, in response to said detected change, to transmit a second identification signal comprising a first image to the processor, wherein the processor is configured to identify one or more possible articles from the first identification signal and to save said articles to a temporary article list, wherein the processor is further configured to identify one or more possible articles from the second identification signal by: receiving a first image of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement; determining, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list; and wherein the processor is further configured to determine a final article by comparing the one or more articles identified from the second identification signal with the articles stored in the dynamic article list, and to add said final article to the registration account.

In a third aspect, a method for training an in-store registration system configured to identify and register articles in a store having at least one article containing area is provided. The method comprising performing a training sequence in order to improve the accuracy of the identification of articles by using training data to recognize different movements of a user’s upper extremity while being in or near the article containing area and/or the article carrying arrangement, and to detect at least a portion of a upper extremity of a person when at least a part of the person’s upper extremity is moving in or near the article containing area and/or the article carrying arrangement, registering one or more signatures of each article by means of a sensor arrangement, and adding or updating information relating to signature and/or the position of each article to the in-store registration system.

In one embodiment, the training data is composed of a synthetically generated

3D-model of at least a part of the store and/or the customer, wherein the synthetic training synthetic data is a 3D-model of at least a part of the store. The synthetic data may be enhanced using Generative Adversarial Networks.

In a fourth aspect, a method for automatic registration of an article in a store having at least one article containing area and an associated automatic in-store registration system is provided. The method comprises associate an article carrying arrangement with a specific registration account, wherein the registration account is associated with a dynamic list of articles, continuously monitor the article carrying arrangement by means of the sensor arrangement, receive a first image from the sensor arrangement of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement, determine, based at least in part on the first image, if the first image includes a representation of an article, and if so identify said article, and add the identity of the identified article to the dynamic list of articles of the registration account.

In one embodiment, the method further comprises the step of receive a second image from the sensor arrangement of at least a portion of a customer’s upper extremity being captured when at least a part the customer’s upper extremity is moving away from the virtual boundary set, determine, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article and if so determine if the second image does not include a representation of the article of the first image, and if so, identify the article of the first image.

In one embodiment, wherein if it is determined that the first image includes a representation of an article and that the second image include a representation of the article of the first image, the method further comprises the step of determine if the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image, and if the first and second images includes representation of different articles: identify the article of the first image and add the identity of the article to the dynamic list of articles of the registration account; and identify the article of the second image and remove the identity of the article from the dynamic list of articles of the registration account.

In one embodiment, wherein if it is determined, based at least in part on a comparison of the first image and the second image, that the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account.

In one embodiment, if it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article, the method further comprises the step of identify said article at least in part based on the second image from the article identities currently included in the dynamic list of articles of the registration account, and remove the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account.

In a fifth aspect, an automatic in-store registration system in a store having at least one article containing area is provided. The system comprises at least one processor, at least one registration account being associated with an article carrying arrangement, and a sensor arrangement comprising a plurality of sensors, the sensor arrangement being configured to continuously monitor the article carrying arrangement. The processor is configured to receive, from the sensor arrangement, a first image of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement, determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article, and wherein the processor further is configured to add said final article to the registration account.

In a sixth aspect, a method for training an in-store registration system configured to identify and register articles in a store is provided. The method comprises performing a training sequence in order to improve the accuracy of the identification of articles by defining a virtual boundary set at a predetermined distance from the article carrying arrangement, using training data to recognize different movements of a user’s upper extremity while being in or near the virtual boundary set, and to detect at least a portion of a upper extremity of a person when at least a part of the person’s upper extremity is moving across the virtual boundary set, registering one or more signatures of each article by means of a sensor arrangement, and adding or updating information relating to signature and the position of each article to the in-store registration system.

In one embodiment, the training data is composed of a synthetically generated

In a seventh aspect, a method for registration of an article in a store having at least one article containing area and an associated automatic in-store registration system is provided. The method comprises to associate a customer with a registration account, monitor the movement of the customer through the store, continuously monitor the at least one article containing area by means of a sensor arrangement, and detect a change in the at least one article containing area and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list, wherein, if said temporary article list includes two or more articles, positively identifying the article which caused the change in a second identification step from the articles currently included in the temporary article list by receiving a first image from the sensor arrangement of at least a portion of a user’s upper extremity being captured after at least a part of the user’s upper extremity is removed from the article containing area, wherein the first image includes a representation of an article, and add the identity of the article which was positively identified in the first or second identification to the registration account.

In one embodiment, the method further comprises the step of receiving a second image from the sensor arrangement of at least a portion of a user’s upper extremity being captured when at least a part of the user’s upper extremity is inside the article containing area, wherein the first and second images are used to identifying the article in the second identification step.

In one embodiment, the second identification step further comprises accessing the article identities saved in the temporary article list and identifying the article as one of the accessed article identities originally positioned within a predetermined distance from the customer and added to the temporary article list within a predetermined time.

In one embodiment, the method further comprises identifying a plurality of customers, and associating two or more customers to the same registration account.

The step of associating the customer with the registration account may be performed using Near Field Communication.

In one embodiment, the first identification step is performed by reducing the total number of possible article identities using the position information of the change in the article containing area. The article containing area may be arranged with a plurality of segments, wherein the position of each segment is known. Each segment of the article containing area may comprise at least one capacitive sensor arranged to detect a change in the article containing area.

In an eight aspect, an automatic in-store registration system in a store having at least one article containing area is provided. The system comprises at least one processor, at least one registration account being associated with a customer in the associated store, and a sensor arrangement comprising a plurality of sensors, the sensor arrangement being configured to, in a first identification step, continuously monitor the article containing area and to detect a change in the article containing area and, in response to said detected change, to transmit a first identification signal comprising information relating to one or more possible articles to the processor, wherein the sensor arrangement further is configured to continuously monitor the customer. The processor is configured to identify one or more articles from said first identification signal and to save the identified one or more articles to a temporary article list, wherein if the temporary article list includes two or more articles, the processor is configured to positively identify the article which caused the change in a second identification step from the articles currently included in the temporary article list by receiving a first image from the sensor arrangement of at least a portion of a user’s upper extremity being captured after at least a part of the user’s upper extremity is removed from the article containing area, wherein the first image includes a representation of an article, and to add the article being positively identified in the first or second identification step to the registration account.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described in the following; reference being made to the appended drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

Fig. 1 schematically shows a top view of a store having a registration system according to an embodiment of the present invention;

Figs. 2a-b show isometric views of parts of a store having a registration system according to an embodiment of the present invention; Fig. 3 shows a schematic view of a registration system according to an embodiment of the present invention;

Fig. 4a shows a schematic method for automatic registration according to an embodiment of the present invention;

Fig. 4b shows a schematic method for automatic registration according to an embodiment of the present invention;

Fig. 4c shows a schematic method for automatic registration according to an embodiment of the present invention;

Fig. 4d shows a schematic method for automatic registration according to an embodiment of the present invention;

Fig. 5 shows a schematic view of the limitation process of a registration method according to an embodiment of the present invention; and

Fig. 6 shows a schematic method for automatic registration according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Fig. 1 shows an illustrative view of a designated area, such as a retail or grocery store 10. The store 10 comprises a plurality of article containing areas A-I, such as shelves, displaying articles available for purchase. The store 10 may also include an entrance or check-in area 12, an entrance gate 14, an exit or checkout area 16, an exit gate 18 and at least one checkout station 20. The entrance gate 14 and the exit gate 18 may be the same gate. Furthermore, the entrance area 12 and the exit area 16 may be the same or they may overlap.

A customer enters the store 10 by passing through the entrance gate 14 and into the entrance area 12, whereby the customer walks through the aisles between the article containing areas A-I and collects the articles to be purchased. The article will be, at least in part, identified by the automatic in-store registration system 100 described herein once the article is removed from its original position at the article containing areas A-I. Providing that the identification by the registration system 100 is successful, the customer does not need to scan the article, or provide any other manual input. Once the customer is finished fetching the articles to be purchased, the customer pays the amount corresponding to the chosen articles in the checkout station 20 and exits through the exit area 16 out of the exit gate 18. The total amount to be paid is automatically determined by the registration system 100. The automatic in-store registration system 100 may also use information relating to discounts, coupons, sales-items etc. in the determination of the total amount.

Fig. 2a shows an embodiment of the registration system 100 monitoring one article containing area A, here in the form of a shelf, comprising a plurality of different articles. The registration system 100 comprises a sensor arrangement 110 comprising a plurality of sensors 112, which at least monitors the shelf A per se and/or the articles arranged on the shelf A. In some embodiments, the sensors 112 also monitor a customer 1 and/or an article carrying arrangement 5 used by the customer 1, as will be described in further details below. Furthermore, the registration system 100 comprises a registration account 130, which together with the sensor arrangement 110 will be described more in detail with reference with Fig. 3.

Although the article containing area A shown in Fig. 2a is the form of a shelf, the article containing area may also take the form of a basket, tray, freezer, fridge and/or other similar arrangements which can contain and display articles to a customer. The articles may be of different dimensions, structure and colors as illustrated in Fig. 2a. Preferably, all articles being the same type and/or brand are arranged next to each other to facilitate selection and retrieving by the customer 1.

The article carrying arrangement, or device, 5 in Fig. 2a is in the form of a shopping cart, but it should be understood that the article carrying arrangement 5 also may be a basket, a bag or similar container which the customer 1 may use to collect and/or transports its chosen articles. The article carrying arrangement may thus be used to collect the by a customer chosen article. The article carrying device may be a movable device or arrangement. In another embodiment, the article carrying arrangement 5 may be a stationary, but pre-defmed area.

In yet one embodiment, the article carrying arrangement 5 could be the upper extremity of a user. More preferably, the article carrying arrangement is the hand and/or arm of the customer. The article carrying arrangement 5 could thus be a part of the customers upper body. This is for example the case when a customer is picking article(s) from an article containing area A and carries the articles in his/her arms. The customer may carry the article(s) in his arms during the whole shopping session, or for time period before he/she adds it to a second article carrying device (such as a shopping cart, basket, or a stationary pre-defmed area).

In Fig. 2a the customer 1 has fetched an article 7 from the shelf A and has the intention to put down the article into the article carrying device 5. This selected article 7 thus needs to be identified in order to be added to the registration account 130 so that the customer can pay for the article 7. For this purpose the sensors 112 monitor the article containing area A and all the articles arranged thereon, at least the articles arranged at the front. By continuously monitoring the article containing area A, it is possible to recognize when a change in the article containing A has occurred. Such a change may indicate that a customer 1 has picked an article 7 from the article containing area A. The registration system 100 is configured to have access to information relating to which types of articles being present on each article containing A-I and the registration system 100 is configured to use such position information together with analyzing data from the sensors 112 in order to identify the identity of the article 7 being selected and picked by the customer 1.

Fig. 2b illustrates an article containing area A being divided into a plurality of specified segments aa, ab, ac, ad. The segments provide a more refined position data of the articles arranged in the article containing area A. The segments may vary in size or all segments may be of the same size. The segments may seen as an area or a volume.

The segment may comprise one kind of article of one single brand, or one kind of article of different brands or each segment comprises a plurality of article types.

In one embodiment, all segments are arranged with at least one sensor. As will soon be described more in detail, the sensors may for example be a weight sensor, LIDAR-sensor, capacitive sensor and/or a camera.

The sensor arrangement

As explained above the sensor arrangement 110 is provided in order to automatically detect the articles 7 being picked by the customer 1 throughout the store 10. The sensor arrangement 110 thus forms part of the registration system 100, which also, as indicated in Fig. 3, comprises a processor 120, being connected to a memory 122, and a registration account 130 being in operative communication with the processor 120. The sensor arrangement 110 is configured to be in communication with the processor 120 as will be explained further below.

The sensor arrangement 110 comprises at least one sensor 112. However, as indicated by the dotted lines in Fig. 3, the sensor arrangement 110 may comprise a plurality of sensors 112. Preferably, the sensor arrangement 110 comprises a plurality of sensors 112 to create a full coverage of the articles 7 arranged in the store. Furthermore, a plurality of sensors 112 may be arranged to cover different views of the article, so that for example an upper, lower, left, right, front, and back side of the article 7 could be monitored by the sensors 112. The sensors 112 may be arranged on the article container itself, on the walls surrounding the article containers and/or on the ceiling of the store.

Additionally or alternatively, one or a plurality of sensors may be arranged on an article carrying device 5, such as on a shopping cart. If at least one sensor is arranged on the article carrying device 5, the sensor can be used to continuously generate data and thus generate a geometry of the store.

The sensor 112 may be configured to provide information corresponding to the dimensions and outer appearance of the article, as well as the position of where the article was picked in the article containing area A-I . The sensor 112 may further provide information relating to size, color, shape, contour, marking with barcode and/or text and/or figure and/or pattern and/or reflection spectrums. The sensor 112 may also provide information relating to weight, movement, different triggers, presence, switching, acceleration, vibration, temperature and/or the picking position in the article container.

The sensor arrangement 110 may preferably comprise different types of sensors 112. The sensors 112 included in the sensor arrangement 110 may for example be any of: a camera, a spectroscopy sensor, a RFID sensor, a contour sensor, a weight sensor (i.e. a scale), a symbol or text recognizing sensor, a color texture sensor, a color histogram sensor, microwave sensor, OCR, 3D-sensor or camera, time of flight sensor, presence sensor, switch sensor, accelerometer, movement sensor, temperature sensor and/or an object sensor and/or microwave sensor and/or a light curtain, an IR camera, and/or LIDAR sensor. The contour sensor, the symbol or text recognizing sensor, the color texture sensor, the color histogram sensor, and the object sensor may all be implemented by one or several cameras that are connected to the processor 120, which processor 120 is configured to execute a specific image-processing algorithm for extracting the required information from the captured images.

In one embodiment at least one sensor 112 in the sensor arrangement 110 is a camera. The camera may be a 2D and/or 3D camera. The camera may have associated image-processing software, or the required processing software may be provided by the processor 120.

The at least one sensor 112 will during operation detect a sensor signal, such as an measured signature, from an article, whereby the processor 120 is configured to match the measured signature with a database of reference signatures. Each reference signature is associated with a unique article identity. Hence, different articles will provide a unique signature whereby an accurate identification may be achieved. The sensor must thus be capable of detecting article signatures for a wide variety of articles having different dimensions. For example, such articles may have small dimensions, as well as large dimensions. The memory unit 122 of the processor 120 may comprises one or more digital reference signatures created by the at least one sensor 112, wherein each one of said reference signature is associated with a corresponding article identity. The identity of the article may be determined by matching a measured signature to a database comprising properties of a plurality of articles. The database may be stored in the memory unit 122 and examples of properties are weight, size, color, shape, contour, pattern, barcode patterns and/or text and/or figure and/or reflection spectrum. In one embodiment, the sensor arrangement 110 comprises at least one sensor in the form of a camera. The camera detects an image from an article, whereby the processor 120 is configured to match the detected image, or a processed digital representation of the image, with a database of reference images, whereby each reference image is associated with a unique article identity. Hence, different articles will provide a unique digital representation of the image whereby an accurate identification may be achieved. The memory unit 122 of the processor 120 may comprises one or more reference images (or reference processed digital representations of the images) created by the at least one camera, wherein each one of said reference images/processed digital representations is associated with a corresponding article identity. The identity of the article may be determined by matching a determined image or a processed digital representation of the image to a database comprising properties of a plurality of articles. The database may be stored in the memory unit 122 and examples of properties are size, color, shape, contour, barcode patterns and/or text and/or figure.

As will be described more in detail later on, the processor 120 may also be configured to receive images of article containing area, article carrying device and/or images of a customer’s upper extremity (such for example the hand and/or the arm)in different situations.

A number of different sensors 112 may be used to provide correct

identification of an article. For example, a shelf A is filled with one hundred different article types. Each type of article is represented by a certain number of individual articles, which means that the shelf A stores several hundred, or even thousands of individual articles. There is thus a need for a reliable identification process.

The shelf A may e.g. be equipped with one or more scales forming part of the sensor arrangement 110. When a customer 1 picks an article from the shelf A the total weight of the shelf A will be reduced, and the information relating to the change in weight will be transmitted to the processor 120. If the weight reduction is determined to be 250g, the processor 120 will determine that the article being picked must belong to one of those article identities not weighing more than 250g. Preferably, the sensor arrangement 110 may comprise a plurality of scales being positioned at a common shelf A. For example, if a shelf A stores ten different article types, ten individual scales may be provided at the shelf A. Each scale may then be configured to measure the weight of a separate part of the shelf A such that the article type may be identified by detecting a change in weight of the part of the shelf A storing the particular article type. Yet further, the data measured by the scales may further be used to identify the number of articles being picked from the shelf A.

In one embodiment the one or more scales are arranged to determine if an article has been picked from or being placed in the article containing area based on a change in weight. Hence, the exact change in weight may not be needed or be connected to a specific article identity, since it may be sufficient to detect that the weight has increased or decreased in order to determine if an article has been picked or placed in the article containing area.

In one embodiment the one or more scales are arranged to determine if an article has been picked from or being placed in the article containing area based on a change in“weight”. Hence, the exact change in weight is not known, but samples from the load cells will be monitored and learned. Based on the learned samples the article type may be identified by detecting the change in the load cells of the part of the shelf A storing the particular article type. Yet further, the data measured by the load cells may further be used to identify the number of articles being picked from the shelf A.

The sensor arrangement 110 may also include one or more capacitive sensors. The one or more capacitive sensors are preferably arranged in conjunction with the shelfs A. The capacitive sensors may be arranged on the bottom of the shelf and/or on the top of the shelf. The capacitive sensors are arranged to detect when a customer’s hand and/or arm (or other part of the upper extremity of the customer) is approaching or touching the article containing area. The capacitive sensor preferably comprises a receiver and a transmitter, where an electric field is formed between the receiver and transmitter. An electric field, so called fringe electric field, also extends out from the transmitter. Once at least a part of the upper extremity of the customer is approaching the sensor, the fringe electric field is changed and a change in environment is detected. As will be described more in detail later, this information may be used to trigger another sensor(s) of the sensor arrangement 110.

The sensor arrangement may also include one or more LIDAR sensors, which may be attached to a shelf or in the ceiling of the article containing area. The LIDAR sensor is preferably mounted so that it creates a vertical plane in front of an article container, e.g. a shelf A. When a customer or attendant reaches for an article in the shelf A the LIDAR sensor will detect an exact x-y coordinate of where the plane was obstructed. The system may be configured to have information of every unique article’s exact position in the shelf A, thus by using the coordinate information from one or more LIDAR sensor the article identity can be accurately determined. The information from the LIDAR sensors may also be used as learning data to the processor 120 when articles are put into the shelf, e.g. as performed when attendants are unpacking new articles for display.

The sensor arrangement 130 may also include one or more IR cameras which may be attached to a shelf or in the ceiling of the article containing area A-I. IR cameras may be specifically advantageous for identifying cold and/or hot articles.

As is evident from the foregoing description, the above example only represents some possible uses of different sensors 112. The above-mentioned description could thus also be implemented using another type of sensor, such as a 30- camera, a spectrometer, etc.,

The communication interface

The sensor arrangement 110, and thus the individual sensors 112,

communicates with the processor 120 by the use of a communication interface. The communication interface is adapted to allow the sensor arrangement 110 to

communicate with the processor 120, through the use of different communication technologies. Such communication technologies may be wired or wireless. Examples of such wired technologies are Universal Serial Bus (USB) and Ethernet to name a few. It should be noted that other technologies exist and are taken to be an obvious equivalent for such wired communication interfaces. Examples of such wireless technologies are IEEE 802.11, IEEE 802.15, ZigBee, WirelessHART, WIFI, Bluetooth®, W-

CDMA/HSPA, GSM, UTRAN and LTE to name a few. It should be noted that other technologies exist and are taken to be an obvious equivalent for such wireless communication interfaces. The processor

The processor 120, or a plurality of processors 120 arranged in communication with each other, is connected to the sensor arrangement 110. The at least one processor 120 is configured to determine the identity of an article picked by the customer 1. The processor 120 is responsible for the overall operation of the sensor arrangement 120 and the registration account 130 and is preferably implemented by any commercially available CPU ("Central Processing Unit"), DSP ("Digital Signal Processor") or any other electronic programmable logic device. The processor could by a physical device, or being part of a network computing system such as a cloud. The processor 120 is configured to read instructions from a memory 122 and to execute these instructions to control the operation of the sensor arrangement 110 and/or the registration account 130. The memory 122 may be implemented using any commonly known technology for computer-readable memories such as ROM, RAM, SRAM, DRAM, CMOS, FLASH, DDR, SDRAM or some other memory technology. The memory 122 is used for various purposes by the processor 120, such as for storing application data and program instructions, as well as storing reference data, such as reference signatures, for the articles of the store 10.

The processor 120 is further connected to the communication interface in order to communicate with the sensor arrangement 110, the registration account 130 and/or an external electronic device (such as a mobile phone, tablet or any other computing device). Such communication technologies may be wired or wireless, as described in relation to the communication interface arranged in the sensor arrangement 110.

The registration account

In some embodiments the registration account 130 is created and connected to the customer in a direct way, i.e. by tracking a customer inside a store, and in some embodiments the registration account 130 is created and connected to an accompanying unit such as an article carrying device 5. The respective embodiments will now be described together with preferred methods of adding identified articles to the registration account. Article carrying device - registration account I^st embodiment

In some embodiments the registration account 130 is created and connected to an accompanying unit such as an article carrying device. The article carrying device 5 may for example be a shopping cart, basket or a bag. The article carrying arrangement 5 could also be the upper extremity of the customer, and more specifically the hands and/or arms of the customer. The article carrying arrangement 5 could thus be a physical or virtual arrangement, and the arrangement 5 could be a part of the customer. In the following, the article device or arrangement 5 will be embodied as a shopping cart, basket or a bag, however it should be understood that the description is applicable to the embodiments where the article carrying arrangement 5 is the upper extremity of the customer and the embodiments where the article carrying arrangement 5 is a stationary predetermined area.

The registration account 130 keeps a list of the articles that has been collected in the store 10. The registration account 130 is in communication with the processor 120 and during checkout the list of articles is automatically transferred from the registration account 130 to an associated payment system. The registration account 130 comprises a link to an article carrying device 5, represented by means of a unique ID or similar, and a dynamic article list of the items being picked from the article containing areas A-I.

As soon as the person pulls out the article carrying device 5 form its idle location, the article carrying device 5 will be associated with a unique registration account 130. Alternatively, the customer may e.g. press an activation button arranged on the article carrying device 5 or at another position at the entrance area for allowing the article carrying device 5 to be associated with the registration account 130. Association of the article carrying device 5 and the registration account 130 is required in order to make use of the registration system 100. The registration account 130 may either be managed using the article carrying device 5 itself or by connecting the article carrying device 5 to a mobile electronic device, as will be described further below.

In one embodiment, the article carrying device 5 in the form of a shopping cart 5 is used to manage the registration account 130. The shopping cart 5 thus needs to have necessary hardware for allowing the customer to interact with the registration account 130. In one embodiment, the article carrying device 5 may comprise an interactive display and equipment for allowing for manual registration of articles if needed. Such articles may e.g. by bulk articles or similar. The communication interface of the article carrying device 5 may also allow the customer to review and/or mange the registration account 130. It is beneficial if the in-store registration system 100 can gain information of the position of the article carrying device 5. This may be done in several ways. In one embodiment the article carrying device 5 comprises a unique identification tag. This identification tag is read by the sensor arrangement 110, wherein the positions of the individual sensors are known, in order to determine a current position of the article carrying device 5. The identification tag may for example be a QR-code or another unique pattern which can be used for the sensor arrangement 110 to identify the article carrying device 5. In an alternative embodiment the article carrying device 5 may comprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE), short-range radio frequency positioning and/or Wifi-based positioning systems.

The use of a smart article carrying device 5 allows the registration account 130 to be associated with a current position in the store without tracking the individual customer itself. It can be assumed that the customer picks an article in an article containing area A-I which is within a specified distance from the article carrying device 5. Such presumption reduces the number of possible articles to choose from when identifying a fetched article. This information is used in the identification process of the article, as will be described more in detail with reference to Fig. 5.

Here the customer does not need any other physical device than the article carrying device 5 in order to properly use the registration system 100. Additionally, there is no need for the customer 1 to manually check in when entering the store and multiple customers may use the same article carrying device 5 and thus the same account 130 without requiring adaptation of the registration account 130. Having the registration account 130 associated with an article carrying device 5 has the benefit that the account is not directly associated with the customer and his/her identity.

The embodiment where the registration account 130 is managed using the article carrying device 5 per se is preferably used when the probability of correct identification of an article is very high, so that manual registration is rarely needed. Article carrying device - registration account 2^nd embodiment

An embodiment in which the registration account 130 is associated with an article carrying device 5 will now be described. In some embodiments, the registration account 130 is managed by connecting the article carrying device 5 to a separate physical device such as an external electronic device. The external device may be a mobile device, such as a mobile phone or other suitable devices for allowing the customer to interact with the registration account 130 In either case, the article carrying device 5 does not need to have any special hardware, display or position detection unit.

As soon as the person pulls out the article carrying device 5 form its idle location, the article carrying device 5 will be associated with a unique registration account 130. Alternatively, the customer may e.g. press an activation button arranged at the entrance area for allowing the article carrying device 5 to be associated with the registration account 130, or use the external device to make the association between the registration account 130 and the article carrying device 5. Association of the article carrying device 5 and the registration account 130 is required in order to make use of the registration system 100.

Preferably, the article carrying device 5 comprises a unique identification tag which the sensors arrangement 120 reads in order to gain the position of the article carrying device 5. Since the positions of the individual sensors in the sensor arrangement are known, it is possible to determine the position of the article carrying device 5. The identification tag may for example be a QR-code or another unique pattern which can be used for the sensor arrangement 110 to identify the article carrying device 5.

When an external device, such as a mobile phone or similar, is used for managing the registration account 130 the article carrying device 5 may be provided with a unique code arranged to be scanned by an application on the external device.

Such a code may be in the form of a QR-code, EAN, UPC code or an RFID tag. By scanning the code arranged on the article carrying device 5, the registration account 130 can e.g. be managed using the application of the external device. This allows the customer to manually scan articles using the external device when the registration system 100 does not manage to correctly identify an article, and to manually remove identified articles which for some reason were decided not to be bought. The application of the external device must thus preferably be able to scan codes of an article. The external device may also allow the customer to review and/or mange its registration account 130. For this purpose the external device may e.g. be a dedicated portable code scanning unit. In one embodiment, the article carrying device 5 is associated with a specific registration account 130 by using Near Field Communication.

In one embodiment, the article carrying device 5 is arranged with a NFC-tag. The article carrying device 5 can thus be connected with the external device having a NFC-reader, such as a mobile phone. Hence, in one embodiment a NFC-tag, being readable by an external device of the customer, is arranged on the article carrying device 5. In an alternative embodiment, a NFC-reader is arranged on the article carrying device 5, is configured to read aNFC-tag on an external device of the customer. The external device may for example be a tag arranged in a credit card.

The tags used by the customer when linking the article carrying device 5 to the registration account 130, and the tag used for identifying the article carrying device 5 by means of the sensor arrangement 110 may not necessarily be the same. In some embodiments, the position of the article carrying device 5 may be determined using a positioning unit on the external device.

The embodiment where the registration account 130 is managed using an external device connected to the article carrying device 5 is preferably used when the probability of correct identification of an article is very high, so that manual registration is rarely needed. Article carrying device - First identification step: Limitation

The method of automatic registration of an article will now be described in more detail with reference to Fig. 4a. The following description is related to the case where the registration account 130 is associated with an article carrying device 5, with or without the use of an external device. As been mentioned above, in a first step 202 the sensor arrangement 110 monitors all article containing areas A-I arranged in the store. By continuously monitoring an article containing area A-I, it is possible to recognize 204 when a change in the article containing area A-I has occurred.

This may be done by comparing data streams generated by the sensor arrangement 110 and by comparing the data frame by frame. Additionally or alternatively, the change in the article containing area A-I may be detected by at least one sensor in the sensor arrangement 110. The detected change will then trigger, i.e. activate, at least one other sensor in the sensor arrangement 110 and/or the process of identifying the article causing the change.

The sensor(s) detecting the change may for example be at least one capacitive sensor arranged in the article containing area A-I. The capacitive sensor detects that something is causing a change in the electromagnetic environment of the containing area A-I. Such a change may for example be the a part of the customer’s upper extremity (such as the hand and/or arm) that is approaching, retracting, occupying or touching the article containing area A-I. This information will be used by the system in order to initiate the identification process.

A change in article containing area A-I may indicate that a customer 1 has selected and removed an article 7 from the containing area A-I. This article 7 needs to be, at least in part, identified 206 in order to finally be added 216 to the registration account 130 associated with the article carrying device 5. In one embodiment, the first identification step is performed once the system 100 recognizes a change in the article containing area A-I and a second identification step is performed once the customer puts an article into the article carrying device 5. In another embodiment, the article is in fact completely identified directly as it is removed from the article container A-I. However, in order to connect the identified article with the correct article carrying device 5, the step of adding the article to a list of possible article identities and having a second identification step at the article carrying device 5 may still be preferred.

The first identification step 206 comprises two main steps; limitation of possible article identities and identification of a unique article, which steps will now be described more in detail. These two steps may be performed in sequence or

simultaneously. The limitation process is based on the fact that the article containing area A-I has a known position in the store and a known range of articles and the limitation process uses this information in order to at least partly identify the article. If an article is removed from one article containing area A, it is a very high probability that said article is one of the articles which usually is accommodated in that article containing area A. This limitation process based on position decreases the number of possible articles. As previously mentioned, each article containing area may be divided into a plurality of segments. The segmentation of the article containing areas further increases the position information of the articles contained therein. Moreover, capacitive sensors may be arranged in each segment of the article containing area to further specify the exact position of the article being removed from the area.

Once the number of possible articles is reduced, the system 100 needs to narrow down the possible articles to only one unique article. This identification may be achieved using different identification techniques. One such technique may be an identification algorithm, which uses information gathered from the sensor arrangement 110 as an input. The sensor arrangement 110 comprises at least one sensor 112 which is configured to determine a measured signature of the article being taken from the article container. The sensors 112 may be switched on, i.e. activated, in sequences to be able to find beneficial combinations or a partial set of or all the sensors may be active until one of the combinations provides a positive decision, wherein one or more of the redundant sensors may be disconnected. A positive decision is here when all sensors in the combination have detected and identified a predetermined property of an article, where the properties in combination together give the article an identity. The identity may be determined through control against a database comprising properties, or signatures, of an amount of articles. The database may be stored in the memory unit according to what has previously been described. Examples of properties are size, color, shape, contour, marking with barcode and/or text and/or figure and/or pattern. In order to provide a successful classification, an activated sensor determines a measured signature of an article. The measured signature is associated with the sensed signal, and may thus be a digital representation of a number of different article properties. The processor 120 is for this purpose configured to compare the measured signature with the digital reference signatures stored in the memory unit, and to calculate a matching probability of a predetermined number of article identities. The latter step is preferably performed by comparing the measured signature with all, or a subset of, the digital reference signatures of the memory unit and subsequently delivering the article identities having the highest matching probabilities to a further classification algorithm, such as a BBN or a neural network. In case of using two or more different sensors thus providing two or more different measured signatures, the processor 120 is configured to independently compare each sensors measured signature with all, or a subset of, the digital reference signatures of every sensor. The identified articles and their matching probabilities from two or more sensors are transmitted to the network for further analysis and to combine the highest matching probabilities of each sensor into one combined matching probability.

The articles may also be identified using a pattern recognition algorithm based on the data collected by the sensor arrangement 110. In one embodiment the pattern recognition algorithm is based on SVM, which preferably uses the measured signatures and the digital reference signatures mentioned above as input to the support vector machine model. In an alternative embodiment of the algorithm the system is continuously learned by using the raw data from the sensor arrangement 110. In this situation, the algorithm preferably uses deep learning methods, such as Convolutional neural network or Fully Connected neural network.

The different identification algorithms discussed above may be used separately or as subsequent steps in a combined identification routine. In the latter method the outcome of each step is combined in order to produce a unique identified article with the highest matching probability.

The classification method may further include a step of comparing the highest matching probability with an alarm threshold and, in case the highest matching probability is below the alarm threshold, the system awaits manual input from the customer in the store before proceeding. In one embodiment the identification includes the step of comparing the highest matching probability with two alarm threshold wherein, in case the highest matching probability is above the lowest alarm threshold but below the upper alarm threshold, the method awaits manual input from the customer in the store, and in case the highest matching probability is below the lowest alarm threshold, the system flags the registration account for additional security check at the checkout station, as will be described more in detail later on. In an alternative embodiment, the attendant is not needed and the customer input is sufficient. In one embodiment the system 100 identifies the article based on at least one image taken of at least a part of the customer’s hand and/or arm when it is in the article containing area and/or when the hand and/or arm is removed from the area.

In one embodiment, once the identification process is initiated, preferably by the sensor arrangement 110 as discussed above, the sensor arrangement 110 will take an image of at least a portion of a user’s hand and/or arm being captured after the user’s hand and/or arm is removed from article containing area. The image will thus contain information of the article being picked from the article containing area. This information may be used, possibly together with other output from the sensor arrangement 110, in order to identify the article. Additionally, or alternatively, the image may be used to determine if the customer is holding an article or not. Hence, the information gathered from the image of the customer’s hand and/or arm may not be used to directly identify the article.

Additionally, or alternatively, the sensor arrangement 110 takes an image of at least a portion of a user’s hand and/or arm being captured when the user’s hand and/or arm is inside the article containing area. This information may be used to determine where in the article containing area the hand and/or arm, i.e. the article to be picked, is located. This information can thus be used to gain more accurate position data.

In one embodiment the sensor arrangement 110 is activated based on one type of sensors, such as weight sensors, LIDAR-sensor and/or capacitive sensors. After the activation, the sensor arrangement 110 takes an image of at least a portion of a user’s hand and/or arm being captured when the user’s hand and/or arm is inside the article containing area and one image of at least a portion of a user’s hand and/or arm being captured after the user’s hand and/or arm is removed from article containing area. This information may be used, possibly together with other output from the sensor arrangement 110, in order to identify the article and/or to gain more accurate position data.

Additionally, or alternatively, the sensor arrangement 110 is arranged to first take an image of the article containing area when a customer’s hand and/or arm is approaching the area. In a next step, an image is taken of the customer’s hand and/or arm while being inside the article containing area. Finally, an image is taken of the customer’s hand and/or arm when it is moving away from the article containing area, possibly holding a picked article. These images are evaluated in order to identify the article. The information gained from the images may be position data, knowing if an article was removed or added to the article containing area and/or for identification of the article.

For the embodiments described above the picking position is used to limit the total number of possible articles, and the network is used to further limit the number of possible articles to a single, positively identified article. However, it may also be possible to determine the picking position accurately so that a single article is positively identified using only position data. As explained above, LIDAR sensors, weight sensors and/or capacitive sensors may be used to accomplish this. In these situations, it is not necessary to make use of the network for further identification of the article using for example images of the customer’s hand and/or arm. Hence, step 204 and 206 in Fig. 4a may be performed merely on position data.

For all cases described above the first identification step may result in a single, positively identified article, or a list of possible articles.

The information that an article, identified or unidentified, has been taken from its position in the article containing area A-I may be used in a second identification process either to verify the identification from the first identification step or to provide a complete identification of an article which was only partly identified in the first identification step. The second identification step comprises two main steps; limitation of possible article identities and identification or verification of a unique article. These two steps may be performed in sequence or simultaneously.

The second identification process may be initiated when something is moved inside the boundaries of the article carrying device 5 (seen from above when something is moving within the railing rectangle of the article carrying device 5), and the system thus detects 208 that an article is being placed in the article carrying device 5. Although the first identification process may have succeeded in identifying the article when being removed from the article containing area, the system must also determine to which registration account 130 said article belongs to or if this has already been determined to verify that the article and the registration account 130 belongs to each other. Once the system detects that an article is being placed in the article carrying device 5, the list of previously identified articles, or partly identified articles, is fetched 210 and the number of articles in the list is reduced 212 based on the position of the article carrying device 5. Finally, the article is identified 214 from the reduced number of article identities in the list and the article is added 216 to the registration account. These steps will now be described more in detail. Fig. 5 illustrates the relationship between an article in an article containing area A-D and the time at which the customer removes the article from the article containing area A-D. Article containing area A comprises articles named a, article containing area B comprises articles named b, article containing area C comprises articles named c and article containing area D comprises articles named d. In this example, the article containing areas A-D are arranged within the predetermined distance from the article carrying device 5. All articles arranged in the article containing areas A-D are thus possible candidates when limiting the number of possible articles. In order to further narrow down the possible articles during the limitation process, the time at which the article was taken from the article container is used.

All articles that are taken from the shelves A-D will be saved in a temporary list, and will be saved thereon for a predetermined period of time tl before they are removed from the list. The memory 122 saves information of articles being taken from each article containing area A-D for a predefined period of time tl. The time period allows the registration system 100 to have a limited amount of articles to choose from. Hence, the shorter period of time the less number of possible articles to choose from in the identification process. It may take a while from the time when the customer takes the article from the article container until he/she puts the article into the article carrying device 5. The customer may for example grab a couple of different articles before putting them all into the article carrying device 5. The time tl needs thus be sufficiently long, so that the article still is among the possible articles during identification even if the customer has been slow in putting the article into the article carrying device 5.

If the article was not identified when picked from the article container, all articles that were within a predetermined distance from said article container will be available in the temporary list. The registration system 100 keeps track of the position of the article carrying device 5, as previously described. This position information is used to limit the possible article candidates as described with reference to Fig. 5. The possible number of articles may be limited by analyzing the location of the article carrying device 5. Only articles that are in a predetermined distance from the article carrying device 5 will be considered to be candidates when the system 100 operates to identify the chosen article. Thus in the second identification step the registration system 100 will use a modified version of the list generated when an article is taken from the article containing area. The list of possible article candidates available during the second identification step will only comprise information relating to articles that have been removed from article containers within a predetermined distance from the article carrying device 5, and within a predetermined time. The number of possible articles which corresponds to the article that has been placed into the article carrying device 5 is thus greatly reduced.

Hence, the registration system 100 uses the list to identify that the identified article corresponds to the right registration account 130. If there are several possible candidates the registration system 100 identifies the article once the article is placed into the article carrying device 5 in the same or similar manner as the first identification process. The second identification step may be performed when the article is approaching the article carrying device 5, such that the article is moving across a virtual boundary set at a predetermined distance from the article carrying device 5. The virtual boundary may coincide with the physical dimensions of the article carrying device 5.

As with the first identification step, the identification may be achieved using different identification techniques. One such technique may be an identification algorithm, which uses information gathered from the sensor arrangement 110 as input. In the second identification step, the input to the algorithm is taken from sensor(s) arranged such that the sensor(s) oversees the article carrying device 5 in the store. The content of the article carrying device 5 may be analyzed continuously so that it is possible to detect once an article is placed into, or close to the article carrying device 5 as explained above. The at least one sensor 112 collects data from the shopping cart before and after the customer approaches it and it is thus possible to detect when an article has been added or removed from the article carrying device 5. The sensor arrangement 110 may use different identification algorithms in order to identify the articles being arranged in the shopping basket.

In one embodiment the system continuously monitor the article carrying device 5 by means of the sensor arrangement 110 and receives a first image, from the sensor arrangement 110, of at least a portion of a customer’s hand and/or arm being captured when the customer’s hand and/or arm is moving across a virtual boundary set at a predetermined distance from the article carrying device 5. In this image the customer is holding an article which he/she intends to put in the article carrying device. In one embodiment, the system will make the final identification of the article, among the reduced number of articles on the temporary article list, at least in part based on this first image.

The system may, in some embodiments, further receive a second image of at least a portion of a customer’s hand and/or arm being captured when the customer’s hand and/or arm is moving away from the virtual boundary set at a predetermined distance from the article carrying device 5. In this image, the customer has placed the article into the article carrying device 5 and is thus no longer holding the article. In one embodiment, the system will make the final identification of the article, among the reduced number of articles on the temporary article list, at least in part based on the first and second image.

Additionally, or alternative, in some embodiments, the system is arranged to receive an image of the interior of the article carrying device 5 prior to the customer’s hand and/or arm moving across a virtual boundary set at a predetermined distance from the article carrying device 5 and to receive an image of the interior of the article carrying device 5 after the customer’s hand and/or arm is removed from the virtual boundary set. The system is further arranged to identify, based at least in part on a comparison of the two images, that an item has been added to, removed from or rearranged in the article carrying device 5. In some embodiments, the system is further arranged to identify the article, among the reduced number of articles on the temporary article list, based at least in part on said comparison of the images. In other

embodiments, the system is arranged to identify the article, based at least in part on said comparison of the images. Hence, in some embodiments the use of a temporary article list is not needed.

If the registration system 100 failed to completely identify the article once removed from the article container, the system 100 gets another chance of identifying the article by analyzing the article carrying device 5 as has been described above. If the article was not identified when picked from the article container, all articles that were within a predetermined distance from said article container will be available in the list.

As soon as the article has been detected and identified, the registration system 100 may determine whether the article was really picked up by the customer or if it was put back in the article containing area A-I. This may be done by analyzing sensor data from the article containing area before and after the customer approached it. It is also possible to use sensor data for searching article specific patterns inside the customers’ article carrying device 5. In one embodiment, in case a customer returns an article that is already registered to its account, the registration system 100 recognizes that the identified article that was returned and removes the article from the registration account 130.

In one embodiment, the processor 120 is further configured to receive a second image from the sensor arrangement 110 of at least a portion of a customer’s hand and/or arm being captured when the customer’s hand and/or arm is moving away from the virtual boundary set. The processor 120 determines, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article held in the customer’s hand and/or arm.

If there is an article in the first image, the processor 120 determines if the second image includes a representation of an article. If it is determined that the first image includes a representation of an article and that the second image includes a representation of a different article as of the first image, both articles are identified. The article identity from the first image is added to the registration account, and the article identity of the second image is removed from the registration account. If it is determined, based at least in part on a comparison of the first image and the second image, that the first image includes a representation of an article held in the customer’s hand and/or arm and that the second image includes a representation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account.

If it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article held in the customer’s hand and/or arm, the method further comprises the step of identifying said article at least in part based on the second image from the article identities currently included in the dynamic list of articles of the registration account 130. The processor is then configured to remove the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account 130.

Once the registration system 100 has identified the article, and possibly verified that it was actually collected by the customer or added to the article carrying device 5, the article identity is added to the associated registration account 130. Article carrying device - Adding weight priced articles to a registration account

In Fig. 4b a method for adding weight priced articles to a registration account associated with an article carrying device 5 is schematically shown. The following description is related to the case where the registration account 130 is associated with an article carrying device 5, with or without the use of an external device.

In Fig. 4b some of the method steps are identical to the method steps of Fig. 4a.

Therefore, these are assigned the same reference numerals.

In step 408 the system detects that an article is being placed on an article weighing device located in the store. The article weighing device is preferably forming part of the sensor arrangement 110. This event triggers the following steps. The position of the article weighing device is determined and following this, in step 410 the list of previously identified articles is fetched, and the number of possible articles is reduced by a factor corresponding to the position of the article weighing device. For this, it is possible that all articles being picked by customers are added to the same list of possible articles. This means that as soon as an article is picked from the article containing area, it is added to the list (optionally together with possible alternatives). The articles remain on the list until an article is added to a registration account or it is added to the scale for price calculation by weighing, at which point the article is also removed from the list. Any additional articles that were linked to the positively identified article are also removed from the list.

However, in step 412 the number of articles on the list is reduced in order to determine the correct article. This reduction is however not made permanent to the list, but only used as a selection for the final determination step 414. The reduction of number of articles is made based on the position of the article weighing device, such that only the articles being present on the list and normally belonging to a defined and limited area surrounding the position of the article weighing device is used for the final identification before weighing. In step 414, the correct article is identified based on the reduced number of articles from the list. This final identification before weighing may e.g. be made by monitoring the article weighing device, and determine, by means of the sensor arrangement 110, the correct identity of the article. This step is greatly simplified by only considering the reduced number of possible articles. If the identification in step 414 can’t determine a unique article, then the customer selects the correct article on the article weighing device screen. The possible articles to select from can either be the reduced articles from step 412 or all weight priced articles in the store. In step 417 is the article weighed by the article weighing device and the price is then calculated. In step 418 is the article again added to the list of identified articles together with the calculated price and measured weight.

In step 208 the system detects that an article is being placed into an article carrying device 5. This event triggers the following steps. The position of the article carrying device 5 is determined and following this, in step 210 the list of previously identified articles is fetched including the weight priced article that was added in step 418, and the number of possible articles is reduced by a factor corresponding to the position of the article carrying device 5. For this, it is possible that all articles being picked by customers are added to the same list of possible articles. This means that as soon as an article is picked from the article containing area, it is added to the list (optionally together with possible alternatives). The articles remain on the list until an article is added to a registration account, at which point the article is also removed from the list. There is also a possibility, in one embodiment, to track the customers between step 418 and step 208, to simplify the identification when placing the weight priced article into the article carrying device 5.

However, in step 212 the number of articles on the list is reduced in order to determine the correct article. This reduction is however not made permanent to the list, but only used as a selection for the final determination step 214. The reduction of number of articles is made based on the position of the article carrying device 5, such that only the articles being present on the list and normally belonging to a defined and limited area surrounding the position of the article carrying device 5 is used for the final identification. In step 214, the correct article is identified based on the reduced number of articles from the list. This final identification may e.g. be made by monitoring the article carrying device 5, and determine, by means of the sensor arrangement 110, the correct identity of the article. This step is greatly simplified by only considering the reduced number of possible articles. In a final step 216 the article is added to the registration account 130 and the customer is preferably notified that the article has been added to the list of his registration account 130.

If the registration system 100 fails to identify the article correctly, e.g. if a matching probability computed by the registration system 100 is not above a predefined threshold, in step 212 the system 100 notifies the customer to manually identify the article. Once the customer has manually added the article, the article identity is added to the registration account 130.

The user interface e.g. provided by the application of the external device, or by a display arranged on the article carrying device 5, may also allow customers to manually add articles to the registration account without first trying automatic identification. This may e.g. be particularly suitable for bulk goods, whereby the registration system 100 may automatically detect that the customer (i.e. the article carrying device 5) is arranged in an area of the store where bulk goods is present. By such detection, the registration system 100 may prompt the customer to manually input picked articles, whereby these articles are stored in the registration account 130.

In one embodiment, as shown in Fig. 4c, the method of automatic registration of an article, where the registration account 130 is associated with an article carrying device 5, with or without the use of an external device, does not comprise the first identification step. Hence, the identification of the article is performed solely by the use of the second identification step. The method comprises, associate 310 an article carrying device 5 with a specific registration account 130. The sensor arrangement 110 is configured to continuously monitor 312 the article carrying device 5. The processor is configured to receive 314 a first image from the sensor arrangement 110 of at least a portion of a customer’s hand and/or arm being captured when the customer’s hand and/or arm is moving across a virtual boundary set at a predetermined distance from the article carrying device 5. The processor is further configured to determine, based at least in part on the first image, if the image contains an article. In other words, the processor is configured to determine 316 if the first image includes a representation of an article held in the customer’s hand and/or arm. If the image includes a representation of an article, the article is identified 320. The identity of the identified article is added 322 to the dynamic list of articles of the registration account 130.

In an optional step, once the system detects an arm and/or hand is moving across a virtual boundary set, the position of the article carrying device 5 is registered. Based on the position of the article carrying device, the number of possible articles which the first image represents is limited. Preferably, only article identities that are arranged on an article containing area within a predetermined area are used when making the identification of article.

In one embodiment, as shown in Fig. 4d, the processor 120 is further configured to receive 323 a second image from the sensor arrangement 110 of at least a portion of a customer’s hand and/or arm being captured when the customer’s hand and/or arm is moving away from the virtual boundary set. The processor 120 determines, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article held in the customer’s hand and/or arm.

If there is an article in the first image, the processor 120 determines 324 if the second image includes a representation of the article of the first image. If it does not include the same article, the article of the first image is identified and added to the registration account. If it is determined 324, based at least in part on a comparison of the first image and the second image, that the first image includes a representation of an article held in the customer’s hand and/or arm and that the second image includes a representation of the same (in step 326) article as of the first image, the article is not added 328 to the dynamic list of articles of the registration account. In some embodiments, the article is identified before step 324. In some embodiments, the article is identified after step 324. In yet some embodiments, where it is determined that the article in the first and second image are the same, the article is not identified at all.

If it is determined in step 326, that the first image includes a representation of an article and that the second image includes a representation of a different article as of the first image, both articles are identified 330. The article identity from the first image is added to the registration account, and the article identity of the second image is removed from the registration account in step 332. As should be understood, the order of these steps may differ, for example the article in the first image may be identified and added to the registration account before the second article is identified and removed from the registration account, and vice versa.

If it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article held in the customer’s hand and/or arm, the method further comprises the step of identifying 334 said article at least in part based on the second image from the article identities currently included in the dynamic list of articles of the registration account 130. The processor is then configured to remove 336 the identity of the article which was identified in the second

identification from the dynamic list of articles of the registration account 130.

Article carrying arrangement - being a part of the customer

In one embodiment the article carrying arrangement 5 is a virtual arrangement, where the arrangement 5 is a part of the customer’s upper body. In one embodiment, the method for automatic registration of an article in a store comprises the steps of associating an article carrying arrangement 5 with the specific registration account 130, where the registration account 130 is associated with a dynamic list of articles as been described in previous embodiments. The method further comprises the step of continuously monitor the at least one article containing area A-I by means of the sensor arrangement 110, and detect a change in the at least one article containing area A-I and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list.

The method further comprises the step of continuously monitor the article carrying arrangement 5 by means of the sensor arrangement 110. Hence, if the article carrying arrangement 5 is a part of the customer’s upper extremity, the hands and/or arms may be monitored. The virtual boundary set at a predetermined distance from the article carrying arrangement may for example be a distance from the customer’s arms and/or hands. The virtual boundary may also be a distance in one or two direction from the center point of the customer’s upper extremity.

In this embodiment, the method further comprises the step of receiving a first image from the sensor arrangement 110 of at least a portion of an article being captured when at least a part of the article is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement 5. In a next step, determine, based at least in part on the first image, the identity of the article being represented in said first image in a second identification step from the article identities currently included in the temporary article list and add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account 130.

In yet one embodiment, a method for automatic registration of an article in a store comprises the steps of associate an article carrying arrangement with a specific registration account and continuously monitor the article carrying arrangement by means of the sensor arrangement. The method further comprises the step of receiving a first image from the sensor arrangement of at least a portion of an article being captured when at least a part of the article is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement, determine, based at least in part on the first image, the identity of the article being represented in said first image, and add the identity of the identified article to the dynamic list of articles of the registration account.

Although the two above described embodiments receives an image of an article and not a hand and/or arm and/or other part of the upper extremity of the customer, the same procedure using a plurality of images to determine if the article is to be placed and/or removed from the registration account as has been described with the other embodiments herein apply.

Customer - registration account

In the previous, the description has been focused on the embodiments where the registration account 130 is associated with an article carrying device 5. An embodiment where the customer is directly associated with a registration account 130 will now be described. The connection between the registration account 130 and the customer 1 may be done in several ways. In one embodiment the customer 1 is tracked inside the store using image recognition software, such as person tracking system, 3D image tracking from area cameras or other person-tracking algorithms known per se.

The sensor arrangement 110 may e.g. be used for providing input data to such positioning. The registration system 100 will therefore know the position of the customer 1 by following the customer as he or she moves around the store 10. Hence, no further position detecting unit may be needed. However, in order to be able to manually register articles and to perform a check-in operation the customer may use some external device associated with the registration account 130.

In one embodiment the customer is connected to a registration account 130 by the use of an external device such as a mobile application on the smart phone or tablet of the customer 1 or any other electronic device carried by the customer.

Once the customer 1 enters the entrance area 12 of the store 10 he or she is associated with a registration account 130. This may be done e.g. by using a fixed unit arranged in the entrance area or on an external device of the customer. In one embodiment, the entrance of the store is arranged with aNFC-tag. The customer can thus be connected to the registration account 130 by using his/hers external device having a NFC-reader and connect with the NFC-tag arranged in the store. In yet some embodiments, the store is arranged with a NFC-reader in the entrance area. The customer is connected to the registration account 130 by aNFC-tag preferably arranged in an external device of the customer. The NFC-reader of the store thus connects with the NFC-tag of the customer.

In some embodiments, the registration system 100 further identifies the appearance of the customer 1 in the entrance area, which allows the system 100 to track the customer throughout the store.

The registration system 100 needs to determine the position of the customer. This may either be done by the external electronic device of the customer, if it comprises a position detecting unit, or the position of the customer may be determined by direct tracking the customer as the customer walks through the store.

If multiple customers belongs together (such as a family) and arrive to the store at different time it is possible to connect them to the same registration account 130 by manually merging their accounts. Preferably, the merge operation is made by the customers themselves without any interaction with store attendants.

If multiple customers belonging together arrive to the store at the same time, it is desirable if the registration system adds these multiple persons to the same registration account 130. Preferably, automatic recognition of multiple persons shopping together is possible in the check-in area 12. These customers will thus be connected to same registration account 130 during their shopping session. It is also possible to merge accounts inside the store (i.e. in the article container area) by use of the external device.

Customer - Adding articles to a registration account

A method of adding an article to a registration account 130 when the registration account is associated with the customer will now be described with reference to Fig. 6. In a first step 402 the sensor arrangement 110 monitors all article containers A-I arranged in the store. By continuously monitoring an article containing area, it is possible in step 404 to recognize when a change in the article containing area has occurred. Such a change may indicate that a customer has selected and fetched an article from the containing area. Once the registration system 100 recognizes that an article has been picked by a customer, the registration system identifies the article in step 406. The article may be identified directly as it is removed from the article container A-I using the sensor arrangement 110.

If the article was identified, in step 408, as belonging to the customer picking the article from the article container A-I, the article identity is in step 410 added to the registration account 130 associated with that customer. The customer is preferably notified that the article has been added to the list of his registration account 130.

The user interface e.g. provided by the application of the external device may also allow customers to manually add articles to the registration account without first trying automatic identification. This may e.g. be particularly suitable for bulk goods, whereby the registration system 100 may automatically detect that the customer (or associated article carrying device 5) is arranged in an area of the store where bulk goods is present. By such detection, the registration system 100 may prompt the customer to manually input picked articles, whereby these articles are stored in the registration account.

Customer - First and second identification step

The description relating to the steps of the first identification process for the embodiment where the registration account is associated with the shopping cart are applicable also to the embodiment where the registration account is associated to the customer. More specifically, the system 100 may identify the article based on at least one image taken of at least a part of the customer’s hand and/or arm when it is in the article containing area and/or when the hand and/or arm is removed from the area.

In one embodiment, the sensor arrangement 110 is arranged to first take an image of the article containing area when a customer’s hand and/or arm is approaching the area. In a next step, an image is taken of the customer’s hand and/or arm while being inside the article containing area. Finally, an image is taken of the customer’s hand and/or arm when it is moving away from the article containing area, possibly holding a picked article. These images are evaluated in order to identify the article. The information gained from the images may be position data, knowing if an article was removed or added to the article containing area and/or for identification of the article.

For the embodiments described above the picking position is used to limit the total number of possible articles, and the recognition system is used to further limit the number of possible articles to a single, positively identified article. However, it may also be possible to determine the picking position accurately so that a single article is positively identified using only position data. As explained above, LIDAR sensors, weight sensors and/or capacitive sensors may be used to accomplish this. In these situations, it is not necessary to make use of the recognition system for further identification of the article.

When the registration account is associated with the customer, the number of possible articles may further be limited by analyzing the location of the customer 1. Only articles that are in a predetermined distance from the customer 1 will be considered to be candidates when the system 100 operates to identify the chosen article 7. That distance may be chosen differently depending on the size of the store and the accuracy in determining position of the customer.

In the embodiment where the registration account is associated to the customer the second identification step is somewhat different compared to the situation where the account is associated to an article carrying device 5. If the first identification step is insufficient to identify the article due to a matching probability computed by the registration system 100 being below a predetermined threshold, different options are possible.

According to a first option, in step 412 (see Fig. 6) the system 100 notifies the customer to manually identify the article by scanning a code associated with the article. Once the customer has manually identified the article, the article identity is added to the registration account.

In a second option, the system 100 may in step 412 alert and instruct the customer to manually choose the correct article among a plurality of possible articles. The articles the customer is able to choose from are either the articles that are saved in the reduced article list or a list of articles having high matching probability. The reduced article list is reduced based on the position of the customer, such that only the articles being present on the list and normally belonging to a defined and limited area surrounding the position of the customer is used for the final identification. The customer chooses the correct article identity by using the external device, such as a mobile phone or a tablet. Once the customer has manually identified the article, the article identity is added to the registration account.

In a third option, the customer is not prompted immediately but instead information relating to any unidentified articles is stored in the registration account 130, and the customer will be notified of such unidentified articles during checkout and requested to manually identify those articles. For manual identification, terminals may be provided inside the store for allowing the customer to manually identify articles.

The user interface

The interaction between the customer and the registration system 100 is preferably done by the use of a display unit. In one embodiment, the display unit may be in communication with the processor 120 for displaying information regarding the registration account 130. The display unit is preferably arranged for use by a customer for inputting information or for approving the displayed information. The display unit may be arranged at an article carrying device 5 or as an external device, such as a mobile phone of the customer. The display unit may be an interactive display or a passive display having buttons arranged in connection thereto so as to be able to interact with the display. The display may be a LCD-screen, a LED-screen or any other suitable screen.

The information provided in the display unit may for example be information relating to the list of articles that already has been identified as picked by the customer, the total price of the selected articles, offers or deals for different articles in the store and/or accessible information relating to registration customer account such as previous purchase history. For this the customer may have the possibility to link the current registration account 130 with his customer account for the specific store.

The information provided in the display unit may also relate to identification of an article. If the sensor arrangement 110 identifies the article, an image or a text may be shown in the display unit and if the user finds the displayed information matching the article which the customer has chosen the customer may approve the identity by manual input. Further information may be displayed, e.g. weight and price, wherein the user may approve the displayed information if it is correct. The display unit may further notify the customer when an article needs manual registration. This may be the case if the automatic identification of the chosen article fails. The display unit may thus prompt the customer to manually register the article, for example by using a barcode reader or by scanning or identifying the article in some other way. The display unit may also be used by the customer if the registration system 100 has identified an article which the customer does not want to buy or when the system 100 has identified the article wrongly. The customer may then manually remove the article from the article list of the registration account.

Incorrect article positioning

The system 100 may also be used to ensure the correct position of the articles, for example, if a customer returns an article at an incorrect position. By continuously monitoring the article containing area A-I by the sensor arrangement 110, the automatic in-store registration system 100 may recognize when a customer adds an article back to the article containing area A-I. The system 100 is thus preferably arranged to differentiate between when an article is added or removed from the article containing area A-I. The added article is identified by the registration system 100 and it is determined if the identified article is an article belonging to that position in that article containing area A-I, i.e. in the article containing area A-I that the automatic in-store registration system 100 is trained for.

The article may be added to a different article containing area A-I, if for example the customer picks out an article from the article containing area A and puts it back to another article containing area D, this may for example occur if the customer realizes that the article chosen was in fact not something he/she intended to buy and returns it into a random article containing area D.

If the system 100 determines that the added article was placed in the wrong, position in the correct article container A-I or if the article was placed in the wrong article container A-I, the system 100 will transmit an alarm signal. Preferably, the alarm signal is associated with the position of the article containing area in question whereby it is possible for a store attendant to correct the position of the wrongly arranged article. The alarm signal may also be transmitted to the processor for storing and evaluating the data.

Payment Once all the articles of the customer are correctly identified by the registration system 100, and when the customer enters the checkout area 16, the processor 120 transmits a transaction signal to the checkout station or the checkout station can retrieve the account to allow the customer to finish the payment transaction. Once the customer enters the checkout area 16 the information stored in the registration account 130 is automatically transmitted to the checkout station or the checkout station retrieves the account. The checkout station then transfers the registered articles to a receipt for payment.

In one embodiment, the checkout station transmits a control signal to a surveillance system which will be used in the determination whether or not to open a passage gate. A person is in such embodiments only allowed to exit through the gate(s) 18 of the store 10 if the person 1 has paid for its article(s). This is determined based on information from the registration system 100 and information regarding the payment. The system may initiate security check based on abnormal activity of a certain registration account.

Special conditions for certain articles

In one embodiment, some articles may be associated with a special condition. Such condition may for example be articles that are provided with age restrictions such as alcoholic beverages or pharmaceuticals. It is preferred if the system is capable to prevent articles with age restrictions to be purchased by minors in order to fulfil different regulations.

Some article identities may be arranged with a condition flag. Once an article is identified as an article identity having a condition flag the registration account will in turn be provided with a flag. A registration account being provided with a flag will need assistance from a store attendant, this process may be initiated in different ways.

In one embodiment, the customer will receive a notification that a condition article has been chosen. This may be notified to the customer using the article carrying device 5 if present and/or by an external device. In one embodiment, the customer will then be instructed to go to a special area of the store to get an evaluation of the store attendant if the conditions of the article is fulfilled. Such an area may for example the exit area 16 of the store.

In an alternative embodiment, the customer is notified about the article condition and the system will evaluate if there is enough information about the customer stored in the registration account. If enough data is stored in connection with the customer and the conditions are fulfilled, the condition flag is removed automatically. This may for example be the case if the customer was identified as being at a specific age when registration the registration account.

A special condition of an article may also include an alarm feature. For expensive and/or article being small in size that are easier to steel, the article may be arranged with an alarm device. The alarm device may be an alarm tag, a RFID-tag, a barcode or similar. Such an article is thus arranged with a condition flag in order to notify the customer and/or the store attendant that the alarm device needs to be removed and/or deactivated. In one embodiment, the customer will then be instructed to go to a special area of the store to get the alarm removed and/or deactivated by the store attendant. Such an area may for example the exit area 16 of the store.

Optional weight sensor at exit

In one embodiment, where the registration account is associated with an article carrying device 5, such as a shopping cart, a bag, basket or the like, a weight sensor may be arranged in the exit area 16 of the store to act as a safeguard to the registration system 100. The weight sensor may either be provided before or after the payment process. The total weight of the article carrying device 5, containing all picked articles, are compared to the estimated weight based on the weight of an empty article carrying device 5 and the total weight of the identified articles or it is compared to a trained weight which is saved in the processor 120 for all possible article carrying devices such as shopping cart, bag, basket or the like. If all articles inside the article carrying device 5 are identified by the registration system 100, the estimated weight and the actual weight shall be equal, or the difference shall be within a small permissible range. If the actual weight does not sufficiently correspond to the estimated weight, such information is transmitted to the processor 120 and the customer and/or a store attendant is alerted.

If the article carrying device 5 has a weight lower than the estimated weight, the registration system 100 has either wrongly identified an article or an identified article has been removed from the article carrying device 5 without being correctly removed from the registration account 130.

If the article carrying device 5 has a weight higher than the estimated weight, the registration system 100 has either wrongly identified an article or an unidentified article has been placed into the article carrying device 5 but not being added to the registration account 130. If an article has been placed in the article carrying device 5 but has not been added to the registration account 130, the customer may then perform manual registration of that article.

System learning

The automatic in-store registration system 100 may be subject to training, or learning, in order to improve the accuracy of the identification of articles. For example, the sensor arrangement 110 may be activated during predetermined training sessions, in which a store attendant picks articles in a consecutive order. For each picked article type, the attendant scans at least one article for a secure identification. The system 100 will thus not only learn to identify the article correctly, but also the exact position of the article. For determining the position of the article it is preferred to know the position of the scanner, either by detecting the position by means of the sensor arrangement 110 itself, or by associating the article position (i.e. the shelf or similar) with a unique code which is scanned together with the article code. As a third option the scanner may comprise a position detecting unit such as iBeacon, Bluetooth Low Energy (BLE), short-range radio frequency positioning and/or Wifi-based positioning systems.

System learning may further be improved by using checkout counters, either manually operated, semi-automatic operated, or fully automatically operated.

The training, or learning, may also occur during normal operation of the store, where the store attendant adds or refills articles into the article containing area. For each picked article type, the attendant scans at least one article by a handheld device. The sensor arrangement is thus trained with every single article which the store attendant adds into the article container. Even if the scanner is used only for one or a few articles of each article type, the sensor arrangement 110 can register signatures and position of each article for every article type. The system 100 will thus learn to identify the article correctly by using both the appearance of the article as well as the exact position of the article. Hence, in one embodiment the in-store registration system is trained by defining a virtual boundary set at a predetermined distance from the article carrying arrangement, allowing a store attendant or a customer to add or remove one or more articles to or from the article carrying arrangement, detecting at least a portion of the upper extremity of a person when at least a part of the person’s upper extremity is moving across the virtual boundary set, registering one or more signatures of each article by means of a sensor arrangement 110, and adding or updating information relating to signature and the position of each article to the in-store registration system 100.

In one embodiment the sensor arrangement 110 may be activated during predetermined training sessions in order to learn the system to recognize different movements of a user’s (customer or store attendant) hand/and or arm while being in or near to the article containing area and/or the article carrying device. The system is preferably learned to differentiate between different movement directions, i.e.

from/away from the article containing area and/or the article carrying device. This is beneficial when determining if the article is fetching an article from the article containing area and/or the article carrying device and when determining if an article is removed from the article containing area and/or the article carrying device.

In one embodiment, the automatic in-store registration system 100 is trained or learned using synthetic training data. The system is trained on a synthetically generated dataset with the intention of transfer learning data to real data. The use of synthetic data has several advantages, for example once the synthetic environment is ready it is fast and cheap to produce as much data as needed, and the synthetic environment can be modified to improve the model and training. Moreover, synthetic data can be used as a substitute for certain real data segments that contain, e.g., sensitive information.

In a preferred embodiment, the synthetic environment is a 3D-model of the store and/or of the article carrying device and/or the article containing areas. Hence, the training data may comprise of a synthetically generated 3D-model of at least a part of the store and/or the customer and/or the article containing areas.

The synthetic environment is used to train the system to recognize customer of different skin colors and skin variations. Additionally, the system is trained to recognize customers of different sizes, such as being of different height and weight.

The synthetic data may be enhanced by using Generative Adversarial Networks (GAN). GAN is able to adapt the synthetic data so that it increases it resemblance to the reality. GAN is a deep neural net architectures comprised of two nets, pitting one against the other.

GAN will now shortly be described. A neural network generates new data instances, this neural network may be referred to as a generator. The generator takes in random numbers and then returns an image, this image is then feed into another neural network called the discriminator. The discriminator evaluates the data instances for authenticity, hence this neural network decides whether each instance of data it reviews belongs to the actual training dataset or not. The image received from the generator is transmitted to the discriminator together with a stream of images taken from the actual dataset. The discriminator is arranged to receive both real and fake images. Based on these images it returns probabilities in the form of a number between 0 and 1. The number 0 represents a fake image and with the number 1 represents a prediction of authenticity. In a next step, a double feedback loop is created. The discriminator is in a feedback loop with the ground truth of the images, while the generator is in a feedback loop with the discriminator. During the training phase the generator will continuously improve and eventually be able to generate images that closely resembles real data.

It should be appreciated that even though numerous characteristics and advantages of the present invention have been set forth in the foregoing description, together with details of the structure and function of the invention, the description is only illustrative and changes may be made in detail, especially in matters of shape, size and arrangement of parts within the scope of the invention to the full extent indicated by the appended claims.

Claims

1. A method for automatic registration of an article in a store having at least one article containing area (A-I) and an associated automatic in-store registration system (100), wherein the method comprises:

associate an article carrying arrangement (5) with a specific registration account (130), wherein the registration account (130) is associated with a dynamic list of articles;

continuously monitor the at least one article containing area (A-I) by means of a sensor arrangement (110), and detect a change in the at least one article containing area (A-I) and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list;

continuously monitor the article carrying arrangement (5) by means of the sensor arrangement (110);

receive a first image from the sensor arrangement (110) of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement (5);

determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list; and

add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account (130).

2. The method according to claim 1, wherein the method further comprises the step of:

receive a second image from the sensor arrangement (110) of at least a portion of a customer’s upper extremity being captured when the at least a part of the customer’s upper extremity is moving away from the virtual boundary set; determine, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article and if so determine if the second image does not include a representation of the article of the first image,

and if so, identify the article of the first image in a second identification step from the article identities currently included in the temporary article list, and

add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account (130). 3. The method according to claim 2, wherein if it is determined that the first image includes a representation of an article and that the second image include a representation of the article of the first image, the method further comprises the step of: determine if the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image,

and if the first and second images includes representation of different articles: identify the article of the first image and add the identity of the article to the dynamic list of articles of the registration account (130); and

identify the article of the second image and remove the identity of the article from the dynamic list of articles of the registration account (130).

4. The method according to claim 3, wherein if it is determined that the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account (130).

5. The method according to any one of claims 2 to 4, wherein if it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article, the method further comprises the step of: identifying said article from the second image in a second identification step from the article identities currently included in the dynamic list of articles of the registration account (130), and

remove the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account (130).

6. The method according to any one of the preceding claims, wherein the method further comprises the step of:

receive a third image from the sensor arrangement (110) of the virtual boundary set prior to at least a part of the customer’s upper extremity is moving across the virtual boundary set;

receive a fourth image from the sensor arrangement (110) of the virtual boundary set after to at least a part of the customer’s upper extremity is moving away from the virtual boundary set;

determine, based at least in part on a comparison of the first, third and fourth image, if the article has been added to or removed from the article carrying arrangement

(5),

identify said article in a second identification step from the article identities currently included in the temporary article list,

wherein if is determined that the article has been added to the article carrying arrangement (5), add the identity of the article which was identified in the second identification to the dynamic list of articles of the registration account (130) and wherein if is determined that the article has been removed from the article carrying arrangement (5) remove the identity of the article which was identified in the second identification from the dynamic list of articles of the registration account (130).

7. The method according to any one of the preceding claims, wherein the first identification step is performed by identifying an article by at least using information relating to the position of the change in the article containing area (A-I), wherein the position of each one of the plurality of articles arranged in the article containing area (A-I) is predetermined and accessible for the automatic in-store registration system

(100).

8. The method according to claim 7, wherein the first identification step is performed by reducing the total number of possible article identities using the position information of the change in the article containing area (A-I).

9. The method according to claim 7 or 8, wherein the article containing area (A-I) is arranged with a plurality of segments, wherein the position of each segment is known.

10. The method according to claim 9, wherein each segment of the article containing area comprises at least one capacitive sensor arranged to detect a change in the article containing area (A-I).

11. The method according to any one of the preceding claims, wherein the second identification step is performed by reducing the total number of possible article identities using the information saved in the temporary article list. 12. The method according to claim 11, wherein the second identification step is performed by accessing the article identities saved in the temporary article list and identifying the article as one of the accessed article identities originally positioned within the virtual boundary set at a predetermined distance from the article carrying arrangement (5) and added to the temporary article list within a predetermined time.

13. The method according to any of the preceding claims, wherein the sensor arrangement (110) comprises at least one sensor (112) being selected from the group comprising a 2D-camera, a 3D-camera, an IR camera, a scale, a LIDAR sensor, a capacitive sensor and a spectrometer.

14. The method according to any of the preceding claims, further comprising determining a current position of said article carrying arrangement (5).

15. The method according to any of the preceding claims, wherein step of associating the article carrying arrangement (5) with a specific registration account

(130) is performed using Near Field Communication.

16. An automatic in-store registration system (100) in a store having at least one article containing area (A-I), the system comprising:

at least one processor (120);

at least one registration account (130) being associated with an article carrying arrangement (5); and

a sensor arrangement (110) comprising a plurality of sensors (112), the sensor arrangement (110) being configured to, in a first identification step, continuously monitor the article containing area (A-I) and to detect a change in the article containing area (A-I) and, in response to said detected change, to transmit a first identification signal comprising information relating to one or more possible articles to the processor (120),

and wherein the sensor arrangement (110) is further configured to, in a second identification step, continuously monitor the article carrying arrangement (5) and to detect a change in or near the article carrying arrangement (5) and, in response to said detected change, to transmit a second identification signal comprising a first image to the processor (120);

wherein the processor (120) is configured to identify one or more possible articles from the first identification signal and to save said articles to a temporary article list, wherein the processor (120) is further configured to identify one or more possible articles from the second identification signal by:

receive a first image of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement (5); determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article in a second identification step from the article identities currently included in the temporary article list; and

wherein the processor (120) is further configured to determine a final article by comparing the one or more articles identified from the second identification signal with the articles stored in the dynamic article list, and to add said final article to the registration account (130). 17. A method for training an in-store registration system (100) configured to identify and register articles in a store having at least one article containing area (A-I), said method comprising performing a training sequence in order to improve the accuracy of the identification of articles by:

using training data to recognize different movements of a user’s upper extremity while being in or near the article containing area and/or the article carrying arrangement, and to detect at least a portion of a upper extremity of a person when at least a part of the person’s upper extremity is moving in or near the article containing area and/or the article carrying arrangement,

registering one or more signatures of each article by means of a sensor arrangement (110), and

adding or updating information relating to signature and/or the position of each article to the in-store registration system (100).

18. The training method according to claim 17, wherein the training data comprises of a synthetically generated 3D-model of at least a part of the store and/or the customer.

19. The training method according to claim 17 or 18, wherein the synthetic data is enhanced using Generative Adversarial Networks (GAN).

20. A method for automatic registration of an article in a store having at least one article containing area (A-I) and an associated automatic in-store registration system (100), wherein the method comprises:

continuously monitor the article carrying arrangement (5) by means of a sensor arrangement (110);

determine, based at least in part on the first image, if the first image includes a representation of an article, and if so identify said article, and

add the identity of the identified article to the dynamic list of articles of the registration account (130).

21. The method according to claim 20, wherein the method further comprises the step of:

receive a second image from the sensor arrangement (110) of at least a portion of a customer’s upper extremity being captured when at least a part the customer’s upper extremity is moving away from the virtual boundary set;

determine, based at least in part on a comparison of the first image and the second image, if the first image includes a representation of an article and if so determine if the second image does not include a representation of the article of the first image,

and if so, identify the article of the first image.

22. The method according to claim 21, wherein if it is determined that the first image includes a representation of an article and that the second image include a representation of the article of the first image, the method further comprises the step of: determine if the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image,

23. The method according to claim 22, wherein if it is determined, based at least in part on a comparison of the first image and the second image, that the first image includes a representation of an article and that the second image includes a representation of the same article as of the first image, the article is not added to the dynamic list of articles of the registration account (130). 24. The method according to any one of claims 21 to 23, wherein if it is determined, based at least in part on a comparison of the first image and the second image, that the first image does not include a representation of an article and that the second image includes a representation of an article, the method further comprises the step of:

identify said article at least in part based on the second image from the article identities currently included in the dynamic list of articles of the registration account (130), and

25. The method according to any one of claims 20-24, further comprising determining a current position of said article carrying arrangement (5).

26. An automatic in-store registration system (100) in a store having at least one article containing area (A-I), the system comprising:

at least one processor (120); at least one registration account (130) being associated with an article carrying arrangement (5); and

a sensor arrangement (110) comprising a plurality of sensors (112), the sensor arrangement (110) being configured to continuously monitor the article carrying arrangement (5);

wherein the processor (120) is configured to:

receive, from the sensor arrangement (110), a first image of at least a portion of a customer’s upper extremity being captured when at least a part of the customer’s upper extremity is moving across a virtual boundary set at a predetermined distance from the article carrying arrangement (5);

determine, based at least in part on the first image, if the first image includes a representation of an article, and if so, identify said article; and

wherein the processor (120) further is configured to add said final article to the registration account (130).

27. A method for training an in-store registration system (100) configured to identify and register articles in a store, said method comprising performing a training sequence in order to improve the accuracy of the identification of articles by:

defining a virtual boundary set at a predetermined distance from the article carrying arrangement (5);

using training data to recognize different movements of a user’s upper extremity while being in or near the virtual boundary set, and to detect at least a portion of a upper extremity of a person when at least a part of the person’s upper extremity is moving across the virtual boundary set;

adding or updating information relating to signature and the position of each article to the in-store registration system (100).

28. The training method according to claim 27, wherein the training data comprises of a synthetically generated 3D-model of at least a part of the store and/or the customer. 29. The training method according to 27 or 28, wherein the synthetic data is enhanced using Generative Adversarial Networks (GAN).

30. A method for registration of an article in a store having at least one article containing area (A-I) and an associated automatic in-store registration system (100), wherein the method comprises:

associate a customer with a registration account (130);

monitor the movement of the customer through the store,

continuously monitor the at least one article containing area (A-I) by means of a sensor arrangement (110), and detect a change in the at least one article containing area (A-I) and, in response to said detected change, identify one or more possible articles causing the change in a first identification step and save said one or more possible articles to a temporary article list, wherein, if said temporary article list includes two or more articles, positively identifying the article which caused the change in a second identification step from the articles currently included in the temporary article list by receiving a first image from the sensor arrangement (110) of at least a portion of a user’s upper extremity being captured after at least a part of the user’s upper extremity is removed from the article containing area (A-I), wherein the first image includes a representation of an article; and

add the identity of the article which was positively identified in the first or second identification to the registration account (130).

31. The method according to claim 30, wherein the method further comprises the step of receiving a second image from the sensor arrangement (110) of at least a portion of a user’s upper extremity being captured when at least a part of the user’s upper extremity is inside the article containing area (A-I), wherein the first and second images are used to identifying the article in the second identification step.

32. The method according to claim 30 or 31, wherein the second identification step further comprises accessing the article identities saved in the temporary article list and identifying the article as one of the accessed article identities originally positioned within a predetermined distance from the customer and added to the temporary article list within a predetermined time.

33. The method according to any of claims 30 to 32, further comprising identifying a plurality of customers, and associating two or more customers to the same registration account (130).

34. The method according to any one of claims 30 to 33, wherein the sensor arrangement (110) comprises at least one sensor (112) being selected from the group comprising a 2D-camera, a 3D-camera, an IR camera, a scale, a LIDAR sensor, a capacitive sensor and a spectrometer.

35. The method according to any one of claims 30 to 33, wherein the first identification step is performed by identifying an article by at least using information relating to the position of the change in the article containing area (A-I), wherein the position of each one of the plurality of articles arranged in the article containing area (A-I) is predetermined and accessible for the automatic in-store registration system (100).

36. The method according to claim 35, wherein the first identification step is performed by reducing the total number of possible article identities using the position information of the change in the article containing area (A-I).

37. The method according to claim 35 or 36, wherein the article containing area (A-I) is arranged with a plurality of segments, wherein the position of each segment is known.

38. The method according to claim 37, wherein each segment of the article containing area comprises at least one capacitive sensor arranged to detect a change in the article containing area (A-I). 39. The method according to any one of claims 30 to 38, wherein step of associating the customer with the registration account (130) is performed using Near Field Communication.

40. An automatic in-store registration system (100) in a store having at least one article containing area (A-I), the system comprising:

at least one processor (120);

at least one registration account (130) being associated with a customer in the associated store; and

a sensor arrangement (110) comprising a plurality of sensors (112), the sensor arrangement (110) being configured to, in a first identification step, continuously monitor the article containing area (A-I) and to detect a change in the article containing area (A-I) and, in response to said detected change, to transmit a first identification signal comprising information relating to one or more possible articles to the processor (120), wherein the sensor arrangement (110) further is configured to continuously monitor the customer; and

wherein the processor (120) is configured to identify one or more articles from said first identification signal and to save the identified one or more articles to a temporary article list, wherein if the temporary article list includes two or more articles, the processor (120) is configured to positively identify the article which caused the change in a second identification step from the articles currently included in the temporary article list by receiving a first image from the sensor arrangement (110) of at least a portion of a user’s upper extremity being captured after at least a part of the user’s upper extremity is removed from the article containing area (A-I), wherein the first image includes a representation of an article, and to add the article being positively identified in the first or second identification step to the registration account (130).