EP4115370A1

EP4115370A1 - Creating and updating a product database

Info

Publication number: EP4115370A1
Application number: EP21706897.2A
Authority: EP
Inventors: Philipp Kleinlein; Philip Koene; Frank Schaefer
Original assignee: BSH Hausgeraete GmbH
Current assignee: BSH Hausgeraete GmbH
Priority date: 2020-03-02
Filing date: 2021-02-18
Publication date: 2023-01-11
Also published as: DE102020202657A1; WO2021175601A1

Abstract

A device comprises an optical scanning unit for providing optical scanning of an article; a processing unit and an interface to a cash register system. The cash register system is designed to provide product information relative to the article, and the processing unit is designed to associate the product information with an optical scan.

Description

Creation and updating of a product database

The invention relates to the provision of a product database, in particular for recognizing a household item.

In an intelligent household, an inventory of existing consumables can be kept automatically. For example, a supply of vacuum cleaner bags can be replenished in good time without having to keep large stocks and without the risk of not having any bags available when needed. A best-before date can be tracked for perishable goods, especially food. If the goods threaten to spoil, a notice on how to use the goods can be given in good time. Used or expired goods can be placed on a shopping list or ordered automatically.

It has proven difficult to fully inventory goods entering or leaving a household. Existing solutions often require the scanning of a bar code or other marking that is attached to the goods. A user can find this to be time-consuming, so that he carries out the scanning less precisely or not at all. The inventory can then quickly become unusable. In addition, this approach requires constant updates to the information indicated by the barcode. For example, if a manufacturer changes a packaging quantity of a product, the changed quantity cannot be available in the barcode itself, but only in assigned information. Keeping product information from a large number of manufacturers constantly up-to-date is also time-consuming, so that errors can easily arise, for example due to outdated information.

In addition, refrigerators with a camera located therein are known from the prior art, which can detect the objects located in the refrigerator by means of a neural network. However, the probability of correct object recognition depends heavily on the quality of the training of the neural network.

An object on which the present invention is based is to provide an improved technique for creating or maintaining a product database with goods that can be procured especially for a household. The invention solves this task by means of the subjects of the independent claims. Subclaims reflect preferred embodiments.

According to a first aspect of the present invention, an apparatus comprises an optical scanning device for providing optical scanning of goods, a cash register system with preferably a scanning device for providing product information about the goods, and a processing device for assigning product information to optical scanning. The scanning device can for example comprise an optical camera. By assigning the product information to the optical scanning, so-called labeled training data can be provided for an artificial neural network, the training data each comprising a scanning and product information. The device is preferably set up to transmit the training data to an external point.

The device can be used, for example, in a supermarket or another retailer as part of a billing or payment process, which can also be called “checkout”. Usually, all goods collected by a customer in the shop are recorded (e.g. scanned) by means of the cash register system, whereby the quantities and prices of the goods are also determined. It is proposed to link an optical scan of a product with information provided by the checkout system. The information can in particular include a designation or a unique code such as an EAN code for the goods. By means of the Vorrich device, a large number of optical scans can be created and clearly assigned to the goods in a normal operation of the cash register system. Optionally, information from several cash register systems can be evaluated. A relationship between an optically detectable property of the goods and the goods themselves can be determined automatically and with a high degree of reliability.

As mentioned, the processing device is set up to provide training data for an artificial neural network, the training data each comprising a scan and product information. Such data are also called “labeled”, whereby the product information can represent the “label”, that is, an indication of the identity of the item presented. The training data can be used for supervised learning of the neural network. Especially when trai- When data is collected from many of the devices described herein, a necessary number of labeled training data for learning to recognize a product can be generated efficiently and in a timely manner.

The scanning device can be attached, for example, in the area of a scanner, a scale or a manual detection station of the cash register system. For example, in a retail trade, the optical scanning can be installed in the area of a shop counter or a conveyor belt for goods to be purchased. It is preferred that the goods do not have a predetermined preferred orientation during scanning, so that a statically attached scanning device can scan the goods from a random perspective. A neural network trained with a large number of such scans can then recognize the goods from all practically occurring perspectives.

The scanning device can comprise an optical camera. It is preferred that the camera provides a colored, high-resolution and high-contrast image. Active lighting can be provided for this purpose, optionally in a predetermined wavelength range. The camera can support a short exposure time in order to reduce motion blur in the scan.

The scanning device can be set up for three-dimensional scanning of the goods. A device for three-dimensional scanning of the goods can be used in addition or as an alternative to an optical camera. In one embodiment, a plurality of optical cameras are used which are mounted at locations that are slightly offset from one another and whose scanning fields overlap one another. Such a camera can also be called a stereo camera. In another embodiment, a ToF camera can be used in which a distance of a point on the surface of the goods is determined on the basis of a travel time of light from a light source in the region of the camera to the goods and from there to the camera. Such a camera can also be referred to as a depth camera.

The scanning device can comprise a laser scanner. In particular, a Li-DAR sensor can be used to determine a three-dimensional extension of the goods. Such sensors are now available at moderate costs and can allow goods to be scanned quickly and accurately. According to a second aspect of the invention, a system comprises a device described herein for scanning the goods; and an external location configured to receive scans and associated product information from a plurality of devices. The external location is preferably set up to receive information from a large number of the devices described. More preferably, a dedicated processing device is provided on the part of the external point, for example in order to process received information, to store it in a data memory or to process stored information. In particular, the external point can be set up to train an artificial neural network to recognize goods on the basis of the received scans and the associated product information.

In one embodiment, the external location comprises a data memory for recording a multiplicity of scans of the goods and associated product information in each case. The information collected can be compared or merged with one another. Deviating provisions can be excluded. Particularly when the goods are mass-produced, information that is determined at a large number of sales points can be quickly and reliably updated, for example a change in a presentation, a packaging or container size or another aspect of the goods.

In one embodiment, the processing device of the external location is set up to train an artificial neural network for recognizing the goods on the basis of an optical scan of the goods. The provided neural network can be used, for example, in a household in order to recognize goods on the basis of optical scanning, as will be explained in more detail below.

According to a third aspect of the present invention, a method comprises the following steps: providing an optical scanning of a product; Acquiring product information of the goods; and assigning the product information to the optical scanning. The optical scanning can in particular take place with the aid of an optical camera. The product information can in particular be provided by a cash register system, for example as part of an invoice or payment process for the goods. The cash register system can use a scanning device. The method can be used to provide data sets that are used to train a neural network can be used. The training data can each include a sample and product information. The training data can be transferred to an external location. The method can preferably be carried out in the area of a cash register system, in particular a supermarket or retailer. Further processing of the collected data is preferably carried out at an external point. The external point can be provided centrally for a large number of points at which the method can be carried out.

The method can be carried out in whole or in part by means of a processing device which is comprised by a device described herein. For this purpose, the processing device can comprise a programmable microcomputer or microcontroller and the method can be in the form of a computer program product with program code means. The computer program product can also be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to the device or the system or vice versa.

According to yet another aspect of the present invention, an apparatus comprises a processing device, a communication device and an optical scanning device. The communication device is set up to receive an artificial neural network from the external location, and the processing device is set up to recognize goods scanned by the scanning device by means of the neural network. The neural network usually comprises a computerized data structure which is relatively compact and can usually be recorded in a few 100 kBytes. For object recognition, the neural network can be implemented on the processing device, or the processing device can implement the neural network with the data structure.

With the aid of the above disclosure, for example, the following scenario can be realized: Many users buy goods in many supermarkets. These goods are scanned on standard cash register systems in order to enable them to be paid for. During this scanning process, a barcode attached to the goods is usually scanned, which uniquely identifies the goods. The goods can be photographed with the help of a camera located near the cash register system. The photos can be labeled using the barcode scanned by the cash register system. The data obtained in this way can be transferred to an external location, e.g. an Internet server or cloud Service, where they are used to train a neural network. The neural network trained in this way can be downloaded by the users to their refrigeration devices, where it is used to identify the goods in the refrigeration device. The labeled training data can be used location-based to train the neural network. For example, it is conceivable that a neural network is trained only with labeled training data from Bavaria (or another limited area) and that this neural network is only downloaded to cooling devices in Bavaria (or the other limited area). In this way it can be ensured that the training level of the neural network is optimally matched to the locally offered goods, whereby the error rate in object recognition can be significantly reduced.

The invention will now be described in more detail with reference to the embodiments shown in the accompanying figures, in which:

FIG. 1 shows a system with a first device;

FIG. 2 shows a flow chart of a method; and Figure 3 shows a further system with a second device.

FIG. 1 shows a system 100 with a first device 105 and a second device 110. The devices 105 and 110 can be designed to be integrated with one another.

In the illustrated embodiment, however, devices 105 and 110 are physically separate from one another, but communicatively connected to one another, as detailed below. The second device 110 can be designed as a central component which is set up to process information from a plurality of first devices 105. The second device 110 can then also be referred to as an external location.

The first device 105 is preferably attached in the area of a cash register system 115, which can be used in particular in retail trade in order to determine a purchase price for a number of goods 120 selected by a customer. The goods 120 can be intended in particular for use in a household, in particular a private household, and include, for example, food, drugstore items, animal feed, printed matter or household appliances. Goods 120 shown as an example include sen a poultry, a detergent and a vegetable. Other goods 120 are also possible and in principle the present invention can also be used in the intermediate or wholesale trade.

The checkout system 115 usually comprises a processing device 120, which can be connected to a first scanning device 125, for example a scanning device, an input device 130 and / or a scale 135. Information about a recorded product 120 or a single or total price can be provided by means of an output device 140. The output device 140 can comprise a visual display and / or a printer. Product information about a product 120 can be provided via an interface 142, in particular an identification of the product 120. The identification can include an output that can be made available on the output device 140. The identification is preferably directed to a human user and includes a naturally linguistic designation.

The goods 120 are usually detected by scanning a label attached to it, for example a barcode or barcode, by means of the first scanning device 125. Loose goods 120 can be identified and counted by a person or weighed using the scale 135. A certain quantity of the goods 120 can be entered by means of the input device 130.

The first device 105 includes at least one of a second scanner 145 and a third scanner 150; as well as a processing device 155 and an optional communication device 160 for communication with the second device 110 if this is set up separately from the first device 105.

The second scanning device 145 preferably comprises an optical camera for providing a still or moving image of a product 120 that is processed in the checkout system 115. For this purpose, the scanning device 145 can comprise one or more lighting devices that can emit light of predetermined wavelength ranges. Both visible and invisible wavelength ranges can be used for this.

The third scanning device 150 is preferably set up to determine geometric dimensions of a product 120 and preferably also works optically. At- For example, the third scanning device 150 can comprise a LiDAR sensor, a depth camera, a stereo camera or a ToF camera. In one embodiment, both scanning devices 145, 150 are designed to be integrated with one another. Each of the scanning devices 145, 150 can also be provided multiple times in the area of the cash register system 115, preferably with a variation of a perspective of a product 120.

It should be noted that the scanning devices 145, 150 are set up less to detect an identification of the goods 120 - for example a barcode - than for an optically recognizable property, for example a color, a design, a size, an aspect ratio or a Labeling. The purpose of the scanning is preferably to determine reproducibly detectable optical features of the goods 120. Goods 120 with great optical variation, such as loose goods, fruit, vegetables or fresh meat, can have individually different optical features, which can nevertheless be used to identify them.

The scanning devices 145, 150 are preferably mounted in the area of the checkout system 115 in such a way that their scans can be assigned as clearly as possible to other scans, inputs or determinations of the cash register system 115, so that it is ensured that the scans of the devices 145, 150 relate to the same Wa re 120 like a product information provided by the cash register system 115. As a result, the scanning devices 125, 145, 150 can be spatially close to one another. If the goods 120 are always aligned in the same way for scanning by means of the first scanning device 125, for example so that a barcode is always located on the underside, then it is preferred that the scanning devices 145, 150 are positioned in such a way that they not yet align the goods 120 feel, or the goods 120 after a reorientation that is as random as possible, for example at the end of an inclined plane.

The first device 105 and the second device 110 each include a communication device 165 for mutual data transmission. The second device 110 further comprises a processing device 165 and preferably a data memory 170. It is proposed to assign a scan of a product 120 by means of one of the scan devices 145, 150 to product information provided by the checkout system 115. A data record can preferably be created in which the product information includes a label of an optical scan. Generated data sets can be used to train an artificial neural network to create a to enable identification of the goods 120 by means of the network. In one embodiment, a trained neural network is trained and / or provided by the second device 110.

FIG. 2 shows a flow chart of a method 200 which can be carried out in particular by means of one or both devices 105, 110 of a system 100. It should be noted that the method 200 can comprise a plurality of sub-methods, one of which can also be carried out by means of another system, as is described in more detail below with reference to FIG. 3.

In a step 205, goods 120 that are received in the area of a cash register system 115, in particular for billing purposes, can be optically scanned. In a parallel step 210, product information of the goods 120 provided by the checkout system 115 can be recorded. It is preferably ensured that steps 205 and 210 are based on the same goods 120.

In a step 215, a data record can be created in which the product information is assigned to the scanning that has taken place. The product information can be specified in the form of a label. In addition, a boundary, for example as a “bounding box” of the depicted goods 120, can be inserted on the scanning. The limitation can easily be determined if the scanning device 145, 150 is attached statically, so that an image part that does not belong to the goods 120 is known.

In a step 220, a repository of specific data records can be created. The repository can include a large number of specific data records that can be sorted according to certain criteria. Exemplary ordering criteria include an acquisition time, a determination time, a destination or a determination confidence. Outdated or incorrectly recognized data records can be removed from the repository. Data records from the repository can be made available for training a neural network.

In a step 225, an artificial neural network can be trained using previously determined data sets. The trained network can be set up to recognize a commodity 120 on the basis of its optical scanning. In particular, information associated with the goods 120 can be provided, for example its designation. In In one embodiment, further information is assigned to the goods 120, for example an ingredient, a place of purchase, a purchase price, a quantity or an estimated or specified expiry date. This information can then also be provided.

The neural network can be used in a different context than one of the devices 105, 110. For example, the network can be used in a household to identify goods 120 that have been procured. In particular, the goods 120 can be optically scanned in a step 230, with graphical features and / or dimensions being able to be recorded. In a step 235, the goods 120 can be recognized. Associated information can be provided. An inventory of goods 120 present in the household can be updated to include goods 120. In a further embodiment, a used or withdrawn item 120 can be identified in the same way and an inventory can be updated accordingly.

FIG. 3 shows a system 300 with a third device 305, which is attached to a cooling device 310 by way of example. The refrigerator 310 may include, for example, a refrigerator, a freezer, a combination of both, a wine cooler, or an uncooled pantry. A product 120 is arranged in the cooling device 310 by way of example.

The third device 305 comprises a processing device 315 and an optical scanning device 320 preferably directed onto an interior of the cooling device 310. A communication device 325 is optionally provided. By means of the communication device 325, a data structure can be obtained from an external point, for example the second device 110, which represents an artificial neural network. The neural network can be implemented by executing the data structure by the processing device 315.

The scanning device 320 may include one or more cameras. The scanning device 320 is optionally set up to determine a geometric dimension of a product 120. A light source for illuminating the interior of the cooling device 310 is also optionally provided. In one embodiment, the scanning device 320 is adjustably attached to the cooling device 310 in order to determine optical scans from different perspectives. The third device 305 can recognize which goods 120 are in the refrigerator 310 by performing an optical scan of the interior using the scanning device 320 and identifying one or more goods 120 recognizable on the scan by means of the neural network. The device 305 can then provide an inventory of the cooling device 310. For example, it can be determined in an automated manner whether or not certain quantities of predetermined goods 120 are present in front of them. In one embodiment, for example, it can be determined which ingredients are available for the preparation of a predetermined recipe, or which quantities of which ingredients have to be procured in order to convert the recipe into a meal of a predetermined quantity.

Reference number

100 system

105 first device

110 second device, external location, central component

115 POS system

120 goods

125 first scanning device

130 input device

135 scales

140 Output device

142 interface

145 second scanning device

150 third scanning device

155 processing facility

160 communication facility

165 processing facility

170 data storage

200 procedures

205 Scanning goods

210 Entering product information

215 Creating a data record

220 Creating a repository

225 training neural network

230 Scanning goods

235 Recognize goods

305 third device

310 refrigerator

315 processing facility

Claims

PATENT CLAIMS

A device (105) comprising: an optical scanning device (145, 150) for providing an optical scanning of a product (120), the scanning device comprising an optical camera (145, 150); a checkout system (115) with a scanning device (125) for providing product information on the goods (120); and a processing device (155) for associating the product information with the optical sampling so as to provide training data for an artificial neural network, the training data each comprising a sampling and product information; wherein the device (105) is set up to transmit the training data to an external location (110).

2. Device (105) according to one of the preceding claims, wherein the scanning device (145) is set up for three-dimensional scanning.

3. Device (105) according to one of the preceding claims, wherein the scanning device from (145, 150) comprises a laser scanner.

4. System (100), comprising a device (105) for scanning the goods (120) according to one of the preceding claims; and an external location (110) which is set up to receive scans and respectively associated product information from a plurality of devices (105) in order to train an artificial neural network to recognize goods.

5. System (100) according to claim 4, wherein the external location (110) further comprises a data memory (170) for receiving a plurality of scans of the goods (120) and each associated product information.

6. Method (200) with the following steps:

Providing (205) an optical scan of a product (120) with the aid of an optical camera (145, 150); Acquisition (210) of product information on the goods (120), preferably with the aid of a scanning device (125) of a cash register system (115);

Associating (215) the product information with the optical sampling so as to provide training data for an artificial neural network, the training data each comprising a sampling and product information; and

Transmission of the training data to an external location (110).

7. The device (305), comprising a processing device (315), a communication device (325) and an optical scanning device (320), the communication device (325) being set up to send an artificial neural network from an external point (110) receive; and the processing device (315) is set up to recognize goods (120) scanned by the scanning device (320) by means of the neural network.

8. Cooling device (310) with a device according to claim 7.