GB2567446A

GB2567446A - Checkout scanning system

Info

Publication number: GB2567446A
Application number: GB1716657.0A
Authority: GB
Inventors: Hassan Waqas
Original assignee: Facit Data Systems Ltd
Current assignee: Facit Data Systems Ltd
Priority date: 2017-10-11
Filing date: 2017-10-11
Publication date: 2019-04-17
Anticipated expiration: 2037-10-11
Also published as: GB201716657D0; GB2567446B

Abstract

A system for detecting the presence of retail products in an open product carrier includes a detector that senses the presence of a product carrier, a camera that captures an image of the product carrier, and an image analyser that analyses the captured image to detect the presence of retail products in the product carrier. The image analyser is configured to identify the presence and location of the product carrier, convert the captured image to the YUV colour space, measure the standard deviation (STD) of each pixel in the V channel, extract the maximum and minimum STD values for each pixel, generate a first binary image based on a matrix of the maximum and minimum STD values, apply a colour segmentation process to the U and V channels to identify bright red and bright green colours in the captured image, generate a second binary image based on a matrix of the colour segmentation values, create a third binary image based on the summed matrix values of the summed matrix values of the first and second digital images, sum the matrix values in the third binary image, compare the summed matrix values with a threshold, and generate an alert signal if the summed matrix values exceed the threshold.

Description

10 18

SYSTEM FOR DETECTING RETAIL PRODUCTS

INTRODUCTION

Scan avoidance (i.e. failure by customers to scan and pay for purchased goods) is huge problem currently faced by retailers around the globe. A number of solutions are currently available in the market (hardware and software) that focus on scan avoidance on self-checkouts. It’s a common perception that since customers themselves use self-checkouts, they are more prone to scan avoidance incidences.

However, scan avoidance is also a problem with cashier-operated checkouts, where customers may accidentally or deliberately leave items in a trolley, with the result that they are not scanned by the cashier or paid for. The present invention is concerned with scan avoidance detection in cashieroperated checkouts, in particular belted checkouts. Customers with trolleys can leave items in the trolleys. These items are usually not visible to the cashier using the checkout. The solution uses a vision-based solution to first of all detect a trolley and then detect any items that might be left in the trolley.

The proposed solution is divided into two parts

1. Detecting and locating the Trolley

2. Finding the products in the Trolley

See Fig. 1

1. Detection and Location of Trolleys

A metal detector is used to detect a trolley and trigger the image capture, for example by means of a camera located on the checkout. The image is then fed into the Trolley Detection module to find the exact location of the trolley in the captured image.

In Trolley Detection module, first we smooth the image for removing the noise. For smoothing the image, we pass the image through different filters. After removing the noise, we find the edges of all the objects present inside the image.

After removing the noise and detecting the object in the image, we have to separate trolley from other objects. A trolley is composed of specific straight line patterns, which are connected together to form a network of straight lines intersecting each other. So, for this case we connect

10 18 the edges of each object and identify whether it represent a straight line or not. Lines other than straight lines are ignored. In the end we get a 2 dimensional straight-line image, compose of only edges of those objects that corresponds to straight lines like that of a trolley.

In the next step, we compute a count vector. A count vector is 1 dimensional matrix, whose height is equal to 1 and width is equal to the width of the 2 dimensional straight-line image. Each index value of count vector represents the number of lines passing through each vertical axis or column of the 2 dimensional straight-line image. To separate irrelevant objects in the count vector, we pass the count vector through a threshold value a, values less than threshold are set to zero. If all the values are less than threshold then the image is considered to be without presence of any trolley and it is ignored. All other images are considered to have trolley present in them and are passed through the next steps.

For detecting the location of the trolley in the image, we plot the final count vector. The plotted count vector is considered as a histogram (see Fig. 2), where the y-axis represents the frequency of straight lines crossing that area and the x-axis represents the number of indexes or bins of each count vector.

Bins of Count Vector

Area covering the maximum peaks of the histogram is considered to be the location of the trolley. As the bins of the x-axis are equal to the width of the image, the location of the peaks in the histogram corresponds to the location of the trolley in the image.

2. Finding Products in the Trolley

For Finding the Products inside a trolley, the captured image is first converted from RGB to YUV colour space. Where Y channel of the YUV colour space represent the luminance or brightness of the image and U and V represent the colour itself. We split the YUV colour into separate colour channels. After splitting into separate channels, following two methods are applied

1. Standard Deviation Method

2. UV Channels Method.

10 18

Standard Deviation Method:

In this technique, we take a kernel matrix of size 9x9 pixel, an example is given in Fig. 3.

We slide the kernel over the V channel of the YUV colour space, in such a way that each pixel of the V channel comes under the central pixel ‘C’ of kernel matrix. The value of the pixel is replaced with the standard deviation of all the pixels of V channel under the kernel. We keep on sliding the kernel and replace the pixel values with the standard deviation value. In the end we get a resultant standard deviation matrix. This resultant matrix is then passed through a mixture of filters function, which extracts the maximum and minimum range values from the standard derivation matrix to generate a binary image. These maximum and minimum range values represents the bright and dark colour product values.

UV Channels Method:

In this technique, we pass the U and V channels through a red and green colour range modulator module, which increases the pixel value of the object having the bright red and bright green coloured objects and convert the resultant matrix into binary image.

We add the resultant matrix of both techniques index by index and if the resultant index value is greater than a threshold β, it is considered to indicate the presence of a product (marked A in Fig. 4), and if the resultant index value is less than a threshold β it is considered to indicate the absence of a product: i.e. those parts of the image are considered to be background (marked B in Fig. 4).

In the end, the overall sum of resultant matrix is greater than a threshold γ this is taken as an indication that the trolley contains products, an alert is generted and an image is popped up on the display. This allows the cashier at the checkout to confirm the presence of unscanned items in the trolley.

Claims

1. A system for detecting the presence of retail products in an open product carrier, the system including a detector that senses the presence of a product carrier, a camera that captures an image of the product carrier, and an image analyser that analyses the captured image to detect the presence of retail products in the product carrier, wherein the image analyser is configured to:

identify the presence and location of the product carrier, convert the captured image to the YUV colour space comprising Y, U and V channels, measure the standard deviation (STD) of each pixel in the V channel, extract the maximum and minimum STD values for each pixel, and generate a first binary image based on a matrix of the maximum and minimum STD values, apply a colour segmentation process to the U and V channels to identify bright red and bright green colours in the captured image, and generate a second binary image based on a matrix of the colour segmentation values, create a third binary image based on the summed matrix values of the first and second digital images, and sum the matrix values in the third binary image, compare the summed matrix values with a threshold, and generate an alert signal if the summed matrix values exceed the threshold.