WO2017215362A1 - Intelligent shopping system based on supermarket shopping cart and control method therefor - Google Patents

Abstract

Disclosed are an intelligent shopping system based on a supermarket shopping cart and a control method therefor. The complete shopping process of a customer is divided into six states; and in each state, an anti-theft function of the intelligent shopping cart system in the shopping process is realised by utilising, to different degrees and in combination with a plurality of image analysis algorithms, weight change information fed back by a weight sensor in real time, image information about each frame collected by a monitor apparatus, and commodity information acquired by a code scanner. By performing grey scale image statistics and feature analysis or pattern recognition on a detected moving target image or a skin colour detection image, the detection of an empty hand or a hand holding a commodity within a monitoring area is realised, the customer is timely reminded about rule-breaking operations and same are corrected, and judgement on a rule-breaking behaviour of the customer is realised. Payment free from queuing, labour cost reduction and improvement of shopping experience are realised while guarding against theft and stopping loss.

Description

Intelligent shopping system based on supermarket shopping cart and control method thereof Technical field

The invention relates to the field of intelligent shopping control, in particular to a smart shopping system based on a supermarket shopping cart and a control method thereof.

Background technique

Although online shopping is very convenient today, supermarket shopping is still an important shopping route and lifestyle. Especially in the first-tier cities or holidays, the phenomenon of queuing and paying is prominent, which makes the original fast shopping method long after waiting in line for checkout. The shopping experience reduces the consumer's desire to consume, but increases the operating costs of the supermarket. Therefore, research and development of a smart shopping system that integrates self-scanning code checkout and real-time monitoring of theft, has broad market prospects and significant commercial value.

There have been many similar studies at home and abroad. Wal-Mart first developed RFID technology in the retail industry, and developed an RFID-based smart shopping cart and tried it in some Wal-Mart supermarkets. Then Dalian University of Technology and Shanghai Jiaotong University studied RFID-based shopping. Intelligent terminal, the research uses RFID tags to replace multi-dimensional barcodes, which not only contains various information of commodities, but also has good anti-counterfeiting technology, encryption technology and information control. However, RFID tags can not replace ordinary two-dimensional for a long time. Code, so now only Wal-Mart is very popular in some supermarkets. In addition, Central South University, School of Information Science and Engineering, University of Electronic Science and Technology, etc. have studied intelligent shopping carts based on embedded systems. This solution combines ordinary barcodes and intelligent operating systems of embedded operating systems to achieve indoor navigation, barcode scanning, Manage shopping lists, generate payment codes, etc., but the program does not involve the anti-theft feature of the shopping cart, so it cannot be widely used. However, Liu Bing and others of Northeast Forestry University used the electronic product anti-theft system, ie EAS system, to burglar the shopping cart products. Using the soft label of each product, the EAS system can be used to decode the soft label after the product is settled, thus realizing the anti-theft. This is similar to RFID tags. It requires different soft tags for each product. It has a large workload and cannot be widely used. At present, the anti-theft measures of supermarkets are basically on the electronic gate of the supermarket door, including the patent CN201520043135.6 and the patent CN201510559549.9. The logical judgment of this method is too simple, and the operation of stealing goods during the shopping process cannot be detected.

At present, there are tens of thousands of types of large-scale shopping supermarkets in China. With the expansion of supermarkets and the diversification of consumers' shopping choices, they will encounter such things as being unable to quickly find the products they want to buy in supermarkets, and different brands for similar products. I don’t know enough about the product information, I don’t know the quantity and price of the items purchased in the smart shopping cart, and the queues that are long queued at the cashier’s desk during the peak shopping period are waiting to be settled. The existing smart shopping cart RFID reads nearby goods, resulting in dozens or dozens of catalogues on the nearby shelves on the screen, which itself changes the customer's shopping habits to the user, resulting in a high selection error rate in the selection of subjects. Because of the speed of response to reading dozens of commodities at a time, it is bound to slow down. And when encountering fresh weighing and other commodities, the price is uncertain. For example, if you attach an RFID tag to each weighing product, it will bring huge cost increase. Can solve this kind of real problem. The problem of anti-theft can only be solved by pressure measurement and control. There are many goods in the supermarket with the same weight, but the price is not the same. For example, many products have the same specifications, but the price difference is a few yuan or even hundreds of yuan. Only relying on a high rate of weight error to bring some unscrupulous customers to the opportunity of Zhang Guan Li Dai, bringing losses to the supermarket.

One of the main functions of the supermarket smart shopping cart is to be able to identify the customer's hand and whether there is goods on the hand. The existing skin color detection method is mainly based on the face detection method, in order to ensure that the face part is as much as possible. Included in the metering area, the size of the face frame must be increased, so that the background and hair data are more or less included in the face metering area, because the brightness information of the background changes under different scenes and illumination conditions. Large, and combined with the effects of hair and eye color, the face-based metering is very unstable, which greatly affects the metering effect.

Summary of the invention

The technical problem to be solved by the present invention is to provide a smart shopping system based on a supermarket shopping cart and a control method thereof for the deficiencies of the background technology, and utilize the barcode data of the products collected during the shopping process, the weight change data in the shopping basket, and the shopping cart. The image analysis data in the central area realizes the anti-theft function of the shopping cart, and improves the user experience of the self-service settlement of the customer. The specific judgment step is provided to prevent the user from accidentally operating or stealing the goods, and the merchant is guaranteed. Interests.

The present invention adopts the following technical solutions to solve the above technical problems.

A smart shopping system based on a supermarket shopping cart, comprising a controller installed between the handle of the shopping cart, a scanner, a load cell placed at the bottom of the basket, and a camera at the upper end of the basket, the scanner Located on the lower side of the controller, opposite to the shopping cart basket mouth, the scanner, the weighing sensor and the camera are respectively connected with the controller, wherein two planes are sequentially set below the frame of the basket The H1 plane and the H3 plane are respectively placed, wherein the load cell is placed at the bottom of the basket below the H3 plane, the scanning port of the scanner overlaps with the H1 plane, and the camera is placed above the H1 plane and monitors the H1 plane and above. Moving objects in space.

A further preferred solution of the smart shopping system based on the supermarket shopping cart of the present invention further includes a hand segmentation calculation module for judging and obtaining a picture of the product that the hand may hold according to the trend of the hand movement;

The skin color calculation module, the opponent can not hold the picture of the product to do the skin color segmentation process;

Hand-held commodity judgment module to determine whether the hand actually holds the commodity;

The statistics module determines whether the user violates the rules by counting the statistics of the results of the handheld product judgment module.

A control method for a smart shopping system based on a supermarket shopping cart, comprising the following steps:

S1, the start controller, the scanner, the load cell and the camera start collecting data information; wherein each component operates according to the following state:

A0 state: It is judged by the camera whether there is a hand entering the monitoring area;

A1 status: It is judged by the scanner and the camera whether there is a product scan operation and the action of putting the product into the bottom of the shopping basket below the H3 plane;

A2 status: It is judged by the camera whether there is an empty hand in the monitoring area;

A3 state: the product is weighed by the load cell and the result is fed back to the controller for judgment;

A4 state: Waiting for the weighing sensor to weigh the goods, the result is stable and prompt the user's hand not to enter the monitoring area during this period;

A5 status: compare the product weight information obtained by the scanner with the result obtained by weighing, and if the two match, the shopping is completed;

B0 state: the user operates the controller to enter the return procedure, and the camera monitors whether the user enters the shopping cart H1 plane empty hand;

B1 state: the camera monitors whether the user holds the item from below the H1 plane into the monitoring area, and whether the user scans the code through the commodity scanner;

B2 status: The camera monitors whether the user takes the item out of the monitoring area after scanning the code;

B3 state: If the user does not scan the code and takes out the item out of the monitoring area, the controller waits for the load cell to decrease the value and queries whether it is the same as the theoretical weight of any product purchased;

B4 state: After the user scans the code and takes the item out of the monitoring area, the controller waits for the weighing sensor to reduce the value and queries whether it is the same as the theoretical weight of the scanned product;

B5 status: Whether the actual product weight information is compared with the actual detected value of the B3 or B4 status.

When the weight of the weighing sensor is constant, the customer does not put the product into the shopping basket or operates empty hand. If the product has been scanned, the controller will prompt the scanning code to be successful, and proceeds to step S4 to detect whether there is a violation operation; If the code is not scanned, the camera collects the image information in the shopping basket and transmits it to the controller for image judgment to determine whether the empty hand is still a hand-held object; if the object is hand-held, the process proceeds to step S2, and if the hand is empty, the action is ignored;

When the weighing weight of the weighing sensor increases, it indicates that the product has not been scanned or the scanning code is not successfully placed directly into the shopping basket, and then prompts to take out the rescanning code, and proceeds to step S6 to detect whether the taken product is just placed. Products of;

When the weight of the weighing sensor is reduced, it means that the purchase process has been completed, then the return code is scanned, and the process proceeds to step S3 to check whether there is a violation operation that does not meet the return;

S2, the step is after determining that there is an item in the hand;

If the weight of the weighing sensor does not change, the product does not enter the shopping basket, the controller prompts to scan the code, and when the scanning code is successful, the process proceeds to step S4;

If the weight of the weighing sensor is reduced, it means that the customer has the illegal operation of stealing the goods, and the controller will prompt the illegal operation to take the artificial channel;

If the weighing weight of the weighing sensor is increased, it means that the customer has placed a new product without scanning the code, and the controller will prompt the new product to be taken out, and proceed to step S6 to check whether the taken product is in conformity, so as to avoid stealing. commodity;

S3. When the weight reduction signal is detected in step S1, the hand is already in the area detected by the camera, and the judgment is made according to the detected position of the center of mass of the moving target;

When the center of mass is outside the center of the shopping basket, it is considered that the customer has taken out the product and has not scanned the code, and cannot determine which product the customer took out, so the illegal operation will be prompted, please take the artificial channel;

When the center of mass is in the center of the shopping basket, the real-time weight change information of the weight is collected; if the weight is unchanged, it means that the code needs to be scanned again during the normal return process, so the feedback code of the feedback determines whether the return is successful; If the code is not scanned, the controller will prompt the return, please scan the code; if the code has been scanned, the weight will be matched to determine whether the goods that are taken out meet the product of the scan code; if it meets the prompt, please take it out and proceed to step S5 to detect Take out the illegal operation in the process; if it does not, it means that the customer's goods are not correct, please prompt to take the manual channel; if the weight is increased, the weight matching will be carried out. If the matching does not match, the customer is considered to have placed other Goods, the controller prompts to take the artificial channel; if the match is met, the image is judged to prevent the customer from putting the same weight of the goods; if the weight is reduced, it is considered that the two items are returned at the same time, the controller will remind the customer The image is judged by performing gradation conversion on the color original image, then performing motion detection and determining the moving target centroid, when the centroid is in the shopping basket monitoring After the region, the color original image is detected by the skin color, the motion detection result and the skin color detection result are summed to obtain the complete moving target image, and then the complete moving target result is subtracted from the skin color result to obtain the skin color moving target; then the skin color motion is calculated. The number of pixels in the image with a pixel value of 255 is taken as the skin-removing area, and the information entropy of the complete moving target below the skin color centroid is calculated. If the area of the skin color detected by the continuous multi-frame is greater than the threshold A or the information entropy of the motion region below the skin color centroid is greater than The threshold B can be considered as a hand-held object in the shopping basket, and vice versa is considered to be an empty hand, and the background needs to be updated;

S4. After the step S1 prompts that the scan code is successful, the camera detects the centroid position of the moving target. If the centroid is detected outside the central area of the shopping basket, the customer is deemed to be returning, so the user is prompted to rescan the code to complete the return process;

The real-time weight change information fed back by the weight is divided into three cases; when the weight is constant, only the position of the centroid is judged;

When the weight is reduced, it is considered that the customer takes out the goods without scanning the code, reminding the customer to operate illegally, and needs to take the artificial passage;

When the weight is increased, the product has been placed in the shopping cart, and the weight is matched at this time; if it is met, the image judges whether the customer's hand holds the item, and if the item is held, it is considered that the object has two objects of equal weight but not equal price. The product of the scan code may be different from the product put in, so it is considered to be a violation of the operation and prompt to take the artificial channel; if it is empty, it is considered that the purchase is successful, it will prompt to continue shopping, and return to step S1; if not, Will remind you to take out new products, go to step S6, and check if there is any illegal operation during the process of taking out;

S5. This step is to check whether there is a violation operation when the product is taken out after the weight is matched in the returning process;

When the weight is increased or the weight is reduced, it is considered to be a violation operation;

When the weight is constant, according to the difference binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the commodity has exited the monitoring area, the return is over, and the process returns to step S1 to continue shopping;

S6. In this step, the weight mismatch is detected during the purchase process, and the product detected when the product is taken out is the same as the inserted product;

When the weight is increased, it is an unconventional shopping operation that the product is not taken out and continues to be placed in the new product, which will remind the customer;

When the weight is reduced, according to the differential binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the product has exited the monitoring area, and after the correct product is taken out, step S1 is continued to continue shopping. .

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, each state is specifically:

A0 state: When the tablet controller responsible for the algorithm starts, the electronic weighing scale, the commodity scanner and the camera on the shopping cart start working; according to the image data fed back by the camera, the image processing algorithm gives the current monitoring area whether there is motion. If there is no motion, the judgment ends, the next logical judgment is still A0 state, until the movement occurs in the monitoring area, the user is considered to start the series of operations in the purchase behavior, and it is judged whether the user has multi-hand operation; if there is any monitoring area Multiple hands, end with ERROR, otherwise, according to whether the user enters the monitoring process whether the item is held, the flag variable flag is assigned as one of the basis for subsequent judgment in the A1 state, and the current logic judgment is ended in the A1 state;

A1 status: In this state, the controller first determines whether the user completes the operation of scanning the product barcode; if the current scan code is performed, the user is considered to complete the product information entry by scanning the code, and the next action should be to put the scanned product in the hand. In the shopping cart, the logic judgment is ended in the A2 state; if the code has not been scanned at this time, according to the image processing algorithm, there are two cases in which the monitoring area has motion and the monitoring area has no motion;

When there is motion in the monitoring area, it is judged whether the motion is lower than the H3 plane, and the value of the flag variable flag obtained in the A0 state is combined to prevent the user from replacing the purchased item in the shopping cart by the unpurchased item before scanning the barcode; Below the H3 face, if the value of flag is false, it is considered that there is a potential risk of replacing theft. This logic judges to end with ERROR, waiting for the user to empty the shopping cart; the motion is lower than the H3 plane but the value of flag is true or the motion is higher than H3 plane, the logic judgment ends, the next judgment is still in the A1 state, with the A1 position as the starting position; if there is no motion in the monitoring area, it is considered that the movement into the monitoring area in the A0 state has ended, according to the movement The monitoring result determines the end point of this logical judgment. If it is found that there is a phenomenon that the hand-held item is out of the monitoring area during the monitoring process, it ends with ERROR, otherwise it ends normally, and the next time it judges to enter the A3 state;

A2 state: The user enters the A3 state after completing the scan operation. Firstly, it is judged whether there is motion in the current monitoring area; if there is motion in the monitoring area, it is considered that the user has not completed the action of putting the scanned product in the shopping cart and withdrawing the hand from the field of view; in order to prevent the user from completing the theft by the replacement operation, the provision is not Allow multiple hands to operate on the merchandise; if there is multi-hand operation, end with ERROR, if one-handed operation, end the current logic judgment in A2 state; no motion in the monitoring area, next Steps to determine that the monitoring process is a hand-held commodity phenomenon, the existence ends with ERROR; does not exist, enters the A4 state, and ends the logic judgment;

A3 state: the user enters the monitoring area, does not scan the code, and enters the A3 state after emptying the monitoring area; the A3 state completes the matching with the theoretical weight after waiting for the weight to stabilize, and does not allow the user to enter the monitoring area during the period; first judge the monitoring area The state of motion, the movement ends with ERROR; if there is no movement, the weight is determined according to the electronic weighing scale; the weight is unstable, and the logic judgment is still ended with A3; the weight is stable to match the current actual weight and the theoretical weight; The weight matching returns to the initial state A0, ending the current judgment; if it does not match, it ends with ERROR;

A4 status: After scanning the code, the user puts the product into the shopping cart and exits the monitoring area to enter the A4 state; the A4 state is completed waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period;

A5 status: After the user scans the product barcode, the algorithm needs to obtain information such as the theoretical weight of the product corresponding to the product barcode from the server through the network. The network will have a certain degree of delay, so the A5 state is introduced, waiting for the actual information corresponding to the product barcode; after the weight is stable, waiting for the actual information of the scanned product, according to the weight value and the theoretical weight value provided by the current electronic weighing scale, Match the two; if it matches, the current shopping process is completed, return to the initial state A0, prompting the user to "this shopping is completed"; if it does not match, it ends with ERROR; similarly, the A5 state requires the user not to enter the monitoring area. To avoid the risk of theft, if the surveillance area has exercise during this period, it ends with ERROR;

B0 state, the initial state of the return is B0. In this state, it is first determined whether there is motion in the current monitoring area; if there is no motion in the monitoring area, the user has not started the related operation of returning, and the current logical judgment is ended, and the state is still B0; There is motion in the monitoring area, and the user is considered to enter the monitoring area; if there is a phenomenon that the user enters the monitoring area with more hands, the image algorithm gives a "multi-hand operation" decision, ending with ERROR; for one-hand operation that conforms to the operating specification ", further determine whether the user holds the goods in the process of entering the monitoring area; there is a hand-held product entering the process, ending with ERROR; otherwise, entering the B1 state, ending this logical judgment;

B1 state: The function completed in the B1 state is to wait for the user to pick up the goods to be returned in the shopping cart and scan the code; when the logic judges to enter B1, first check whether the current user completes the scanning operation; if the code has been scanned, the user enters the B2 state. End this logical judgment; if the code is not scanned, it is judged whether there is motion in the monitoring area; the monitoring area has no motion, and the user's behavior in this series is considered to be empty-handed to enter the shopping cart monitoring area, and finally the return is not returned, and the monitoring area is exited to enter the B3 state. End this logical judgment; if there is motion in the monitoring area, continue to judge whether there is multi-hand operation. If there are multiple hand operations, it ends with ERROR; if it does not exist, the B1 state is kept to end this logical judgment;

B2 state: the user enters the B2 state after performing the return scan code; the function completed by the state is to wait for the user to take the scanned code product out of the monitoring area; firstly, it is judged whether the monitoring area has motion; if there is no motion, the user is considered to have taken out the already Scan the goods to enter the B4 state; after completing the scan code, the user is not allowed to be lower than the H3 plane to avoid the replacement risk; therefore, for the case of motion, if there is motion below the H3 surface, the ERROR ends. Otherwise keep the B2 state and end this logic Judgment

B3 state: The user leaves the monitoring area before scanning the code, and enters the B3 state at this time; the B3 state function matches the current actual weight and the theoretical weight after waiting for the weight to stabilize; the user is not allowed to enter the monitoring area in the B3 state;

B4 state: After the user scans the code, the product is taken out of the shopping cart and exits the monitoring area, and enters the B4 state; the B4 state is completed waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period;

B5 status: The B5 status completion function is similar to the A5 status. In order to wait for the actual barcode information corresponding to the product, after waiting for the actual information of the scanned product, the weight value and the theoretical weight value provided by the current electronic weighing scale are obtained. Match the two; if it matches, the current shopping process is completed, return to the initial state A0, prompting the user to "this return is completed"; if it does not match, it ends with ERROR;

Similarly, the B5 status requires the user not to enter the surveillance area to avoid the risk of theft. If the surveillance area has motion during this period, it ends with ERROR.

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the A0 state and the A2 state are specifically performed as follows:

(1) Determine the position of the hand, use the shopping cart opening as the reference surface H1, monitor whether there is a target above the reference plane, and save the picture if any;

(2) image processing, processing the image obtained in the first step by establishing a skin color model;

(3) Judging the holding state, and performing the second judgment on the image obtained in the second step.

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the step (1) includes the following sub-steps:

(1.1) The depth map is compared with H1, and the portion higher than the H1 surface is the foreground. If the area of the current scene is greater than a certain threshold, it is considered that the monitoring area has motion;

(1.2) Statistic foreground movement and the number of intersections on each side of the monitoring area, and the side with the largest number of intersection points is considered to be the direction in which the moving object enters;

(1.3) Make a circumscribing rectangle of the foreground that includes all foregrounds;

(1.4) taking the circle farthest from the entry boundary obtained in step 2 in the inscribed circle of the circumscribed rectangle;

(1.5) Pull out the image of the part of the circle in the color image, which is the position of the hand or the product.

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the step (2) includes the following sub-steps: (2.1) calculating the mean μ of the image I in the RGB three-channel according to the formula 1,

(2.2) Calculate the covariance matrix Cov of the image I according to the mean calculated according to (2.1) according to formula 2,

The formula for calculating ω _i in Equation 2 is as follows:

(2.3) Calculate the projection matrix Pro, Pro=μ*Cov according to the mean and covariance matrix calculated by (2.1) and (2.2).

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the step (3) includes the following sub-steps:

(3.1) Collecting a skin color-based image set, requiring that the grayscale value of other regions be 0 other than the skin color region;

(3.2) Using the image set in (3.1), and calculating the mean, covariance matrix and projection matrix of the skin color region of the image set according to the method in step (2), and determining the projection range of the skin color region according to the projection matrix;

(3.3) Input an image with a skin color region, calculate the projection matrix corresponding to each pixel of the image by using the mean and covariance matrix obtained in (3.2), if the projection matrix is calculated in (3.2) Within the matrix, the pixel is considered to be skin color, otherwise it is not skin color.

A further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention further includes the step of judging the handheld product, specifically:

(4.1) Performing the first judgment on the cut effective moving image provided in the second module, and if the image is empty, that is, the cut image has no moving part, then the direct determination is empty;

(4.2) If the image in the first judgment is not empty, the second determination of the skin color binary image after cutting is performed, and if the image is empty, that is, there is no skin color portion in the motion cut image, then the hand-held article is directly determined;

(4.3) if the image is not empty in the second determination, calculate a maximum contour area of the motion gray image and an area threshold b to perform a second determination;

(4.4) If the maximum contour area is larger than the area threshold b, then the skin color is set;

(4.5) If the maximum contour area is smaller than the area threshold, the color image is converted to the HSV color space, and the S channel is extracted.

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the sub-step in the (4.4) includes:

(4.4.1) calculating the information entropy of the grayscale image after the setting;

(4.4.2) The fourth judgment is made on the information entropy. If the information entropy is greater than the threshold a, then it is judged to carry the commodity, otherwise it is judged to be empty;

The sub-steps included in (4.5) are:

(4.5.1) Calculate the information entropy of the channel, and perform a fifth judgment. If the information entropy is greater than the threshold c, then it is determined to be a carrier. Product, otherwise judged to be empty.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the thresholds a and c are adaptively adjusted according to the actual use environment, and the clearer the image is, the larger the threshold a is; the resolution is 320*240. The area threshold b in the image is equal to 50; under the premise of skin color detection, the information entropy of the pure hand is zero.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, we give an empirical value a=1.4, c=6.5 under the condition of sufficient illumination.

A further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention further comprises determining whether the user operates in violation of the shopping and returning process; in the purchase process, starting from the monitoring field of view, the benchmark is The judgment of whether or not the hand-held product is carried out for each frame where the intersection area of the surface H1 is zero is saved, and whenever the intersection area changes from zero to a non-zero boundary, the judgment result of the stored hand-held commodity is cleared, and the intersection area is changed. When it reaches zero, it restarts the judgment of whether or not the hand-held product is held for each frame with zero intersection area, and repeats these operations until there is no motion in the monitoring field of view, and the last saved data is counted, if the ratio of the hand-held goods is greater than If the threshold is certain, it can be determined that the hand-held commodity exits during the exit process, which is a violation operation; in the return process, starting from the motion in the monitoring field of view, the hand-held commodity is judged and saved for each frame with the intersection area of the reference plane H1 being zero. Judging the result, when the first occurrence of the intersection area is non-zero, the accumulated data is calculated. If the ratio of the hand-held goods is greater than a certain threshold, it can be determined that the hand-held goods enter, which is a violation operation, and then the result is cleared until the next return process is restarted.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the user is allowed to make the logical state by manually clicking the return button of the user interface if and only when the logic state is in the A0 state during the shopping process. Enter the B0 status of the return; the return process stipulates that the user can only use one hand to operate the goods, and the return is to pick up the goods from the shopping cart for return, so the user is allowed to hold the goods when entering the monitoring area to avoid replacement. The risk of theft caused by the act.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, a single operation process only allows one item to be purchased; the user must purchase the item after entering the monitoring area, and pass the A1 status and the A2 status. The monitoring process is judged whether the product is held by hand; the user is allowed to flip through the purchased goods in the A0 state during the purchase process, but the user is required to be empty when entering the monitoring area.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, a single operation process only allows one item to be returned; the user is allowed to flip through the B0 state of the returned item to view the purchased item in the vehicle, but Users are required to be empty when entering the monitoring area.

As a further preferred solution of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, after the user completes scanning the barcode, puts in the shopping cart, and withdraws from the monitoring area, the user waits for the “this shopping is completed” to appear, and a complete At the end of the purchase operation, the next purchase is allowed, and the monitoring area reflected in the A4 and A5 states is Judgment of movement.

As a further preferred embodiment of the control method of the smart shopping system based on the supermarket shopping cart of the present invention, the user is allowed to enter the return initial state B0 by clicking the "return" button in the initial A0 state, and only supports one purchased product at a time. The return behavior includes scanning the barcode of the product to be returned in the shopping cart and taking out the process. After the end of the process, if it is judged as a normal return behavior, the logic state returns to the initial state A0.

Compared with the prior art, the present invention has the following technical effects:

1. Divide the customer's complete shopping process into six states. In each state, the weight change information of the real-time feedback of the load cell, the image information of each frame collected by the monitoring device, and the product information acquired by the scanner are combined to some extent. A plurality of image analysis algorithms implement an anti-theft function of the smart shopping cart system during the shopping process; and perform grayscale image statistical feature analysis or pattern recognition on the detected moving target image or skin color detection image to realize empty hand or handheld products in the monitoring area The detection, promptly remind and correct the customer's illegal operation, realize the judgment of the customer's illegal shopping behavior, and realize the payment-free queuing, reduce the labor cost and improve the shopping experience while preventing the anti-theft and the stop loss;

2. When the above-mentioned smart shopping cart is used for shopping and returning operations, the user experience is improved by the customer self-checkout, and the specific judgment step is used to prevent the user from accidentally operating or stealing the goods, thereby ensuring the merchant. Interests.

DRAWINGS

Figure a1 is a schematic structural view of a module of the present invention;

Figure a2 is a flow chart of step 1 of the method of the present invention;

Figure a3 is a flow chart of step 2 of the method of the present invention;

Figure a4 is a flow chart of step 3 of the method of the present invention;

Figure a5 is a flow chart of step 4 of the method of the present invention;

Figure a6 is a flow chart of step 5 of the method of the present invention;

Figure a7 is a flow chart of step 6 of the method of the present invention;

Figure a8 is a flow chart of image determination of the method of the present invention;

Figure b1 is a schematic structural view of a smart shopping cart;

Figure b2 is a simplified diagram of the process of shopping and returning;

Figure b3 is a schematic flow chart of the A0 state during the shopping process;

Figure b4 is a schematic flow chart of the A1 state during the shopping process;

Figure b5 is a schematic flow chart of the A2 state during the shopping process;

Figure b6 is a schematic flow chart of the A3 state during the shopping process;

Figure b7 is a schematic flow chart of the A4 state during the shopping process;

Figure b8 is a schematic flow chart of the A5 state during the shopping process;

Figure b9 is a schematic flow chart of the B0 state during the return process;

Figure b10 is a schematic flow chart of the B1 state during the return process;

Figure b11 is a schematic flow chart of the state of B2 in the return process;

Figure b12 is a schematic flow chart of the B3 state during the return process;

Figure b13 is a schematic flow chart of the B4 state during the return process;

Figure b14 is a schematic flow chart of the B5 state during the return process;

Figure c1 is a schematic flow chart of the first step of the present invention;

Figure c2 is a schematic flow chart of the second step of the present invention;

Figure c3 is to determine whether the user violates the rules during the shopping process;

Figure c4 is to determine whether the user violates the rules during the return process.

detailed description

The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

Embodiment 1

The smart shopping system based on the supermarket shopping cart provided by the embodiment includes a controller installed between the handles of the shopping cart, a scanner, a load cell placed at the bottom of the basket, and a monitoring at the upper end of the basket, as shown in FIG. The camera of the basket mouth, the scanner is located on the lower side of the controller, opposite to the shopping cart basket mouth, and the scanner, the weighing sensor and the camera are respectively connected with the controller.

The resolution of the camera is 40-50 (800 * 600) megapixels, and the frame rate is not less than 25 frames per second.

The load cell has a measuring range of 0-100 kg and a full-scale weighing accuracy of ±1 g.

The scanner is connected to the controller via a USB interface.

As shown in FIG. b1, two planes are sequentially arranged under the frame of the basket as an H1 plane and an H3 plane, wherein the load cell is placed at the bottom of the basket below the H3 plane, and the scanning port of the scanner is H1. The planes overlap, the camera is placed above the H1 plane and the moving objects in the H1 plane and the space above it are monitored.

Also includes:

The hand segmentation calculation module determines and obtains a picture of the hand holding the product according to the trend of the hand movement;

The skin color calculation module, the opponent can not hold the picture of the product to do the skin color segmentation process;

Hand-held commodity judgment module to determine whether the hand actually holds the commodity;

The statistics module determines whether the user violates the rules by counting the statistics of the results of the handheld product judgment module.

A smart shopping system based on a supermarket shopping cart and a control method thereof, comprising the following steps,

S1, start controller, scanner, load cell and camera start collecting data information; as shown in Figure a2,

When the weight of the weighing sensor is constant, the customer does not put the product into the shopping basket or operates empty hand. If the product has been scanned, the controller will prompt the scanning code to be successful, and proceeds to step S4 to detect whether there is a violation operation; If the code is not scanned, the camera collects the image information in the shopping basket and transmits it to the controller for image judgment to determine whether the empty hand is still a hand-held object; if the object is hand-held, the process proceeds to step S2, and if the hand is empty, the action is ignored;

When the weighing weight of the weighing sensor increases, it indicates that the product has not been scanned or the scanning code is not successfully placed directly into the shopping basket, and then prompts to take out the rescanning code, and proceeds to step S6 to detect whether the taken product is just placed. Products of;

When the weight of the weighing sensor is reduced, it means that the purchase process has been completed, then the return code is scanned, and the process proceeds to step S3 to check whether there is a violation operation that does not meet the return;

S2, the step is after determining that there is an item in the hand; as shown in Figure a3,

If the weight of the weighing sensor does not change, the product does not enter the shopping basket, the controller prompts to scan the code, and when the scanning code is successful, the process proceeds to step S4;

If the weight of the weighing sensor is reduced, it means that the customer has the illegal operation of stealing the goods, and the controller will prompt the illegal operation to take the artificial channel;

If the weighing weight of the weighing sensor is increased, it means that the customer has placed a new product without scanning the code, and the controller will prompt the new product to be taken out, and proceed to step S6 to check whether the taken product is in conformity, so as to avoid stealing. commodity;

S3, as shown in FIG. a4, when the weight reduction signal is detected in step S1, the hand is already in the area detected by the camera, and the judgment is made according to the detected position of the center of mass of the moving target;

When the center of mass is outside the center of the shopping basket, it is considered that the customer has taken out the product and has not scanned the code, and cannot determine which product the customer took out, so the illegal operation will be prompted, please take the artificial channel;

When the center of mass is in the center of the shopping basket, the real-time weight change information of the weight is collected; if the weight is unchanged, it means that the code needs to be scanned again during the normal return process, so the feedback code of the feedback determines whether the return is successful; If the code is not scanned, the controller will prompt the return, please scan the code; if the code has been scanned, the weight will be matched to determine whether the goods that are taken out meet the product of the scan code; if it meets the prompt, please take it out and proceed to step S5 to detect Take out the illegal operation in the process; if it does not, it means that the customer's goods are not correct, please prompt to take the manual channel; if the weight is increased, the weight matching will be carried out. If the matching does not match, the customer is considered to have placed other Goods, the controller prompts to take the artificial channel; if the match is met, the image is judged to prevent the customer from putting the same weight of the goods; if the weight is reduced, it is considered that the two items are returned at the same time, the controller will remind the customer ;

S4, as shown in FIG. 5, when the step S1 prompts the scan code to be successful, the camera detects the centroid position of the moving target. If the centroid is detected outside the center of the shopping basket, the customer is deemed to be returning, so the user is prompted to rescan the code. , complete the return process;

The real-time weight change information fed back by the weight is divided into three cases; when the weight is constant, only the position of the centroid is judged;

When the weight is reduced, it is considered that the customer takes out the goods without scanning the code, reminding the customer to operate illegally, and needs to take the artificial passage;

When the weight is increased, the item has been placed in the shopping cart, and the weight is matched at this time. If it matches, the image judges whether the customer's hand holds the item. If the item is held, it means that there are two objects of equal weight but different price. The product of the scanned code may be different from the inserted product, so it is considered to be Violation of the operation and prompting to take the artificial channel; if empty, the purchase is considered successful, will prompt to continue shopping, and return to step S1. If it does not meet, it will remind you to take out the new product, and proceed to step S6 to detect whether there is any illegal operation during the process of taking out;

S5, as shown in Figure a6, this step is to check whether there is a violation operation when the goods are taken out after the weight matching in the return process;

When the weight is increased or the weight is reduced, it is considered to be a violation operation;

When the weight is constant, according to the difference binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the commodity has exited the monitoring area, the return is over, and the process returns to step S1 to continue shopping;

S6, as shown in FIG. a7, the step is to detect a weight mismatch during the purchase process, and to prompt whether the detected product is the same as the inserted product when the product is taken out;

When the weight is increased, it is an unconventional shopping operation that the product is not taken out and continues to be placed in the new product, which will remind the customer;

When the weight is reduced, according to the differential binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the product has exited the monitoring area, and after the correct product is taken out, step S1 is continued to continue shopping. .

Image judgment is performed by first performing gray scale conversion on the original color image, then performing motion detection and determining the motion target centroid. When the centroid is in the shopping basket monitoring center area, the color original image is detected by the skin color, and the motion detection result is obtained. And the skin color detection result is summed to obtain a complete moving target image, and then the complete moving target result is subtracted from the skin color result to obtain a skin color moving target; and then the number of pixels with a pixel value of 255 in the skin color moving image is calculated as the skin color removing area. And calculate the information entropy of the complete moving target below the skin color centroid. If the skin area of the continuous multi-frame detection is greater than the threshold A or the information entropy of the motion region below the skin color centroid is greater than the threshold B, it can be considered that the handheld object is in the shopping basket, and vice versa. Empty hand, need to update the background.

Shopping process complete statistical process

1. Summary:

The complete shopping process is shown in Figure b2. It consists of purchase status A0, A1, A2, A3, A4, A5, return status B0, B1, B2, B3, B4, B5, as shown in Figure 1. Each state is used as the starting position of each logical judgment. For example, if it is currently in the A0 state, in the logic judgment process, starting from A0, according to the sensor feedback data and the judgment flow, the branches in the A0 state are judged. To determine whether the next logical judgment process is still in the A0 state, enter other states, or end with ERROR. When the user has a violation operation, that is, the user behavior is judged to have a large risk of theft, the logical judgment ends with ERROR. Then wait for the user to take out all the items in the shopping cart and empty the shopping cart. Through a series of images After the algorithm determines that the user has finished emptying the shopping cart operation, the logic state is set back to the initial state A0, allowing the user to restart the purchase and the like.

2 shopping process:

(1) A0 state is shown in Figure b3

The A0 state is the initial state of the shopping cart. When the tablet responsible for the algorithm is started, the sensors of the shopping cart start working, including electronic weighing scales, commodity barcode scanners, and monitoring devices.

According to the image data fed back by the monitoring device, the image processing algorithm gives whether the current monitoring area has motion. If there is no motion, the judgment ends, and the next logical judgment is still in the A0 state until the movement occurs in the monitoring area. It is considered that the user starts the series operation in the purchase behavior, and starts to judge whether the user has multi-hand operation (the algorithm sees claim 4) .

If there are many hands in the monitoring area. End with ERROR; otherwise, according to whether the user enters the monitoring process whether the item is hand-held, the flag variable flag is assigned as one of the basis for subsequent judgment in the A1 state, and the current logic judgment is ended in the A1 state.

(2) A1 state is shown in Figure b4

In this state, the logic algorithm first determines whether the user has completed the operation of scanning the commodity barcode. If the code has been scanned, the user is considered to complete the product information entry by scanning the code. The next action should be to put the scanned product in the shopping cart into the shopping cart and end the logic judgment in the A2 state; if the code has not been scanned at this time, According to the image processing algorithm, there are two cases in which the monitoring area has motion and the monitoring area has no motion.

When there is motion in the monitoring area, it is judged whether there is motion below the H3 plane (the algorithm sees claim 3), in combination with the value of the flag variable flag obtained in the A0 state, preventing the user from replacing the unpurchased item in the hand before scanning the barcode. The goods have been purchased in the car. When the motion is lower than the H3 plane, if the value of the flag is false (that is, the user holds the item when entering the monitoring area in the A0 state), it is considered that there is a potential risk of replacing theft. This logic judges to end with ERROR, waiting for the user to empty the shopping cart; The motion is lower than the H3 plane, but the value of the flag is true or the motion is higher than the H3 plane. The logic judgment ends, and the next judgment is still in the A1 state, with the A1 position as the starting position.

If there is no motion in the monitoring area, it is considered that the movement into the monitoring area in the A0 state has ended. At this time, the end point of this logical judgment is determined according to the monitoring result in the motion process. If it is found that there is a phenomenon that the hand-held item is out of the monitoring area during the monitoring process, it ends with ERROR; otherwise, it ends normally, and the next time it is judged to enter the A3 state.

(3) A2 state is shown in Figure b5

The user enters the A3 state after completing the scan operation. First, it is judged whether there is motion in the current monitoring area.

If there is motion in the surveillance area, the user is considered to have not completed the action of placing the scanned product in the shopping cart and withdrawing the hand from the field of view. In order to prevent the user from completing the theft by the replacement operation, it is stipulated that many hands are not allowed to operate on the goods. If there is a multi-hand operation. End with ERROR; if one-handed operation, the current logic judgment is ended in the A2 state.

There is no motion in the monitoring area. The next step is to judge that there is a hand-held product phenomenon in the monitoring process. If it exists, it ends with ERROR. If it does not exist, it enters the A4 state and ends the logic judgment.

(4) The A3 state is shown in Figure b6.

The user enters the monitoring area, does not scan the code, and enters the A3 state after emptying the monitoring area. What is done in the A3 state is to wait for the weight to stabilize and match the theoretical weight, and the user is not allowed to enter the monitoring area.

First, judge the motion state of the monitoring area. If there is motion, it ends with ERROR. If there is no motion, judge whether the weight is stable according to the electronic weighing scale.

The weight is unstable, and the logic judgment is still ended with A3; the weight is stable to match the current actual weight and the theoretical weight. The weight matching returns to the initial state A0, ending the current judgment; if it does not match, it ends with ERROR.

(5) The A4 state is shown in Figure b7.

After scanning the code, the user puts the product into the shopping cart and exits the monitoring area to enter the A4 state. The A4 state is completed waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period.

(6) A5 status as shown in Figure b8

After the user scans the product barcode, the algorithm needs to obtain information such as the theoretical weight of the product corresponding to the product barcode from the server through the network. The network will have a certain degree of delay, so the A5 state is introduced, waiting for the actual barcode information to be obtained.

After the weight is stable, after waiting for the actual information of the scanned product, the two are matched according to the weight value provided by the current electronic weighing scale and the theoretical weight value. If it matches, the current shopping process is completed, returning to the initial state A0, prompting the user to "this shopping is completed"; if not, ending with ERROR.

Similarly, the A5 status requires the user not to enter the surveillance area to avoid the risk of theft. If the surveillance area has motion during this period, it ends with ERROR.

3. Return process

(1) B0 state is shown in Figure b9

If and only if the logic state is at A0, the user is allowed to enter the return status by manually clicking the "Return" button of the user interface. The return process stipulates that the user can only use one hand to operate the goods. The return is to pick up the goods from the shopping cart and return the goods. Therefore, the user is not allowed to hold the goods when entering the monitoring area to avoid the risk of theft caused by the replacement behavior. .

The initial status of the return is B0. In this state, it is first determined whether there is motion in the current monitoring area.

If the monitoring area has no motion, it is considered that the user has not started the operation related to the return, and the current logic judgment is ended, and the status is still B0.

For the monitoring area to have motion, the user is considered to enter the monitoring area. If there is a phenomenon that the user enters the monitoring area with more hands at this time, the image algorithm gives a "multi-hand operation" decision, ending with ERROR. For "one-handed operation" that complies with the operating specifications, It is further determined whether the user holds the goods in the process of entering the monitoring area. There is a hand-held product entering the process, ending with ERROR; otherwise, entering the B1 state, the current logic judgment is ended.

(2) The B1 state is as shown in Figure b10.

The function completed in the B1 state is to wait for the user to pick up the item to be returned in the shopping cart and scan the code.

When the logic judges to enter B1, it first checks whether the current user completes the scan operation. If the code has been scanned, the logic judgment is terminated by entering the B2 state; if the code is not scanned, it is judged whether there is motion in the monitoring area.

There is no movement in the monitoring area. It is considered that the user's behavior in this series is empty hand entering the shopping cart monitoring area, and finally the return is not returned, and the monitoring area is exited to enter the B3 state to end the logic judgment.

If there is motion in the surveillance area, continue to determine if there is a multi-hand operation. If there are multiple hand operations, it ends with ERROR; if it does not exist, it keeps the B1 state and ends this logic judgment.

(3) The B2 state is as shown in Figure b11.

The user enters the B2 state after performing the return scan code. The function completed by this state is to wait for the user to take the scanned goods out of the monitoring area.

First, determine if there is motion in the monitored area.

If there is no movement, it is considered that the user has taken out the scanned product to end the B4 state.

After the scan code is completed, the user is not allowed to be below the H3 plane to avoid the risk of replacement. Therefore, in the case of motion, if there is motion below the H3 plane, it ends with ERROR, otherwise the B2 state is maintained to end this logic determination.

(4) The B3 state is shown in Figure b12.

The user leaves the monitoring area before scanning the code, and enters the B3 state at this time. The B3 status function matches the current actual weight and theoretical weight after waiting for the weight to stabilize. The user is not allowed to enter the monitoring area in the B3 state. The execution process is similar to the A3 status in the purchase.

(5) The B4 state is as shown in Figure b13.

After scanning the code, the user takes the product out of the shopping cart and exits the monitoring area to enter the B4 state. The B4 state is completed by waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period. Similar to the A4 status.

(6) B5 state is shown in Figure 14.

The B5 status completion function is similar to the A5 status, waiting to obtain the actual information corresponding to the product barcode.

After the weight is stable, after waiting for the actual information of the scanned product, the two are matched according to the weight value provided by the current electronic weighing scale and the theoretical weight value. If it matches, the current shopping process is completed, returning to the initial state A0, prompting the user to "this return is completed"; if it does not match, it ends with ERROR.

Similarly, the B5 status requires the user not to enter the surveillance area to avoid the risk of theft. If the surveillance area has motion during this period, it ends with ERROR.

4. User Code of Conduct:

This process framework has the following user behavior specifications:

A single operation allows only one item to be purchased or returned;

In the purchase state, the user must purchase the goods after entering the monitoring area, which is reflected in the judgment of whether the monitoring process is holding the goods in the A1 and A2 states;

After the user finishes scanning the barcode, puts it into the shopping cart, and withdraws from the monitoring area, after waiting for the “this shopping is completed”, a complete purchase operation is completed, and the next purchase is allowed, which is reflected in the A4 and A5 states. Whether the monitoring area has a judgment of movement;

It is not allowed to use multiple hands during the shopping and return process to avoid the risk of replacement, which is reflected in the single/multi-hand operation judgments in A2, B0 and B1;

For the purchase process, the goods to be purchased should be scanned first and then placed in the shopping cart;

In the initial A0 state, the user is allowed to enter the return initial state B0 by clicking the "return" button, and only supports the return behavior of one purchased item at a time, including the process of scanning the barcode of the product to be returned in the shopping cart and taking out the process. After the end, if it is judged as a normal return behavior, the logic state returns to the initial state A0;

Allows the user to flip through the in-car purchases in the initial state of purchase or return, but requires the user to be empty when entering the surveillance area.

Second, the H1 surface update part

When the product piles over the H1 surface, it will affect the foreground. At this time, the H1 surface should be updated.

As shown below, the steps:

Use the most primitive H1 surface to obtain a foreground above the H1 surface;

Calculate whether there is an intersection with the boundary in the foreground of the motion. If there is an intersection with the boundary, it is considered to be a moving object (hand); otherwise, it is an object higher than the H1 surface;

If there is no intersection between the motion foreground and the boundary, the H1 plane is updated. The gray value of the corresponding position of the original H1 surface is covered by the non-zero portion in the foreground, that is, the H1 surface is updated.

Other than the above-described embodiments, the present invention may have other embodiments. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

A supermarket shopping cart hand identification method such as state B0 and state B2 is performed as follows:

Judging the position of the hand, using the shopping cart opening as the reference surface H1, monitoring whether there is a target above the reference plane, and if so, saving the picture.

As shown in Figure c1; the step (1) includes the following sub-steps: (1.1) comparing the depth map to the H1 surface, the portion higher than the H1 surface is the foreground, and the area of the current scene is greater than a certain threshold, then the monitoring area is considered Have exercise;

(1.2) Statistic foreground movement and the number of intersections on each side of the monitoring area, and the side with the largest number of intersection points is considered to be the direction in which the moving object enters;

(1.3) Make a circumscribing rectangle of the foreground that includes all foregrounds;

(1.4) taking the circle farthest from the entry boundary obtained in step 2 in the inscribed circle of the circumscribed rectangle;

(1.5) Pull out the image of the part of the circle in the color image, which is the position of the hand or the product.

Image processing, processing the image obtained in the first step by establishing a skin color model;

As shown in FIG. c2, step (2) includes the following sub-steps: (2.1) calculating the mean μ of the image I in the RGB three-channel according to the formula 1,

(2.2) Calculate the covariance matrix Cov of the image I according to the formula 2 according to the mean μ calculated by (2.1),

The formula for calculating ω _i in Equation 2 is as follows:

(2.3) Calculate the projection matrix Pro, Pro=μ*Cov according to the mean and covariance matrix calculated by (2.1) and (2.2).

The state of the object is judged, and the image obtained in the second step is judged twice.

Step (3) includes the following sub-steps:

(3.1) Collecting a skin color-based image set requires that the gray value of other areas be 0 except for the skin color area.

(3.2) Using the image set in (3.1), and calculating the mean, covariance matrix and projection matrix of the skin color region of the image set according to the method in step (2), and determining the projection range of the skin color region according to the projection matrix;

(3.3) Input an image with a skin color region, calculate the projection matrix corresponding to each pixel of the image by using the mean and covariance matrix obtained in (3.2), if the projection matrix is calculated in (3.2) Within the matrix, the pixel is considered to be skin color, otherwise it is not skin color.

(4) Handheld commodity judgment steps, specifically:

(4.1) Performing the first judgment on the cut effective moving image provided in the second module, and if the image is empty, that is, the cut image has no moving part, then the direct determination is empty;

(4.2) If the image in the first judgment is not empty, the second determination of the skin color binary image after cutting is performed, and if the image is empty, that is, there is no skin color portion in the motion cut image, then the hand-held article is directly determined;

(4.3) if the image is not empty in the second determination, calculate a maximum contour area of the motion gray image and an area threshold b to perform a second determination;

(4.4) If the maximum contour area is larger than the area threshold b, then the skin color is set;

In (4.4), the sub-step includes (4.4.1) calculating the information entropy on the grayscale image after the setting;

(4.4.2) The fourth judgment is made on the information entropy. If the information entropy is greater than the threshold a, then it is judged to carry the commodity, otherwise it is judged to be empty;

In (4.5), the sub-step is (4.5.1) to calculate the channel information entropy, and the fifth judgment is performed. If the information entropy is greater than the threshold c, it is judged to carry the commodity, otherwise it is judged to be empty.

(4.5) If the maximum contour area is smaller than the area threshold, the color image is converted to the HSV color space, and the S channel is extracted.

The thresholds a and c are adaptively adjusted according to the actual use environment, and the clearer the image is, the larger the threshold a is; in the image with the resolution of 320*240, the area threshold b is equal to 50; under the premise of the skin color detection, the pure hand information Entropy is zero. We gave an empirical value a = 1.4, c = 6.5 under well-lit laboratory conditions.

(5) Judging whether the user violates the rules during the shopping and returning process; as shown in Figure c3, in the purchase process, starting from the motion in the monitoring field of view, whether to hold the product for each frame with the intersection area of the reference plane H1 is zero. The judgment is saved and saved. Whenever the intersection area changes from zero to a non-zero boundary, the judgment result of the saved hand-held goods is cleared. When the intersection area is changed to zero, each of the intersection areas is zero. The frame performs judgment and storage of the hand-held product, and repeats these operations until there is no motion in the monitoring field of view, and the last saved data is counted. If the ratio of the hand-held product is greater than a certain threshold, it can be determined that the hand-held product exits during the exit process. As shown in Figure c4, it is a violation operation; in the return process, starting from the motion in the monitoring field of view, the hand-held commodity is judged for each frame with the intersection area of the reference plane H1 is zero, and the judgment result is saved, and the first intersection occurs. When the area is non-zero, the ratio of the hand-held goods is calculated cumulatively for the saved data. If it is greater than a certain threshold, it can be determined as a hand-held product. Into, for the illegal operations, and the result is clear until the next re-start the return process.

The invention finally obtains the judgment of whether the hand holds the commodity through the steps of motion detection, skin color detection and hand and commodity judgment; the method does not impose additional burdens and requirements on the user, and the whole judging process is fully automatic. Very simple and convenient.

Other than the above-described embodiments, the present invention may have other embodiments. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims (18)

1. A smart shopping system based on a supermarket shopping cart, comprising: a controller mounted between the handles of the shopping cart, a scanner, a load cell placed at the bottom of the basket, and a camera positioned at the upper end of the basket. The scanner is located on the lower side of the controller, opposite to the shopping cart basket mouth, and the scanner, the load cell and the camera are respectively connected with the controller, wherein the frame of the basket is sequentially arranged below The two planes are respectively an H1 plane and an H3 plane, wherein the load cell is placed at the bottom of the basket below the H3 plane, the scanning port of the scanner overlaps with the H1 plane, the camera is placed above the H1 plane and the H1 is monitored A moving object in the plane and the space above it.
2. A smart shopping system based on a supermarket shopping cart, characterized in that it further comprises a hand segmentation calculation module, which determines and obtains a picture of a hand that may be held by the hand according to the trend of the hand movement;

   The skin color calculation module, the opponent can not hold the picture of the product to do the skin color segmentation process;

   Hand-held commodity judgment module to determine whether the hand actually holds the commodity;

   The statistics module determines whether the user violates the rules by counting the statistics of the results of the handheld product judgment module.
3. A method for controlling a smart shopping system based on a supermarket shopping cart, characterized in that the method comprises the following steps:

   S1, the start controller, the scanner, the load cell and the camera start collecting data information; wherein each component operates according to the following state:

   A0 state: It is judged by the camera whether there is a hand entering the monitoring area;

   A1 status: It is judged by the scanner and the camera whether there is a product scan operation and the action of putting the product into the bottom of the shopping basket below the H3 plane;

   A2 status: It is judged by the camera whether there is an empty hand in the monitoring area;

   A3 state: the product is weighed by the load cell and the result is fed back to the controller for judgment;

   A4 state: Waiting for the weighing sensor to weigh the goods, the result is stable and prompt the user's hand not to enter the monitoring area during this period;

   A5 status: compare the product weight information obtained by the scanner with the result obtained by weighing, and if the two match, the shopping is completed;

   B0 state: the user operates the controller to enter the return procedure, and the camera monitors whether the user enters the shopping cart H1 plane empty hand;

   B1 state: the camera monitors whether the user holds the item from below the H1 plane into the monitoring area, and whether the user scans the code through the commodity scanner;

   B2 status: The camera monitors whether the user takes the item out of the monitoring area after scanning the code;

   B3 state: If the user does not scan the code and takes out the item out of the monitoring area, the controller waits for the load cell to decrease the value and queries whether it is the same as the theoretical weight of any product purchased;

   B4 state: After the user scans the code and takes the item out of the monitoring area, the controller waits for the load cell to reduce the value and queries whether The theoretical weight is the same as the product scanned code;

   B5 status: Whether the actual product weight information is compared with the actual detected value of the B3 or B4 status.

   When the weight of the weighing sensor is constant, the customer does not put the product into the shopping basket or operates empty hand. If the product has been scanned, the controller will prompt the scanning code to be successful, and proceeds to step S4 to detect whether there is a violation operation; If the code is not scanned, the camera collects the image information in the shopping basket and transmits it to the controller for image judgment to determine whether the empty hand is still a hand-held object; if the object is hand-held, the process proceeds to step S2, and if the hand is empty, the action is ignored;

   When the weighing weight of the weighing sensor increases, it indicates that the product has not been scanned or the scanning code is not successfully placed directly into the shopping basket, and then prompts to take out the rescanning code, and proceeds to step S6 to detect whether the taken product is just placed. Products of;

   When the weight of the weighing sensor is reduced, it means that the purchase process has been completed, then the return code is scanned, and the process proceeds to step S3 to check whether there is a violation operation that does not meet the return;

   S2, the step is after determining that there is an item in the hand;

   If the weight of the weighing sensor does not change, the product does not enter the shopping basket, the controller prompts to scan the code, and when the scanning code is successful, the process proceeds to step S4;

   If the weight of the weighing sensor is reduced, it means that the customer has the illegal operation of stealing the goods, and the controller will prompt the illegal operation to take the artificial channel;

   If the weighing weight of the weighing sensor is increased, it means that the customer has placed a new product without scanning the code, and the controller will prompt the new product to be taken out, and proceed to step S6 to check whether the taken product is in conformity, so as to avoid stealing. commodity;

   S3. When the weight reduction signal is detected in step S1, the hand is already in the area detected by the camera, and the judgment is made according to the detected position of the center of mass of the moving target;

   When the center of mass is outside the center of the shopping basket, it is considered that the customer has taken out the product and has not scanned the code, and cannot determine which product the customer took out, so the illegal operation will be prompted, please take the artificial channel;

   When the center of mass is in the center of the shopping basket, the real-time weight change information of the weight is collected; if the weight is unchanged, it means that the code needs to be scanned again during the normal return process, so the feedback code of the feedback determines whether the return is successful; If the code is not scanned, the controller will prompt the return, please scan the code; if the code has been scanned, the weight will be matched to determine whether the goods that are taken out meet the product of the scan code; if it meets the prompt, please take it out and proceed to step S5 to detect Take out the illegal operation in the process; if it does not, it means that the customer's goods are not correct, please prompt to take the manual channel; if the weight is increased, the weight matching will be carried out. If the matching does not match, the customer is considered to have placed other Goods, the controller prompts to take the artificial channel; if the match is met, the image is judged to prevent the customer from putting the same weight of the goods; if the weight is reduced, it is considered that the two items are returned at the same time, the controller will remind the customer The image is judged by performing gradation conversion on the color original image, then performing motion detection and determining the moving target centroid, when the centroid is in the shopping basket monitoring After the region of the color original skin color detection, skin color motion detection result and the detection result obtained by summing target image full motion, full motion then the target color and the results obtained to the subtraction result The skin color moving target; then calculating the number of pixel points in the skin color moving image with the pixel value of 255 as the skin color area, and calculating the information entropy of the complete moving object below the skin color center of mass, if the skin color area of the continuous multi-frame detection is greater than the threshold A or The information entropy of the motion region below the skin color center of mass is greater than the threshold B, which can be considered as a hand-held object in the shopping basket, and vice versa is considered to be an empty hand, and the background needs to be updated;

   S4. After the step S1 prompts that the scan code is successful, the camera detects the centroid position of the moving target. If the centroid is detected outside the central area of the shopping basket, the customer is deemed to be returning, so the user is prompted to rescan the code to complete the return process;

   The real-time weight change information fed back by the weight is divided into three cases; when the weight is constant, only the position of the centroid is judged;

   When the weight is reduced, it is considered that the customer takes out the goods without scanning the code, reminding the customer to operate illegally, and needs to take the artificial passage;

   When the weight is increased, the product has been placed in the shopping cart, and the weight is matched at this time; if it is met, the image judges whether the customer's hand holds the item, and if the item is held, it is considered that the object has two objects of equal weight but not equal price. The product of the scan code may be different from the product put in, so it is considered to be a violation of the operation and prompt to take the artificial channel; if it is empty, it is considered that the purchase is successful, it will prompt to continue shopping, and return to step S1; if not, Will remind you to take out new products, go to step S6, and check if there is any illegal operation during the process of taking out;

   S5. This step is to check whether there is a violation operation when the product is taken out after the weight is matched in the returning process;

   When the weight is increased or the weight is reduced, it is considered to be a violation operation;

   When the weight is constant, according to the difference binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the commodity has exited the monitoring area, the return is over, and the process returns to step S1 to continue shopping;

   S6. In this step, the weight mismatch is detected during the purchase process, and the product detected when the product is taken out is the same as the inserted product;

   When the weight is increased, it is an unconventional shopping operation that the product is not taken out and continues to be placed in the new product, which will remind the customer;

   When the weight is reduced, according to the differential binary image of the previous frame and the current frame, if the white pixel area is less than a small threshold, the product has exited the monitoring area, and after the correct product is taken out, step S1 is continued to continue shopping. .
4. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 2, wherein each state is specifically:

   A0 state: When the tablet controller responsible for the algorithm starts, the electronic weighing scale, the commodity scanner and the camera on the shopping cart start working; according to the image data fed back by the camera, the image processing algorithm gives the current monitoring area whether there is motion. If there is no motion, the judgment ends, the next logical judgment is still A0 state, until the movement occurs in the monitoring area, the user is considered to start the series of operations in the purchase behavior, and it is judged whether the user has multi-hand operation; if there is any monitoring area Multiple hands, end with ERROR, otherwise, according to whether the user enters the monitoring process whether the item is held, the flag variable flag is assigned as one of the basis for subsequent judgment in the A1 state, and the current logic judgment is ended in the A1 state;

   A1 state: In this state, the controller first determines whether the user completes the operation of scanning the product barcode; if the current scan has been performed Code, it is considered that the user completes the product information entry by scanning the code. The next action should be to put the scanned product in the shopping cart into the shopping cart, and end the logic judgment in the A2 state; if the code has not been scanned at this time, according to the image processing The algorithm is divided into two situations: the monitoring area has motion and the monitoring area has no motion;

   When there is motion in the monitoring area, it is judged whether the motion is lower than the H3 plane, and the value of the flag variable flag obtained in the A0 state is combined to prevent the user from replacing the purchased item in the shopping cart by the unpurchased item before scanning the barcode; Below the H3 face, if the value of flag is false, it is considered that there is a potential risk of replacing theft. This logic judges to end with ERROR, waiting for the user to empty the shopping cart; the motion is lower than the H3 plane but the value of flag is true or the motion is higher than H3 plane, the logic judgment ends, the next judgment is still in the A1 state, with the A1 position as the starting position; if there is no motion in the monitoring area, it is considered that the movement into the monitoring area in the A0 state has ended, according to the movement The monitoring result determines the end point of this logical judgment. If it is found that there is a phenomenon that the hand-held item is out of the monitoring area during the monitoring process, it ends with ERROR, otherwise it ends normally, and the next time it judges to enter the A3 state;

   A2 state: The user enters the A3 state after completing the scan operation. Firstly, it is judged whether there is motion in the current monitoring area; if there is motion in the monitoring area, it is considered that the user has not completed the action of putting the scanned product in the shopping cart and withdrawing the hand from the field of view; in order to prevent the user from completing the theft by the replacement operation, the provision is not Allow multiple hands to operate on the merchandise; if there is multi-hand operation, end with ERROR, if one-handed operation, the current logic judgment is ended in the A2 state; there is no motion in the monitoring area, and the next step is to judge that there is a hand-held commodity phenomenon in the monitoring process. Existence ends with ERROR; does not exist, enters the A4 state, and ends this logic judgment;

   A3 state: the user enters the monitoring area, does not scan the code, and enters the A3 state after emptying the monitoring area; the A3 state completes the matching with the theoretical weight after waiting for the weight to stabilize, and does not allow the user to enter the monitoring area during the period; first judge the monitoring area The state of motion, the movement ends with ERROR; if there is no movement, the weight is determined according to the electronic weighing scale; the weight is unstable, and the logic judgment is still ended with A3; the weight is stable to match the current actual weight and the theoretical weight; The weight matching returns to the initial state A0, ending the current judgment; if it does not match, it ends with ERROR;

   A4 status: After scanning the code, the user puts the product into the shopping cart and exits the monitoring area to enter the A4 state; the A4 state is completed waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period;

   A5 status: After the user scans the product barcode, the algorithm needs to obtain information such as the theoretical weight of the product corresponding to the product barcode from the server through the network. The network will have a certain degree of delay, so the A5 state is introduced, waiting for the actual information corresponding to the product barcode; after the weight is stable, waiting for the actual information of the scanned product, according to the weight value and the theoretical weight value provided by the current electronic weighing scale, Match the two; if it matches, the current shopping process is completed, return to the initial state A0, prompting the user to "this shopping is completed"; if it does not match, it ends with ERROR; similarly, the A5 state requires the user not to enter the monitoring area. To avoid the risk of theft, if the surveillance area has exercise during this period, it ends with ERROR;

   B0 state, the initial state of the return is B0, the first judgment in this state is whether the current monitoring area has motion; There is no motion in the monitoring area. The user has not started the operation related to the return of the goods. The current logic judges that the status is still B0. If there is motion in the monitoring area, the user is considered to enter the monitoring area. If there are multiple users entering the monitoring area at this time, The image algorithm gives a "multi-hand operation" decision, ending with ERROR; for the "one-hand operation" that conforms to the operating specification, it is further determined whether the user holds the goods in the process of entering the monitoring area; ERROR ends; otherwise, enter the B1 state, and end this logical judgment;

   B1 state: The function completed in the B1 state is to wait for the user to pick up the goods to be returned in the shopping cart and scan the code; when the logic judges to enter B1, first check whether the current user completes the scanning operation; if the code has been scanned, the user enters the B2 state. End this logical judgment; if the code is not scanned, it is judged whether there is motion in the monitoring area; the monitoring area has no motion, and the user's behavior in this series is considered to be empty-handed to enter the shopping cart monitoring area, and finally the return is not returned, and the monitoring area is exited to enter the B3 state. End this logical judgment; if there is motion in the monitoring area, continue to judge whether there is multi-hand operation. If there are multiple hand operations, it ends with ERROR; if it does not exist, the B1 state is kept to end this logical judgment;

   B2 state: the user enters the B2 state after performing the return scan code; the function completed by the state is to wait for the user to take the scanned code product out of the monitoring area; firstly, it is judged whether the monitoring area has motion; if there is no motion, the user is considered to have taken out the already Scan the goods to enter the B4 state; after completing the scan code, the user is not allowed to be lower than the H3 plane to avoid the replacement risk; therefore, for the case of motion, if there is motion below the H3 surface, the ERROR ends. Otherwise, the B2 state is maintained to end this logical judgment;

   B3 state: The user leaves the monitoring area before scanning the code, and enters the B3 state at this time; the B3 state function matches the current actual weight and the theoretical weight after waiting for the weight to stabilize; the user is not allowed to enter the monitoring area in the B3 state;

   B4 state: After the user scans the code, the product is taken out of the shopping cart and exits the monitoring area, and enters the B4 state; the B4 state is completed waiting for the weight to be stable, and the user is not allowed to enter the monitoring area during the specified period;

   B5 status: The B5 status completion function is similar to the A5 status. In order to wait for the actual barcode information corresponding to the product, after waiting for the actual information of the scanned product, the weight value and the theoretical weight value provided by the current electronic weighing scale are obtained. Match the two; if it matches, the current shopping process is completed, return to the initial state A0, prompting the user to "this return is completed"; if it does not match, it ends with ERROR;

   Similarly, the B5 status requires the user not to enter the surveillance area to avoid the risk of theft. If the surveillance area has motion during this period, it ends with ERROR.
5. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 2, wherein the A0 state and the A2 state are specifically performed according to the following steps:

   (1) Determine the position of the hand, use the shopping cart opening as the reference surface H1, monitor whether there is a target above the reference plane, and save the picture if any;

   (2) image processing, processing the image obtained in the first step by establishing a skin color model;

   (3) Judging the holding state, and performing the second judgment on the image obtained in the second step.
6. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 5, wherein the step (1) comprises the following sub-steps:

   (1.1) The depth map is compared with H1, and the portion higher than the H1 surface is the foreground. If the area of the current scene is greater than a certain threshold, it is considered that the monitoring area has motion;

   (1.2) Statistic foreground movement and the number of intersections on each side of the monitoring area, and the side with the largest number of intersection points is considered to be the direction in which the moving object enters;

   (1.3) Make a circumscribing rectangle of the foreground that includes all foregrounds;

   (1.4) taking the circle farthest from the entry boundary obtained in step 2 in the inscribed circle of the circumscribed rectangle;

   (1.5) Pull out the image of the part of the circle in the color image, which is the position of the hand or the product.
7. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 5, wherein the step (2) comprises the following sub-steps: (2.1) calculating the image I in the RGB three-channel according to the formula 1. Mean μ,

   

   (2.2) Calculate the covariance matrix Cov of the image I according to the formula 2 according to the mean μ calculated by (2.1),

   

   The formula for calculating ω _i in Equation 2 is as follows:

   

   (2.3) Calculate the projection matrix Pro, Pro=μ*Cov according to the mean and covariance matrix calculated by (2.1) and (2.2).
8. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 5, wherein the step (3) comprises the following sub-steps:

   (3.1) Collecting a skin color-based image set, requiring that the grayscale value of other regions be 0 other than the skin color region;

   (3.2) Using the image set in (3.1), and calculating the mean, covariance matrix and projection matrix of the skin color region of the image set according to the method in step (2), and determining the projection range of the skin color region according to the projection matrix;

   (3.3) Input an image with a skin color region, calculate the projection matrix corresponding to each pixel of the image by using the mean and covariance matrix obtained in (3.2), if the projection matrix is calculated in (3.2) Within the matrix, the pixel is considered to be skin color, otherwise it is not skin color.
9. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 5, further comprising: a step of determining a hand-held product, specifically:

   (4.1) Perform the first judgment on the cut effective motion image provided in Module 2, and if the image is empty, the cut image Like no moving part, it is directly judged to be empty;

   (4.2) If the image in the first judgment is not empty, the second determination of the skin color binary image after cutting is performed, and if the image is empty, that is, there is no skin color portion in the motion cut image, then the hand-held article is directly determined;

   (4.3) if the image is not empty in the second determination, calculate a maximum contour area of the motion gray image and an area threshold b to perform a second determination;

   (4.4) If the maximum contour area is larger than the area threshold b, then the skin color is set;

   (4.5) If the maximum contour area is smaller than the area threshold, the color image is converted to the HSV color space, and the S channel is extracted.
10. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 9, wherein the step (4.4) comprises the following steps:

    (4.4.1) calculating the information entropy of the grayscale image after the setting;

    (4.4.2) The fourth judgment is made on the information entropy. If the information entropy is greater than the threshold a, then it is judged to carry the commodity, otherwise it is judged to be empty;

    The sub-steps included in (4.5) are:

    (4.5.1) Calculate the information entropy of the channel, and perform a fifth judgment. If the information entropy is greater than the threshold c, it is determined to carry the commodity, otherwise it is judged to be empty.
11. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 10, wherein the thresholds a and c are adaptively adjusted according to the actual use environment, and the clearer the image, the larger the threshold a; The area threshold b is equal to 50 in the image of 320*240; under the premise of skin color detection, the information entropy of the pure hand is zero.
12. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 11, characterized in that, under the condition of sufficient light, we give an experience value a=1.4, c=6.5.
13. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 5, further comprising: determining whether the user operates illegally during the shopping and returning process; and during the purchasing process, there is motion from the monitoring field of view. Initially, whether or not the hand-held product is judged for each frame having the intersection area of the reference plane H1 is zero, and is saved, and whenever the intersection area changes from zero to a non-zero boundary, the judgment result of the stored hand-held product is cleared. When the intersection area is changed to zero, the judgment of whether or not the hand-held product is judged and saved for each frame with the intersection area of zero is restarted, and these operations are repeated until there is no motion in the monitoring field of view, and the last saved data is counted. If the ratio of the hand-held goods is greater than a certain threshold, it can be determined that the hand-held goods are withdrawn during the exiting process, which is an illegal operation; in the returning process, starting from the motion in the monitoring field of view, each frame having an intersection area with the reference plane H1 is zero. Hand-held goods judge and save the judgment result. When the intersection area is non-zero for the first time, the saved data is saved. Calculating a ratio of the count holding goods, if more than a certain threshold, can be determined for the goods into a handheld, for the illegal operation, and the result is returned until the next clear the process begins anew.
14. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 3, wherein the user is allowed to manually click the return button of the user interface if and only when the logic state is in the A0 state during the shopping process. , the logic state enters the B0 state of the return; the return process stipulates that the user can only use one hand to operate the goods, and the return is to pick up the goods from the shopping cart for return, so the user is allowed to hold the goods when entering the monitoring area. To avoid the risk of theft caused by the replacement behavior.
15. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 3, wherein: a single operation process only allows one item to be purchased; the user must purchase the item after entering the monitoring area, and pass the A1 status and Whether the monitoring process of the A2 state is held by the handhold product; the user is allowed to flip through the purchased goods in the A0 state during the purchase process, but the user is required to be empty when entering the monitoring area.
16. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 3, wherein: a single operation process only allows one item to be returned; allowing the user to flip through the returned B0 state to view the purchased item in the vehicle. Commodities, but all require users to be empty when entering the surveillance area.
17. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 3, wherein after the user completes scanning the barcode, puts it into the shopping cart, and exits the monitoring area, the user waits for the "this shopping is completed" to appear. At the end of a complete purchase operation, the next purchase is allowed, and the movement in the monitoring area in the A4 and A5 states is judged.
18. The method for controlling a smart shopping system based on a supermarket shopping cart according to claim 3, characterized in that: in the initial A0 state, the user is allowed to enter the return initial state B0 by clicking the "return" button, and only one piece is supported at a time. The return behavior of the purchased goods, including the process of scanning the barcode of the goods to be returned in the shopping cart and taking out the process, after the end of the process, if it is judged to be a normal return behavior, the logic state returns to the initial state A0.

Images