WO2020103487A1

WO2020103487A1 - Self-service settlement method and device and storage medium

Info

Publication number: WO2020103487A1
Application number: PCT/CN2019/097950
Authority: WO
Inventors: 石海林; 赵何; 刘武; 梅涛; 周伯文
Original assignee: 北京京东尚科信息技术有限公司; 北京京东世纪贸易有限公司
Priority date: 2018-11-20
Filing date: 2019-07-26
Publication date: 2020-05-28
Also published as: CN109214806A; US11861584B2; CN109214806B; EP3859638A1; US20220076227A1; EP3859638A4

Abstract

The present disclosure relates to the technical field of self-service shopping, and provided thereby are a self-service settlement method and device and a storage medium, the method comprising: obtaining a monitoring image corresponding to products to be settled which are placed on a settlement sales counter, the monitoring image being taken by an image acquisition device; obtaining information about the products to be settled by means of image recognition; obtaining a first weight acquired by a weight detection device and a second weight obtained on the basis of the information about the products to be settled; according to a comparison result for the weights, determining whether the information about the products to be settled and products to be affirmed match, and if yes, obtaining product purchase settlement information and carrying out payment settlement processing. According to the method, device and storage medium of the present disclosure, information about products to be detected is determined by means of combining image recognition technology and weight detection technology, which may achieve rapid self-service checkout, reduce the wait time for customers when settling a payment, and improve the shopping experience of customers.

Description

Self-service settlement method, device and storage medium

Cross-reference of related applications

This application is based on the application with the CN application number 201811380178.8 and the application date is November 20, 2018, and claims its priority. The disclosure content of the CN application is hereby incorporated into this application as a whole.

Technical field

The present disclosure relates to the field of self-service shopping technology, and in particular, to a self-service settlement method, device, and storage medium.

Background technique

Supermarkets, convenience stores, shopping malls and other shopping places usually use manual settlement methods. After shopping in the supermarket, customers need to go to the cashier for settlement. The cashier scans the barcode on each item and then calculates the total amount through the computer. The customer pays for the payment , It takes a long time to check out, and the experience is not high. With the development of smart retail technology, automatic settlement equipment has gradually appeared in some shopping venues, but most of the related automatic settlement equipment adopts commodity scanning methods or RFID (Radio Frequency Identification) and other electronic label methods to confirm product identification.

Summary of the invention

The inventor of the present disclosure found that the technical solution of the shopping settlement in the above-mentioned related art has defects: when a customer quickly passes through the RFID settlement channel, the label may be missed due to excessive speed and RFID tag blocking.

According to one or more embodiments of the present disclosure, a self-service settlement method is provided, including: obtaining a monitoring image collected by an image collection device corresponding to a commodity to be settled placed on a settlement counter; identifying the monitoring image to obtain Commodity information to be settled; wherein, the commodity information to be settled includes: the category and quantity of the commodity to be settled; obtaining the first weight corresponding to the commodity to be settled collected by the weight detection device provided on the settlement counter, based on Obtaining the second weight corresponding to the commodity to be settled by the commodity information to be settled; judging the commodity information to be settled according to a preset matching judgment rule and a comparison result of the first weight and the second weight Whether the goods to be determined match; if yes, obtain the purchase goods settlement information based on the goods to be settled, and perform checkout processing according to the purchase goods settlement information; wherein, the purchase goods settlement information includes: Category, quantity and settlement price.

In some embodiments, the identifying the monitoring image and obtaining the information of the commodity to be settled includes: determining a first position corresponding to the commodity to be settled in the monitoring image and a position on the settlement counter A second position corresponding to the coordinate scale; intercepting the commodity image to be settled and the coordinate scale image in the monitoring image according to the first position and the second position; based on the coordinate scale image and the commodity image to be settled Determine the size information of the product to be settled; identify the type of product to which the product to be settled belongs in the image of the product to be settled, and obtain the quantity corresponding to each product to be settled; based on the size information of the product to be settled and the The category of the commodity to be settled determines the category of the commodity to be settled.

In some embodiments, the determining the first position corresponding to the merchandise to be settled and the second position corresponding to the coordinate scale on the settlement counter in the monitoring image includes: using a target detection model to The first position and the second position are determined in the monitoring image; wherein the target detection model includes: a convolutional neural network model based on the Faster RCNN algorithm.

In some embodiments, the identification of the commodity category to which the commodity to be settled in the commodity image to be settled includes: establishing a fully connected layer of a convolutional neural network through the Softmax function, and calculating the The commodities to be settled belong to the confidence level of each commodity category; the category of commodities whose confidence level is greater than a preset threshold is taken as the commodity category of the commodity to be settled.

In some embodiments, a pooling layer is provided between each convolutional base layer of the convolutional neural network, and a batch normalization layer is provided after the last convolutional layer.

In some embodiments, the obtaining the second weight corresponding to the commodity to be settled based on the commodity information to be settled includes: obtaining the unit weight of the commodity corresponding to the category of the commodity to be settled; The unit weight and quantity of the settlement commodity obtain the second weight.

In some embodiments, judging whether the commodity information to be settled matches the commodity to be determined according to a preset matching judgment rule and a comparison result of the first weight and the second weight includes: obtaining the first A difference between a weight and the second weight; determining whether the absolute value of the difference is less than a preset difference threshold; if so, determining that the product information to be settled matches the product to be identified.

In some embodiments, the checkout process according to the purchase commodity settlement information includes: sending the purchase commodity settlement information to the first display device and the second display device, respectively, to display the customer and the store clerk respectively Purchase commodity settlement information; if a settlement cancellation or settlement error message for the purchase commodity settlement information sent by either of the customer and the clerk is received, the checkout process is suspended.

In some embodiments, the checkout process based on the purchase commodity settlement information further includes: receiving fee payment information generated from the purchase commodity settlement information sent by the customer terminal; if it is determined that the fee payment information is correct, then Send the fee payment completion information to the first display device and the second display device, and set the goods to be settled to the paid state, so that the goods to be settled pass the detection of the security detection device; if it is determined If the fee payment information is incorrect, the fee payment failure information is sent to the first display device and the second display device.

In some embodiments, acquiring payment settlement information corresponding to the customer terminal for successful payment of the payment and obtaining the commodity category and shopping frequency according to the purchase commodity settlement information; determining recommended products according to the commodity category and the shopping frequency 2. Purchase cycle, based on the purchase cycle to determine the push time; based on the push time to push the recommended products to the customer terminal; received commodity discount information, determine whether the commodity discount information matches the recommended commodity If yes, push the recommended product and the product preferential information to the customer terminal.

According to one or more embodiments of the present disclosure, a self-service settlement device is provided, including: an image acquisition module for acquiring a monitoring image collected by an image acquisition device corresponding to a commodity to be settled placed on a settlement counter; an image recognition module , Used to identify the monitoring image and obtain the information of the commodity to be settled; wherein, the information of the commodity to be settled includes: the category and quantity of the commodity to be settled; the weight obtaining module, which is used to obtain the weight detection set on the settlement counter The first weight corresponding to the commodity to be collected collected by the device obtains a second weight corresponding to the commodity to be settled based on the commodity information to be settled; a matching judgment module is used to determine the matching rule according to the preset And the comparison result of the first weight and the second weight determines whether the commodity information to be settled matches the commodity to be confirmed; a checkout processing module is used to determine if the commodity information to be settled and the commodity to be determined For product matching, the purchase commodity settlement information is obtained based on the commodity information to be settled, and checkout processing is performed according to the purchase commodity settlement information; wherein, the purchase commodity settlement information includes: the category, quantity and settlement price of the commodity to be settled.

In some embodiments, the image recognition module includes: a position determination unit for determining a first position corresponding to the commodity to be settled and a coordinate scale on the settlement counter in the monitored image Corresponding second position; an image intercepting unit, used to intercept the commodity image and coordinate scale image to be settled in the monitoring image according to the first position and the second position; a size obtaining module, used to based on the coordinates The scale image and the to-be-settled commodity image determine the size information of the to-be-settled commodity; the recognition processing unit is used to identify the category of the commodity to be settled in the to-be-settled commodity image, and obtain the correspondence of each to-settled commodity The quantity; determine the category of the commodity to be settled based on the size information of the commodity to be settled and the category of the commodity to which the commodity to be settled belongs.

In some embodiments, the position determining unit is configured to determine the first position and the second position in the monitoring image using a target detection model; wherein the target detection model includes: based on Faster RCNN algorithm Convolutional neural network model.

In some embodiments, the recognition processing unit is used to establish a fully connected layer of a convolutional neural network through the Softmax function, and calculate the confidence that the commodity to be settled belongs to each commodity category through the convolutional neural network; The commodity category whose degree is greater than the preset threshold is used as the commodity category of the commodity to be settled.

In some embodiments, the weight obtaining module is configured to obtain the unit weight of the commodity corresponding to the category of the commodity to be settled; and obtain the second weight according to the unit weight and quantity of the commodity to be settled.

In some embodiments, the matching judgment module is configured to obtain the difference between the first weight and the second weight; determine whether the absolute value of the difference is less than a preset difference threshold; if so, Then, it is determined that the product information to be settled matches the product to be identified.

In some embodiments, the checkout processing module is configured to send the purchase merchandise settlement information to the first display device and the second display device, respectively, to display the purchase merchandise settlement information to the customer and the clerk, respectively; if After receiving the settlement cancellation or settlement error message for the purchase commodity settlement information sent by either the customer or the clerk, the checkout process is suspended.

In some embodiments, the checkout processing module is configured to receive the fee payment information generated from the purchase commodity settlement information sent by the customer terminal; if it is determined that the fee payment information is correct, send the fee payment completion information to The first display device and the second display device, and the commodity to be settled is set to a paid state, so that the commodity to be settled passes the detection of the security detection device; if it is determined that the fee payment information is incorrect , The fee payment failure information is sent to the first display device and the second display device.

In some embodiments, a commodity recommendation module is used to obtain payment commodity settlement information corresponding to the customer terminal for successful payment, and obtain the commodity category and shopping frequency based on the purchase commodity settlement information; according to the commodity category, all The shopping frequency determines the recommended products and the purchase cycle, and determines the push time based on the purchase cycle; pushes the recommended products to the customer terminal based on the push time; after receiving the commodity discount information, it is determined whether the commodity discount information is The recommended commodities match, and if so, push the recommended commodities and the commodity preferential information to the customer terminal.

According to one or more embodiments of the present disclosure, there is provided a self-service settlement apparatus, including: a memory; and a processor coupled to the memory, the processor configured to execute based on instructions stored in the memory The method as described above.

According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium stores computer instructions, and the instructions execute the method as described above by a processor.

By applying the technical solution of the present disclosure, the combination of image recognition technology and weight detection technology to determine the information of the product to be tested can realize fast self-checkout, reduce the average length of a single transaction, improve the settlement efficiency, and reduce the waiting time for customer settlement. Improve the customer's shopping physical examination; can set up a supervision function, effectively avoid the economic losses caused by wrong settlement; the cost is relatively low, it can reduce the cashier staff and reduce the operating cost; through smart advertising technology to establish continuous contact with customers, Not only does it increase the customer experience, it can also increase the sales of supermarkets.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions in the embodiments or related technologies of the present disclosure, the drawings required for the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings in the following description are only For some of the disclosed embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without paying any creative labor.

1 is a schematic flowchart of some embodiments of a self-service settlement method provided by the present disclosure;

2 is a schematic flowchart of identifying the monitoring image in some embodiments of a self-service settlement method provided by the present disclosure;

FIG. 3 is a schematic flowchart of determining whether the commodity information to be settled matches the commodity to be identified in some embodiments of the self-service settlement method provided by the present disclosure;

4 is a schematic flowchart of payment fee processing in some embodiments of a self-service settlement method provided by the present disclosure;

5 is a schematic flowchart of recommending commodities in some embodiments of a self-checkout method provided by the present disclosure;

6 is a schematic block diagram of some embodiments of a self-service settlement device provided by the present disclosure;

7 is a schematic block diagram of other embodiments of a self-service settlement device provided by the present disclosure;

8 is a schematic block diagram of an image recognition module in some embodiments of a self-checkout device provided by the present disclosure;

9 is a schematic block diagram of still other embodiments of a self-service settlement device provided by the present disclosure.

detailed description

The technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the protection scope of the present disclosure.

The technical solutions of related technologies have shortcomings: the barcode scanning method requires consumers to actively cooperate and scan the products one by one to confirm, which is more cumbersome; the cost of the RFID method is higher, and if each low-profit product is attached with an RFID electronic tag, the cost is relatively high And it will cause certain environmental pollution. When the customer quickly passes through the RFID settlement channel, the label may be missed due to the speed too fast and the RFID label is blocked, causing economic losses to the supermarket. Therefore, a new technical solution for shopping settlement is needed.

FIG. 1 is a schematic flowchart of some embodiments of a self-service settlement method provided by the present disclosure. As shown in FIG. 1, the self-service settlement method includes steps 101-105.

Step 101: Obtain a monitoring image collected by an image collection device corresponding to a commodity to be settled placed on a settlement counter.

The image acquisition device may be a camera or the like. The pre-arranged image acquisition device can perform image acquisition on the goods to be settled placed on the settlement counter by the customer, and the goods to be settled can be beverages, snacks and the like.

Step 102: Identify the monitoring image to obtain the product information to be settled. The product information to be settled includes: the category and quantity of the product to be settled.

A variety of image recognition technologies can be used to identify the monitoring images, and the categories and quantities of the products to be settled can be identified from the monitoring images, such as 2 liters of Coca-Cola and 5 liters of Luhua peanut oil.

Step 103: Obtain the first weight corresponding to the commodity to be collected collected by the weight detection device provided on the settlement counter, and obtain the second weight corresponding to the commodity to be settled based on the commodity information to be settled.

The weight detection device provided on the checkout counter may be a weight sensor or the like. The unit weight corresponding to the identified category of the commodity to be settled can be determined in the commodity library, and the second unit weight of the commodity to be settled selected by the customer can be calculated based on the unit weight and the quantity.

Step 104: Determine whether the product information to be settled matches the product to be identified according to the preset matching judgment rule and the comparison result of the first weight and the second weight.

Step 105, if yes, obtain purchase commodity settlement information based on the commodity information to be settled, and perform checkout processing based on the purchase commodity settlement information. The purchase commodity settlement information includes: the category, quantity, and settlement price of the commodity to be settled. The unit price corresponding to the identified category of the commodity to be settled can be determined in the commodity library, and the settlement price of the commodity to be settled can be calculated according to the unit price and quantity.

The self-service settlement method in the above embodiment can be applied to shopping places such as convenience stores, supermarkets, etc., and is particularly suitable for scenarios where the customer consumes many people and the number of one-time purchases is small (generally less than five products). Provide fast and accurate self-service settlement, reduce the average length of a single transaction, reduce the waiting time of customers in queue, and improve customer satisfaction.

In some embodiments, a variety of image recognition methods may be used to identify the monitored image and obtain the product information to be settled. FIG. 2 is a schematic flowchart of identifying a monitoring image in some embodiments of a self-service settlement method provided by the present disclosure. As shown in FIG. 2, the self-service settlement method includes steps 201-205.

Step 201: Determine a first position corresponding to the commodity to be settled and a second position corresponding to the coordinate scale on the settlement counter in the monitoring image.

Step 202: Intercept the commodity image to be settled and the coordinate scale image in the monitoring image according to the first position and the second position.

Coordinate scales are provided on the checkout counter, for example, scales are provided on the checkout counter to provide comparison reference for the size of the goods to be calculated. When the commodity to be settled is placed on the settlement counter, the monitor image collected by the image collection device has the commodity to be settled and the coordinate scale. The first position and the second position can be determined in the monitoring image using the target detection model.

Target detection models include: Convolutional neural network model based on Faster RCNN (Faster Region Convolutional Neural Networks) algorithm. For example, construct a convolutional neural network model based on the Faster RCNN algorithm, obtain location information, images, and classification information such as bags, shoes, and pants in the existing monitoring image in advance, and manually annotate it as a training data set. The RCNN algorithm's convolutional neural network model is used for detection training to obtain the target detection network model.

Use the RPN (Region Proposal Network) network of the target detection network model to extract candidate regions where targets (commodities to be settled and coordinate scales) exist from the monitoring image. Extract feature vectors in the convolutional feature map, send the feature vectors of each candidate area to the classifier for classification, determine the type of the target, and determine the coordinates of the rectangular area containing the target, intercept the product image to be settled in the monitoring image And coordinate scale images.

Step 203: Determine the size information of the commodity to be settled based on the coordinate scale image and the commodity to be settled image.

Step 204: Identify the category of the commodity to be settled in the commodity to be settled image, and obtain the quantity corresponding to each commodity to be settled.

The type and number of commodities to which the commodities to be settled belong in the image of the commodities to be settled can be identified through the neural network model, and the size information of the commodities to be settled can be determined based on the coordinate scale image and the image of the commodities to be settled. For example, the fully connected layer of the convolutional neural network is established through the Softmax function, the features in the coordinate scale image and the product image to be settled are extracted, and the features are compared with the characteristics of various products, and the convolutional neural network calculates the pending settlement Commodity images belong to the confidence level of various commodity categories. The category of the commodity whose confidence level is greater than the preset threshold is taken as the category of the commodity to be settled, and the quantity corresponding to each commodity to be settled is counted.

A pooling layer can be set between each convolutional layer of the convolutional neural network, which can be a max pooling (maximum pooling) layer, which can effectively reduce the sampling rate of the picture, thereby improving recognition efficiency. You can discard the neural network unit without dropout, or use the batch normalization layer layer by layer, and only set the batch normalization layer after the last convolution layer, which can speed up the convergence speed of the recognition and avoid the disappearance of the gradient, thereby improving the recognition efficiency And accuracy.

Step 205: Determine the category of the commodity to be settled based on the size information of the commodity to be settled and the category of the commodity to which the commodity to be settled belongs.

The commodity to be settled in the commodity to be settled image and the coordinate scale in the coordinate scale image can be identified through the neural network model, and the true size of the commodity to be settled can be obtained according to the correspondence between the coordinate scale and the size of the commodity to be settled. For example, the neural network model identifies the commodity type of the commodity to be settled as potato chips, and the actual size of the commodity to be settled can be obtained based on the correspondence between the coordinate scale and the size of the commodity to be settled, and determined according to the commodity size information in the commodity database The category of commodities to be settled is large bags of potato chips.

In some embodiments, the unit weight of the commodity corresponding to the category of the commodity to be settled is obtained, and the second weight is obtained according to the commodity unit weight and quantity of the commodity to be settled. For example, if the category of commodities to be settled is identified as large bags of potato chips and the quantity is two bags, according to the product weight information in the commodity database, the unit weight of large bags of potato chips is 100 grams, and the second weight is determined to be 100 * 2 = 200 grams.

FIG. 3 is a schematic flowchart of judging whether information of a commodity to be settled matches a commodity to be identified in some embodiments of a self-service settlement method provided by the present disclosure. As shown in FIG. 3, the self-service settlement method includes steps 301-303.

Step 301: Obtain the difference between the first weight and the second weight.

Step 302: Determine whether the absolute value of the difference is less than a preset difference threshold.

In step 303, if yes, it is determined that the product information to be settled matches the product to be identified.

For example, if the preset difference threshold is 5 grams, the first weight collected by the weight detection device set on the check-out counter is 300 grams, and the second weight is 200 grams, then the absolute value of the difference between the first weight and the second weight If it is 100 grams, which is greater than the difference threshold of 5 grams, it is determined that the product information to be settled does not match the product to be determined, and an error message can be sent to customers, shop assistants, etc. When receiving the error message, the clerk can check the goods to be settled.

In some embodiments, the purchase commodity settlement information may be sent to the first display device and the second display device, respectively, to display the purchase commodity settlement information to the customer and the store clerk, respectively. If a settlement cancellation or settlement error message for the settlement information of the purchased goods sent by either the customer or the clerk is received, the checkout process is suspended. The settlement process can be changed from the traditional store clerk scanning one-dimensional bar code settlement to the "commodity detection identification settlement + store assistant assisted verification settlement" method, avoiding the tedious process of scanning the commodities one by one, and greatly reducing the settlement time.

The image acquisition device can continuously check the settlement counter, identify and detect the monitoring image every preset number of image frames (such as 3 image frames), obtain the information of the goods to be settled, and judge the information of the goods to be settled and the goods to be recognized If it matches, obtain the purchase commodity settlement information, and realize the effect of real-time synchronization of the settlement counter and the purchase commodity settlement information displayed on the display device.

For example, if a customer places five items A, B, C, D, and E on the checkout counter, the first display device and the second display device can display the purchase information of the five items in real time. When B is removed from the checkout counter , C products, the first display device, the second display device will be updated in real time to the A, D, E three kinds of goods purchase commodity settlement information. The customer can confirm the settlement information of the purchased goods on the first display device, or suspend this checkout. The store clerk can confirm the settlement information of the purchased goods on the second display device, if a settlement error is found, suspend the settlement and payment, re-arrange the goods to be settled, identify the goods to be settled, etc.

FIG. 4 is a schematic flowchart of payment fee processing in some embodiments of a self-service settlement method provided by the present disclosure. As shown in FIG. 4, the self-service settlement method includes steps 401-403.

Step 401: Receive fee payment information generated based on purchase commodity settlement information sent by a customer terminal.

There are many ways for customers to pay, such as card payment. You can also generate a payment QR code based on the purchase commodity settlement information and display it on the first display device, or send the purchase commodity settlement information to the customer terminal. The customer terminal can scan the payment QR code for online payment, or based on the purchase commodity settlement Information for online payment. The customer terminal may be the customer's mobile phone or the like.

In step 402, if it is determined that the fee payment information is correct, the fee payment completion information is sent to the first display device and the second display device. Set the goods to be settled to the paid state, so that the goods to be settled pass the detection of the safety detection device and start the next self-service settlement.

For example, if it is determined that the customer's payment information is correct, deductions are processed, and the status information of the product to be settled is set to the settled state in the product database, the customer can pass the product to be settled through the security detection device.

In step 403, if it is determined that the fee payment information is incorrect, the fee payment failure information is sent to the first display device and the second display device. If the commodity to be settled is not identified within the preset time, the commodity identification ends this time and the next self-service settlement is started.

FIG. 5 is a schematic flowchart of recommending commodities in some embodiments of a self-service settlement method provided by the present disclosure. As shown in FIG. 5, the self-service settlement method includes steps 501-504.

Step 501: Acquire the payment commodity settlement information corresponding to the customer terminal for successful payment, and obtain the commodity category and shopping frequency according to the purchase commodity settlement information.

Step 502: Determine recommended commodities and purchase cycle according to product category and shopping frequency, and determine the push time based on the purchase cycle.

Step 503: Push the recommended product to the customer terminal based on the push time.

Step 504: Receive the commodity preferential information, determine whether the commodity preferential information matches the recommended commodity, and if so, push the recommended commodity and the commodity preferential information to the customer terminal.

For example, after paying through a mobile phone, a customer establishes an association relationship between a mobile phone (such as a mobile phone number) and settlement information for purchased goods. Analyze the information such as the commodities and frequencies purchased by the customer corresponding to this mobile phone according to the settlement information of the historically purchased commodities, and determine the shopping category and shopping cycle of this customer. For example, if the shopping category is rice and the shopping cycle is 20 days, then use rice as a recommended product and push it once every 20 days. When the push time comes, the recommended rice information is pushed to the customer's mobile phone. If the received shopping discount information is a kind of rice promotion information, pushing such rice information and promotion information to the customer's mobile phone can realize intelligent advertisement recommendation.

In some embodiments, as shown in FIG. 6, the present disclosure provides a self-service settlement device 60, including: an image obtaining module 61, an image recognition module 62, a weight obtaining module 63, a matching judgment module 64, and a checkout processing module 65.

The image obtaining module 61 obtains the monitoring image collected by the image collecting device corresponding to the goods to be settled placed on the settlement counter. The image recognition module 62 recognizes the monitoring image and obtains the information of the goods to be settled; among them, the information of the goods to be settled includes: the category and quantity of the goods to be settled. The weight obtaining module 63 obtains the first weight corresponding to the commodity to be collected collected by the weight detection device provided on the settlement counter, and obtains the second weight corresponding to the commodity to be settled based on the commodity information to be settled.

The matching judgment module 64 judges whether the commodity information to be settled matches the commodity to be identified according to the preset matching judgment rule and the comparison result of the first weight and the second weight. If the commodity information to be settled matches the commodity to be determined, the checkout processing module 65 obtains the settlement information of the purchased commodity based on the commodity information to be settled, and performs the settlement processing according to the settlement information of the purchased commodity; where the settlement information of the purchased commodity includes: the category of the commodity to be settled, Quantity and settlement price, etc.

In some embodiments, as shown in FIG. 8, the image recognition module 62 includes: a position determination unit 621, an image interception unit 622, a size acquisition module 623 and a recognition processing unit 624.

The position determination unit 621 determines the first position corresponding to the commodity to be settled and the second position corresponding to the coordinate scale on the settlement counter in the monitoring image. The image intercepting unit 622 intercepts the commodity image to be settled and the coordinate scale image in the monitoring image according to the first position and the second position. The size obtaining module 623 determines the size information of the commodity to be settled based on the coordinate scale image and the commodity to be settled image. The recognition processing unit 624 recognizes the commodity type to which the commodity to be settled belongs in the commodity image to be settled, and obtains the quantity corresponding to each commodity to be settled. The identification processing unit 624 determines the category of the commodity to be settled based on the size information of the commodity to be settled and the commodity category to which the commodity to be settled belongs.

The position determination unit 621 uses the target detection model to determine the first position and the second position in the monitoring image; wherein, the target detection model includes: a convolutional neural network model based on the Faster RCNN algorithm. The recognition processing unit 624 establishes the fully connected layer of the convolutional neural network through the Softmax function, and calculates the confidence that the commodity to be settled belongs to each commodity type through the convolutional neural network. The recognition processing unit 624 regards the category of goods whose confidence is greater than a preset threshold as the category of goods to be settled. A pooling layer is provided between each convolutional base layer of the convolutional neural network, and a batch normalization layer is provided after the last convolutional layer.

In some embodiments, the weight obtaining module 63 obtains the unit weight of the commodity corresponding to the category of the commodity to be settled, and the weight obtaining module 63 obtains the second weight according to the unit weight and quantity of the commodity to be settled. The matching determination module 64 obtains the difference between the first weight and the second weight, determines whether the absolute value of the difference is less than a preset difference threshold, and if so, determines that the product information to be settled matches the product to be identified.

The checkout processing module 65 sends the purchase commodity settlement information to the first display device and the second display device, respectively, to display the purchase commodity settlement information to the customer and the store clerk, respectively. If a settlement cancellation or settlement error message for the settlement information of the purchased goods sent by any one of the customer and the clerk is received, the checkout processing module 65 suspends this checkout processing.

The checkout processing module 65 receives the fee payment information generated from the purchase commodity settlement information sent by the customer terminal, and if it is determined that the fee payment information is correct, then sends the fee payment completion information to the first display device and the second display device, and the payment The commodity is set to the paid state, so that the commodity to be settled passes the detection of the safety detection device. If it is determined that the fee payment information is incorrect, the checkout processing module 65 sends the fee payment failure information to the first display device and the second display device.

In some embodiments, as shown in FIG. 7, the commodity recommendation module 66 obtains the settlement information of the purchased commodity corresponding to the successful payment of the fee and corresponds to the customer terminal, and obtains the commodity category and the shopping frequency based on the purchased commodity settlement information. The commodity recommendation module 66 determines the recommended commodity and the purchase cycle based on the commodity category and shopping frequency, and determines the push time based on the purchase cycle; the recommended commodity is pushed to the customer terminal based on the push time. The commodity recommendation module 66 receives the commodity discount information, determines whether the commodity discount information matches the recommended commodity, and if so, pushes the recommended commodity and the commodity discount information to the customer terminal.

9 is a schematic block diagram of still other embodiments of a self-service settlement device provided by the present disclosure. As shown in FIG. 9, the device may include a memory 91, a processor 92, a communication interface 93, and a bus 94. The memory 91 is used to store instructions, and the processor 92 is coupled to the memory 91. The processor 92 is configured to execute the self-service settlement method based on the instructions stored in the memory 91.

The memory 91 may be a high-speed RAM memory, a non-volatile memory (non-volatile memory), etc. The memory 91 may also be a memory array. The storage 91 may also be divided into blocks, and the blocks may be combined into a virtual volume according to certain rules. The processor 92 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the self-service settlement method of the present disclosure.

In some embodiments, the present disclosure also provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and when the instructions are executed by the processor, the self-service settlement method according to any of the above embodiments is implemented. Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code .

The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagram and a combination of the flow and / or block in the flowchart and / or block diagram may be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processing machine, or other programmable data processing device to produce a machine that enables the generation of instructions executed by the processor of the computer or other programmable data processing device An apparatus for realizing the functions specified in one block or multiple blocks of one flow or multiple flows of a flowchart and / or one block or multiple blocks of a block diagram. .

The self-service settlement method, device and storage medium in the above embodiments determine the information of the product to be tested through a combination of image recognition technology and weight detection technology, which can realize fast self-checkout, reduce the average length of a single transaction, improve settlement efficiency, The waiting time for customer settlement is improved, and the customer's shopping physical examination is improved; the supervision function can be set to effectively avoid the economic loss caused by incorrect settlement; no additional auxiliary verification settlement method is required, and no tools such as RFID price tags are used, and the cost is relatively low ; Can reduce cashier staff and reduce operating costs; through smart advertising technology to establish continuous contact with customers, not only increases the customer experience, but also can increase supermarket sales.

The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the method according to the present disclosure. Thus, the present disclosure also covers the recording medium storing the program for executing the method according to the present disclosure.

The description of the present disclosure is given for the sake of example and description, and is not exhaustive or limits the present disclosure to the disclosed form. Many modifications and changes will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better explain the principles and practical applications of the present disclosure, and enable those of ordinary skill in the art to understand the present disclosure to design various embodiments with various modifications suitable for specific uses.

Claims

A self-service settlement method, including:

Obtain the monitoring image collected by the image collection device corresponding to the goods to be settled placed on the settlement counter;

Identify the monitoring image to obtain the information of the commodity to be settled; wherein the information of the commodity to be settled includes: the category and quantity of the commodity to be settled;

Obtaining a first weight corresponding to the commodity to be collected collected by a weight detection device provided on a settlement counter, and obtaining a second weight corresponding to the commodity to be settled based on the commodity information to be settled;

Judging whether the commodity information to be settled matches the commodity to be determined according to a preset matching judgment rule and a comparison result of the first weight and the second weight;

If yes, the purchase commodity settlement information is obtained based on the commodity commodity information to be settled, and checkout processing is performed based on the purchase commodity settlement information; wherein, the purchase commodity settlement information includes: the category, quantity, and settlement price of the commodity to be settled.
The method according to claim 1, wherein the identifying the monitoring image and obtaining the product information to be settled includes:

Determining a first position corresponding to the commodity to be settled and a second position corresponding to the coordinate scale on the settlement counter in the monitoring image;

Intercept the image of the commodity to be settled and the coordinate scale image in the monitoring image according to the first position and the second position;

Determine the size information of the commodity to be settled based on the coordinate scale image and the commodity to be settled image;

Identify the category of the commodity to be settled in the commodity to be settled image, and obtain the quantity corresponding to each commodity to be settled;

The category of the commodity to be settled is determined based on the size information of the commodity to be settled and the type of commodity to which the commodity to be settled belongs.
The method according to claim 2, wherein the determining the first position corresponding to the commodity to be settled and the second position corresponding to the coordinate scale on the settlement counter in the monitoring image comprises:

The first position and the second position are determined in the monitoring image by using a target detection model; wherein, the target detection model includes: a convolutional neural network model based on the Faster RCNN algorithm.
The method according to claim 2, wherein the commodity category to which the commodity to be settled identified in the commodity image to be settled includes:

Establish a fully connected layer of a convolutional neural network through the Softmax function, and calculate the confidence that the commodity to be settled belongs to each commodity type through the convolutional neural network;

A commodity type with a confidence greater than a preset threshold is used as the commodity type of the commodity to be settled.
The method of claim 4, wherein:

A pooling layer is provided between each convolutional base layer of the convolutional neural network, and a batch normalization layer is provided after the last convolutional layer.
The method of claim 1, the obtaining the second weight corresponding to the commodity to be settled based on the commodity information to be settled comprises:

Obtain the unit weight of the commodity corresponding to the category of the commodity to be settled;

The second weight is obtained according to the unit weight and quantity of the commodity to be settled.
According to the method of claim 6, judging whether the commodity information to be settled matches the commodity to be determined according to a preset matching judgment rule and a comparison result of the first weight and the second weight includes:

Obtaining the difference between the first weight and the second weight;

Determine whether the absolute value of the difference is less than a preset difference threshold;

If yes, it is determined that the product information to be settled matches the product to be identified.
The method according to claim 1, wherein the checkout processing according to the purchase commodity settlement information includes:

Sending the purchase commodity settlement information to the first display device and the second display device, respectively, to display the purchase commodity settlement information to the customer and the store clerk, respectively;

If a settlement cancellation or settlement error message for the purchase merchandise settlement information sent by either the customer or the clerk is received, the checkout process is suspended.
The method according to claim 8, wherein the checkout processing according to the purchase commodity settlement information further comprises:

Receiving the fee payment information generated according to the purchase commodity settlement information sent by the customer terminal;

If it is determined that the fee payment information is correct, the fee payment completion information is sent to the first display device and the second display device, and the goods to be settled are set to a paid state, so that the goods to be settled Commodities pass the inspection of the safety inspection device;

If it is determined that the fee payment information is incorrect, the fee payment failure information is sent to the first display device and the second display device.
The method of claim 9, further comprising:

Obtaining the payment settlement information of the purchase terminal corresponding to the customer terminal and obtaining the commodity category and shopping frequency according to the purchase commodity settlement information;

Determine recommended products and purchase cycles based on the product category and the shopping frequency, and determine the push time based on the purchase cycle;

Push the recommended product to the customer terminal based on the push time;

After receiving the commodity discount information, it is determined whether the commodity discount information matches the recommended commodity, and if so, the recommended commodity and the commodity discount information are pushed to the customer terminal.
A self-service settlement device, including:

The image obtaining module is used to obtain the monitoring image collected by the image collecting device corresponding to the goods to be settled placed on the settlement counter;

The image recognition module is used to recognize the monitoring image and obtain the information of the commodity to be settled; wherein, the information of the commodity to be settled includes: the category and quantity of the commodity to be settled;

A weight obtaining module, configured to obtain a first weight corresponding to the commodity to be collected collected by a weight detection device provided on a settlement counter, and obtain a second corresponding to the commodity to be settled based on the commodity to be settled information weight;

The matching judgment module is configured to judge whether the commodity information to be settled matches the commodity to be determined according to a preset matching judgment rule and a comparison result of the first weight and the second weight;

The checkout processing module is configured to obtain settlement information of the purchased commodity based on the commodity information to be settled based on the commodity information to be settled and perform checkout processing according to the settlement information of the purchased commodity; Purchase settlement information includes: the category, quantity and settlement price of the commodity to be settled.
A self-service settlement device, including:

A memory; and a processor coupled to the memory, the processor configured to perform the method of any one of claims 1 to 10 based on instructions stored in the memory.
A computer-readable storage medium storing computer instructions, the instructions being executed by a processor to perform the method of any one of claims 1 to 10.