WO2019033635A1

WO2019033635A1 - Purchase settlement method, device, and system

Info

Publication number: WO2019033635A1
Application number: PCT/CN2017/115444
Authority: WO
Inventors: 吴一黎
Original assignee: 图灵通诺（北京）科技有限公司
Priority date: 2017-08-16
Filing date: 2017-12-11
Publication date: 2019-02-21

Abstract

The present invention discloses a purchase settlement method, device, and system, and belongs to the technical field of computers. The purchase settlement method comprises: identifying a pre-registered customer to obtain identity information of the customer, wherein the identity information comprises facial data and a payment account of the customer; tracking, in a shopping center in real time, the customer whose identity information has been obtained, so as to obtain a position of the customer; and determining whether the position of the customer is consistent with a position of goods in the shopping center, and if so, associating the customer with a taking or returning action related to the goods, and after the taking or returning action and the goods corresponding to the taking or returning action are identified, generating a shopping list of the customer; and performing purchase settlement according to the shopping list. The invention also discloses a purchase settlement device comprising a top camera, a front-facing camera, a downward camera, and a server. The invention also discloses the purchase settlement system comprising a client and a purchase settlement device. The above-described technical solution of the invention solves the problems that adhering RFID tags requires significant labor and the RFID tags are likely to be damaged.

Description

Settlement method, device and system

Technical field

The invention belongs to the technical field of computers, and in particular relates to a settlement method, device and system.

Background technique

When a customer sees a product he likes or needs in a shopping place such as a supermarket or a store, he/she needs to settle with the operator of the shopping place to obtain the product. It is usually settled by queuing at the checkout counter.

With the development of artificial intelligence technology, the settlement method has also changed. For example, after entering the shopping place, picking up your own goods, you don’t have to wait in line at the receiving station to wait for the checkout, you can leave immediately, commonly known as “get it right away”. the way.

In the prior art, it is mainly based on RFID (Radio Frequency Identification) technology to realize the "get it right away" solution. When applying, a small radio frequency module that does not require a battery is attached to the product. When the product passes through a settlement station (or settlement area) provided with an RFID detection setting, the settlement station transmits a wireless signal to the commodity, and the radio frequency small module receives After the signal, a signal is sent back to the settlement counter, and the feedback signal carries the ID information of the commodity, and the settlement counter generates a bill and settles the bill accordingly. This method has the following drawbacks: due to the need to attach a small RF module to each item, this is extremely labor intensive and costly for the staff of the shopping place, and if the RF small module is dropped from the product or is itself damaged or Man-made damage, the counter can not identify the goods, will cause losses to the business. In addition, some metal products are affixed with RFID, and there may be problems with the signal being shielded.

Summary of the invention

In order to solve the problem that the pasting RFID tag existing in the prior art has a large workload and the RFID tag is easily damaged, the present invention provides a settlement method, which includes: Step S1, identifying a pre-registered customer to obtain the identity information of the customer. The identity information includes the customer's face data and the payment account number; in step S2, the customer who has acquired the identity information is tracked in real time in the shopping place to obtain the customer Position S3, determining whether the location of the customer is consistent with the location of the product in the shopping place, and if so, correlating the customer with the take-in or put-back action of the product, and identifying or taking back the action And taking or returning the product for which the action is directed, generating a shopping list of the customer; and in step S4, performing settlement according to the shopping list.

In the settlement method as described above, preferably, in step S3, it is determined whether the location of the customer is consistent with the location of the product in the shopping place, and specifically includes: facing forwardly mounted on the shelf carrying the product. The position of the camera indicates the position of the product, and when the forward camera captures the customer identity information indicated by the picture containing the customer and the identity information acquired in step S1 is the same, the location of the customer and the goods in the shopping place are determined. The position is the same.

In the settlement method as described above, preferably, in step S3, identifying the take-in or put-back action includes: obtaining a multi-frame consecutive hand image of the customer before the shelf carrying the product, for the multi-frame The coherent hand image establishes a hand movement trajectory on the time axis; when it is detected that the movement trajectory of the hand is moved inward from the preset virtual action boundary line and the item is in the hand, the action is recognized as a reversal action When it is detected that the movement track of the hand is moved outward from the virtual action boundary line and the product is in the hand, the action is recognized as a take action; wherein the virtual action boundary line is away from the shelf Direction, the virtual action boundary is a direction close to the shelf.

In the settlement method as described above, preferably, in step S3, identifying the product for which the action is taken or returned, specifically comprising: S31, performing target detection on the acquired multi-frame hand image containing the commodity to obtain the corresponding a plurality of rectangular area images, the rectangular area image is an image corresponding to a rectangular area including the commodity, and the multi-frame hand image is in one-to-one correspondence with the plurality of cameras; S32, according to the plurality of rectangular area images and the pre-trained first-class classification The model corresponds to obtaining a plurality of primary classification results, and the pre-trained primary classification model is a model based on a convolutional neural network image recognition technology and trained by all commodities in the shopping place, according to a plurality of primary classification results and a pre-trained one. The hierarchical linear regression model obtains a primary classification result of the multi-frame hand image; S33, the first classification result is used as the first classification result; and S34, the first classification result is used as the commodity to be identified.

In the settlement method as described above, preferably, after step S32, before step S34, further comprising: S35, if the primary classification result is a similar commodity, according to the plurality of rectangular region images and the pre-trained secondary classification model Corresponding to obtaining multiple sub-category results, and then obtaining secondary classification results of multi-frame hand images according to multiple sub-category results and pre-trained two-level linear regression models, and The second classification result is used as the first classification result, and the secondary classification model is a model based on the image recognition technology architecture of the convolutional neural network in advance and trained by the commodity in the similar commodity group in the shopping place, otherwise step S33 is performed.

The other party also provides a settlement device, which includes: a registration module, configured to receive identity information input by the customer and obtain identity information of the customer who wants to enter the shopping place when registering; a real-time tracking module is connected to the registration module, And a method for obtaining a location of the customer by acquiring the identity information by the registration module in a shopping mall; generating a shopping list module, and connecting with the real-time tracking module, configured to determine the obtained by the real-time tracking module Whether the location of the customer is consistent with the location of the product in the shopping place, and if so, the customer is associated with the take-in or put-back action of the product, and recognizes the take-in or put-back action and takes or puts back the action After the targeted product, a shopping list of the customer is generated; and a settlement module is connected to the generated shopping list module for settlement according to the shopping list generated by the generated shopping list list module.

In the settlement device as described above, preferably, the generating the shopping list module includes: an associating unit configured to indicate the position of the product with a position of the forward facing camera photographed on the shelf on which the article is carried. When the forward camera captures that the customer identity information indicated by the picture containing the customer is the same as the identity information acquired by the registration module, it is determined that the location of the customer is consistent with the location of the product in the shopping place; the action recognition unit uses Obtaining a multi-frame coherent hand image of the customer before the shelf carrying the product, establishing a hand motion trajectory on the time axis for the multi-frame consecutive hand image; when detecting the movement trajectory of the hand is pre- When the virtual action boundary line is moved inwardly and the product is in the hand, the action is recognized as a return action; when it is detected that the motion track of the hand is moved outward from the virtual action boundary line and the product is in the hand, Identifying the action as a take action; wherein the virtual action boundary is a direction away from the shelf, the virtual action boundary The inside is a direction close to the shelf; the item identification unit is configured to identify the item for which the take-in or put-back action is directed; and the shopping list generating unit, according to the identity information of the customer confirmed by the associated unit, The take-in or put-back action recognized by the motion recognition unit, the product for which the product identification unit recognizes the take-in or put-back action, generates the customer's shopping list.

In the settlement device as described above, preferably, the item identification unit includes: a target detection sub-unit, configured to perform target detection according to the multi-frame hand image containing the item acquired by the action recognition unit to acquire multiple a rectangular area image, the rectangular area image is an image corresponding to a rectangular area including an item, and the multi-frame hand image is in one-to-one correspondence with the plurality of cameras; the first classification sub-unit, The method is configured to obtain a plurality of primary classification results according to a plurality of rectangular area images and a pre-trained first-level classification model, and the pre-trained first-level classification model is an image recognition technology architecture based on a convolutional neural network and passes through all the commodities in the shopping place. a trained model, which obtains a first-class classification result of a multi-frame hand image according to a plurality of primary classification results and a pre-trained first-order linear regression model; and a confirmation unit, configured to use the first-level classification result as a first classification result; And a result identification unit for using the first classification result as the item to be identified

Yet another aspect provides a settlement device, comprising: a top camera for photographing downward from the top of the shopping place to track the customer who has acquired the identity information in the shopping place in real time; the front camera for facing the shelf The front is photographed to obtain a picture of the customer located in front of the shelf carrying the merchandise; the lower camera is used for downward shooting to obtain the customer's hand image; the processor; and a memory for storing instructions executable by the processor; The processor is configured to: identify a pre-registered customer to obtain identity information of the customer, the identity information includes a customer's facial data and a payment account; and control the top camera to track the acquired identity information in real time. Obtaining the location of the customer; determining whether the location of the customer is consistent with the location of the product in the shopping venue obtained by controlling the forward camera, and if so, correlating the customer with the take-in or return action of the product; And taking or retrieving the action according to the hand image acquired by the lower camera and taking or returning After the work for which goods to produce this customer's shopping list; be settled according to the shopping list list.

Still another aspect provides a settlement system, including: a client and a settlement device; the client is configured to receive identity information input by a customer when registering, and send the settlement information to the settlement device and the settlement device List; the settlement device is the above settlement device.

The beneficial effects brought by the above technical solutions in the embodiments of the present invention are as follows:

Low operating costs: saves the workload of pasting RFID compared to RFID solutions.

Wide range of applications: Applicable to any product, not subject to the properties of the product, material and other attributes.

User experience is good: customers can get relevant information in the first time after taking the goods.

DRAWINGS

FIG. 1 is a schematic flowchart diagram of a settlement method according to an embodiment of the present invention;

2 is a schematic flowchart of a settlement method of an image recognition technology based on a convolutional neural network according to an embodiment of the present invention;

3 is a schematic flowchart diagram of another image recognition method based on a convolutional neural network according to an embodiment of the present invention;

4 is a schematic structural diagram of a settlement apparatus according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a shelf for a settlement device according to an embodiment of the present invention.

Detailed ways

The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

An embodiment of the present invention provides a settlement method. Referring to FIG. 1, the method includes the following steps:

In step S1, the pre-registered customer is identified to obtain the identity information of the customer.

Specifically, before the customer enters the shopping place, such as a store or a supermarket, an App (Application, application software) corresponding to the settlement method is installed on the mobile communication device, such as a mobile phone or a tablet computer, to complete the registration and registration of the customer. The customer identification information (ID) to be collected includes but is not limited to: customer's face data, payment account, payment account can be a bank account, or a third party payment account, such as Yu'ebao, WeChat payment, QQ wallet, Jingdong Wallets, etc., can also include name, mobile number, ID number, gender, occupation. The customer registration can also be completed by a small program (or WeChat applet) corresponding to the settlement method in WeChat, and the customer registration can be completed by paying attention to the WeChat public number corresponding to the settlement method. Then, the customer scans the QR code at the door of the shopping place or the store scans the QR code generated by the App on the mobile communication device held by the customer through the App or applet or the public number to realize the verification of the customer ID by the store, that is, the customer is identified as One of the registered users obtains the identity information corresponding to the customer, in other words, who is the customer who enters the shopping place. After obtaining the identity information of the customer, the door lock of the shopping place can be automatically opened, and the customer opens the door after the door is automatically opened or pushes in or out or pushes, and then enters the shopping place to select the goods. If the customer does not complete the registration, the customer can be identified as a non-registered user, the door lock in the shopping place continues to be closed, and the customer cannot enter the shopping place. In other embodiments, a biometric identifier, such as a fingerprint recognizer and a face recognizer, may be disposed at the entrance of the shopping place, and the identification of the customer ID is realized by the biometric technology shopping place, and the customer collects when registering on the App. Customer's biological data, such as fingerprint data. The door of the shopping venue can be opened by the staff member after obtaining the customer identity information.

Step S2, tracking the customer who has acquired the identity information in real time in the shopping place to obtain the customer's position.

The customer needs to identify the identity when entering the shopping place, and then the identified customer moves freely in the shopping place, and takes and returns the product. In order to keep track of the identity information of each customer in the shopping place, it is necessary to keep track of the trajectory of the customer in the shopping place after the customer confirms the identity when entering the shopping place, that is, the customer needs to be positioned in real time.

Specifically, a top camera is deployed on the top of the shopping place, which captures the video stream moved by the customer in the shopping place, and analyzes the video stream to compare the difference between the front and back frames in the video stream to realize the customer in the shopping place. Real-time positioning within. Multiple top cameras can be deployed on the ceiling to cover the entire storefront, with the top camera shooting down diagonally. When the store enters the store, the identity of the customer has been identified. After entering the store, the top camera on the ceiling is photographed from top to bottom, the image of the customer is collected in real time, and the image is bound to the identity information of the customer, that is, The identity information of the customers moving in the store. As the customer moves to a different location within the shopping venue, the other top cameras will keep track of the customer and achieve positioning of the customer within the shopping venue.

In step S3, if the location of the customer matches the location of the product in the shopping place, the customer is associated with the take-in or put-back action related to the product, and after identifying the product for which the action is taken, Generate a shopping list for this customer.

Specifically, after entering the store, the customer will move in the store. When the customer encounters the product he likes, he will stop in front of the shelf carrying the product, and then take the action to indicate that the product belongs to the item to be purchased, or put The action is returned to indicate that the item does not belong to the item to be purchased. Since the current location of the customer can be obtained through step 102, if the current location is consistent with the location of the product, the person who takes the action on the shelf or puts it back is marked as the customer, in other words, the customer and the shelf. The pick-up or put-back action related to the merchandise is associated to know which customer has taken or put back the merchandise on the shelf before the shelf. After identifying the item for which the item targeted for the action is taken or returned, a shopping list corresponding to the customer can be generated. The position of the product can be indicated by the position of the camera photographed on the shelf facing forward. When the customer selects the product before moving to the shelf, the camera shooting toward the front will take a picture of the customer, if the customer information contained in the picture In accordance with the identity information of the identified customer, it is determined that the location of the customer is consistent with the location of the product. When the customer applies the take-up action to the product, the customer's shopping list will correspondingly increase the product. When the customer applies the put-back action to the product, the item will be correspondingly subtracted from the shopping list, that is, the shopping list will be based on The customer's take or put back action is updated in real time.

The method of how to judge whether the action of the product is taken or returned can be as follows:

According to the collected multi-frame continuous hand image of the customer in front of the shelf, a hand movement track is established on the time axis, and according to the hand movement track, it is determined whether the customer applies the taking action to the product or puts back the action. For example, the camera is deployed on the shelf, and the shooting angle is downward, so that the shooting range covers the shelf, and the number of the preferred cameras is plural. This ensures that the shooting is performed from multiple angles, and the accuracy of the product identification is improved. The camera captures multiple frames of images per second, such as 30 frames. The hand image of the customer captured by the camera is detected frame by frame, and the position of the hand in each hand image is marked and saved, repeating frame by frame. The foregoing operation, so that a movement track of the hand is obtained on the time axis, not only the position of the hand in each frame of the image is obtained, but also whether the hand holds the product and the product is determined according to the hand image of a certain frame. The category is confirmed.

Specifically, a virtual action boundary line is marked in the hand image, and the action boundary line is spaced apart from the shelf by a certain distance, such as 10 cm and 20 cm. When it is detected that the movement trajectory of the hand moves from the position away from the shelf through the action boundary line to the position close to the shelf, simply speaking, moving from the action boundary line to the action boundary line, and the hand has the commodity, the hand is considered The action of the part is the reversal action of the product; when it is detected that the movement track of the hand moves from the position close to the shelf through the action boundary line to the position away from the shelf, simply moves from the action boundary line to the action boundary line, and If you have a product in your hand, you will think that the hand movement is the taking action of the product.

The camera continuously collects video data, and captures multiple frames of hand images per second, such as tens of frames. It can take a frame of hand images every second, intercept the hand position, and classify (or identify) the products in the opponent. This can be achieved by pre-training the following classification model.

Referring to FIG. 2, the method for identifying the product for which the action is taken or returned may be as follows:

Step S31, performing target detection on the multi-frame hand image containing the product to acquire a plurality of rectangular area images corresponding to the rectangular area including the product, the multi-frame hand image and the plurality of cameras. correspond.

Specifically, when the target image is detected by the target image, a rectangular frame (or rectangular area) including the product is drawn on the hand image, and the image corresponding to the rectangular frame is an image for classifying the product. In order to acquire a multi-frame hand image, a plurality of cameras are arranged, which may be arranged directly above the merchandise, at this time from below; or may be arranged obliquely above the merchandise, at this time obliquely downward The merchandise is photographed; it may be partially arranged directly above the merchandise, and the other portion is placed obliquely above the merchandise. It should be noted that the distance between the cameras and the ground may be equal or not equal, which is not limited in this embodiment.

Step S32, correspondingly acquiring a plurality of primary classification results according to the plurality of rectangular area images and the pre-trained first-class classification model, and the pre-trained first-level classification model is an image recognition technology architecture based on a convolutional neural network and is all in the shopping place. The model of commodity training acquires the primary classification result of the multi-frame hand image according to the plurality of primary classification results and the pre-trained first-order linear regression model.

Specifically, the data collection data set is collected in advance, and the process of collecting data includes: 1) taking a photo of all the goods in the shopping place from various angles and in each posture to obtain a large number of photos. 2) Then mark these photos: mark the location, size, and category of the items in the photo. The data included in the data set refers to the aforementioned photos and the annotations made on these photos. The first-level classification model is a model based on the convolutional neural network image recognition technology architecture, and the first-level classification model is trained using the data of all commodities in the shopping place, and the training can be performed by gradient descent.

The trained first-level classification model classifies the products in each rectangular area image to obtain the primary classification result, which is an n-dimensional vector, where n represents the total quantity of goods in the shopping place, and each element in the vector The meaning indicates that the primary classification model considers that the commodity to be classified belongs to each of the n commodities, and which element in the vector has the largest value, which means that the model considers that the commodity to be classified is the commodity corresponding to the commodity. When the rectangular area image is five, the number of primary classification results is five n-dimensional vectors.

When training the first-level classification model, the primary classification result output by the primary classification model is used as the input of the first-order linear regression model, and the correct classification of the commodities contained in the hand image corresponding to the primary classification result is used as the output of the first-order linear regression model. In order to train a linear regression model. The trained first-order linear regression model performs data fusion on multiple primary classification results to obtain a first-level classification result, which indicates that the first-order linear regression model predicts which category of the goods in the shopping place in the image.

In step S33, the first classification result is used as the first classification result.

There are many kinds of goods in the shopping place, and there are some goods that are similar in appearance and visually confusing, and these goods are called similar goods, such as the yellow marshal apple and the yellow snow pear. If the individual commodity to be classified is a similar commodity, the primary classification model is difficult to accurately classify the commodity, such as mingling Huang Yuanshuai's apple with the yellow snow pear, and classifying Huang Yuanshuai's apple as Yellow snow pear, so referring to FIG. 3, after step S32, the following step S35 needs to be performed, otherwise step S33 is performed, that is, the first classification result is directly used as the first classification result for settlement.

Specifically, in step 35, if the primary classification result is a similar commodity, according to the plurality of rectangular region images and the pre-trained secondary classification model, multiple secondary classification results are obtained correspondingly, and then according to the multiple secondary classification results and the advance The trained secondary linear regression model obtains the secondary classification results of the multi-frame hand image, and uses the secondary classification result as the first classification result. The secondary classification model is the image recognition technology architecture based on the convolutional neural network in advance and A model of merchandise training in a similar group of goods in a shopping establishment.

Specifically, the secondary classification model is trained using the data of similar commodities in the data set established in step S32, and the training can be performed by gradient descent. The difference between the secondary classification model and the primary classification model is that the data used in the training is different. The data used by the primary classification model is the data of all the commodities in the shopping place, and the data used by the secondary classification model is the similar commodity data in the shopping place. .

The trained secondary classification model classifies the products in each rectangular area image to obtain the secondary classification result. The secondary classification result is also an m-dimensional vector. The meaning of each element in the vector indicates that the secondary classification model considers The probability that a single item to be classified belongs to each of the m similar items. When the rectangular area image is five, the number of secondary classification results is five m-dimensional vectors, m is less than or equal to n, and represents the total number of similar items in the shopping place.

In practice, there are multiple groups of similar goods in the shopping place, such as a group of similar goods including Huang Yuanshuai apple and yellow snow pear, and another group of similar products including bulk salt and bulk sugar; another group of similar products include Alkali noodles and flour. A secondary classification model can be trained for all groups of similar commodities. In order to further improve the accuracy of classification of commodities, a secondary classification model is trained for each group of similar commodities. At this time, if the primary classification result is a similar commodity, then the The secondary classification model corresponding to the primary classification result.

When the secondary classification model is to be trained, the secondary classification result output by the secondary classification model is used as the input of the secondary linear regression model, and the correct classification of the commodities included in the corresponding image of the secondary classification result is used as the secondary linear regression model. Output to train a two-level linear regression model. The trained secondary linear regression model performs data fusion on multiple sub-category results, and obtains a secondary classification result, which is used as the first classification result. The secondary classification result represents the secondary linear regression model prediction image. The item is which category of the item in the shopping place.

In step S34, the first classification result is taken as the commodity to be identified.

After the first classification result is obtained, and the commodity price corresponding to the first classification result is obtained, the fee for the customer selected product is determined.

In step S4, when the customer leaves the shopping place, the settlement is performed according to the customer's shopping list.

After the customer selects the product, the customer leaves the shopping place through the door of the shopping place, and when passing through the door of the shopping place from the inside to the outside, it is judged that the customer is in the state of leaving the shopping place, and is settled according to the customer's shopping list, such as input from the customer registration. The fee corresponding to the shopping list is deducted from the payment account.

In order to facilitate the customer to verify the purchased goods, the identification results will be sent to the customer in real time. If the identification result of each product is uploaded to the cloud server, then the cloud server delivers the recognition result to the app installed by the customer's mobile phone, and the app adds the recognition result to the virtual shopping cart to generate a shopping list, thereby taking the goods or After returning the goods, let the customer know the first time. When the customer comes to the store and intends to leave, the final payment link is completed at the store entrance.

Referring to FIG. 4, another embodiment of the present invention provides a settlement apparatus, including:

The registration module 401 is configured to receive identity information input by the customer and obtain identity information of the customer who wants to enter the shopping place when registering.

The real-time tracking module 402 is connected to the registration module 401 for tracking the customer who obtains the identity information by the registration module in the shopping place in real time to obtain the location of the customer.

The generated shopping list module 403 is connected to the real-time tracking module 402, and is configured to determine whether the location of the customer acquired by the real-time tracking module is consistent with the location of the product in the shopping place, and if the content is consistent, the customer and the product are taken or placed. Retrieving the action association, and generating a shopping list of the customer after identifying the item taken or putting back the action and taking or returning the action;

The settlement module 404 is coupled to the generated shopping list module 403 for settlement based on the shopping list generated by the generated shopping list module 404.

Specifically, the generated shopping list module 403 includes: an associating unit configured to indicate the position of the product with the position of the forward facing camera photographed on the shelf mounted on the product, when the front camera captures the image represented by the customer When the customer identity information is the same as the identity information obtained by the registration module, the location of the customer is determined to be consistent with the location of the product in the shopping place; the action recognition unit is configured to obtain a multi-frame consecutive hand image of the customer before the shelf of the product. , for a multi-frame coherent hand image to establish a hand movement trajectory on the time axis; when detecting the movement trajectory of the hand is pre- When the virtual action boundary line is moved inward and the product is in the hand, the action is recognized as a return action; when it is detected that the motion track of the hand is moved outward from the virtual action boundary line and the product is in the hand, The action recognition is a take action; wherein, the virtual action boundary line is away from the shelf direction, the virtual action boundary line is a direction close to the shelf; the product identification unit is configured to identify the product targeted for taking or returning the action; The shopping list generating unit generates the shopping list of the customer based on the identity information of the customer confirmed by the association unit, the take-in or put-back action recognized by the action recognition unit, and the product for which the product identification unit recognizes the take-in or put-back action. .

Specifically, the item identification unit includes: performing target detection according to the multi-frame hand image containing the item acquired by the action recognition unit to acquire a plurality of rectangular area images corresponding to the rectangular area including the product, The multi-frame hand image is in one-to-one correspondence with the plurality of cameras; the first classification sub-unit is configured to obtain a plurality of primary classification results according to the plurality of rectangular area images and the pre-trained first-class classification model, and the pre-trained first-class classification The model is a model based on the image recognition technology architecture of the convolutional neural network and trained by all the commodities in the shopping place, and obtains the first-class classification result of the multi-frame hand image according to the plurality of primary classification results and the pre-trained first-order linear regression model; a confirmation unit for using the primary classification result as the first classification result; and a result identification unit for using the first classification result as the commodity to be identified.

It should be noted that the specific description of the registration module 401 can be referred to the related content of the step S1 in the foregoing embodiment. For a detailed description of the real-time tracking module 402, refer to the related content of step S2 in the foregoing embodiment. For details, refer to the related content of step S3 and steps S31, 32, 33, 34 and 35 in the above embodiment, and details are not described herein again.

Yet another embodiment of the present invention provides a settlement apparatus based on a convolutional neural network image recognition technology, including: a top camera, a forward facing camera 51, a lower camera 52, a processor, and a memory.

The top camera is used to shoot down from the top of the shopping venue to track customers who have obtained identity information in real time in the shopping venue; the front camera is used to shoot toward the front of the shelf to get a picture of the customer in front of the shelf carrying the merchandise The lower camera is used to photograph below to obtain a customer's hand image; the processor; and a memory for storing processor-executable instructions; wherein the processor is configured to:

Identifying pre-registered customers to obtain customer identity information, the identity information includes customer's facial data and payment account number; controlling the top camera to track the acquired identity information in real time to obtain this The location of the customer; determining whether the location of the customer is consistent with the location of the product in the shopping venue obtained by controlling the front camera, and if so, correlating the customer with the take-in or put-back action of the product, and obtaining according to the camera below After the hand image recognition takes or puts back the action and takes or replaces the product targeted by the action, the customer's shopping list is generated; and the settlement is performed according to the shopping list.

Referring to Fig. 5, in order to facilitate accurate identification and photographing of goods in the settlement method or settlement device and the settlement system described below, it is possible to more clearly and accurately see the arrangement of goods on each of the carrying platforms 53. The shelf carrying the merchandise (or the shelf for the settlement device) includes a base 56 for providing support and placing it on the ground. The post 55 is disposed on the base 56 and may be disposed in a vertical manner, such as may be disposed at one end of the base 56 such that the combination of the post 55 and the base 56 is L-shaped, or may be disposed at an intermediate portion of the upper surface of the base 56. The combination of the column 55 and the base 56 is inverted T-shaped; it can also be disposed in an inclined manner, which is not limited in this embodiment. The plurality of bearing platforms 53 are sequentially disposed on the same side of the column 55 in the vertical direction (when the column 55 is vertically disposed on the base 56, the vertical direction is the length direction of the column 55), and any two adjacent bearing platforms Spaces are left between the 53 to form a space for containing the goods to be placed, and the goods are mounted on each of the carrying platforms 53. One end of the carrying platform 53 away from the pillar 55 is a free end, and the free end of the upper loading platform 53 of any two adjacent carrying platforms 53 is closer to the pillar 55 than the free end of the lower loading platform, that is, the plurality of carrying platforms 53 The width (i.e., the length of the carrying platform 53 in the horizontal direction in Fig. 6) is wider and wider from the top to the bottom, the width of the lowermost carrying platform 53 is the widest, and the width of the uppermost carrying platform 53 is the narrowest, so When the goods are photographed from top to bottom, the arrangement of the goods on each of the carrying platforms 53 can be seen more clearly and accurately.

In practice, the racks may be referred to according to the number of the carrying platforms 53. When the carrying platform 53 is plural, the racks may be referred to as multi-layer racks. The number of carrying platforms 53 is preferably three or four.

The carrying platform 53 can be a flat plate, and the carrying platform 53 is a continuous body. When the goods are placed, the goods are placed on the carrying platform 53, so that the placing of the heavy and difficult hanging goods can be facilitated.

In other embodiments, the carrying platform 53 can include a crossbar and a plurality of hangers. The crossbar is horizontally disposed on the column 55, and the plurality of hangers are vertically disposed on the crossbar in parallel with each other and spaced apart from each other. At this time, the carrying platform 53 is a discontinuous body, and when the goods are placed, The cargo is suspended below the hanging rod, that is, the cargo is located below the carrying platform 53, so that the light is convenient and the package is easily deformed. The placement of the goods.

The shelf is particularly suitable for the ready-to-go shopping place. The front camera 52 and the lower camera 52 are disposed at the upper portion of the column 52, and the front camera 51 is photographed in the front direction of the shelf (the left side in FIG. 6). Shooting, that is, photographing the customer who selects the goods in front of the shelf; the lower camera 52 is located above the carrying platform 53, and the shooting direction is downward shooting from above the carrying platform 53, that is, photographing the goods on the carrying platform 53, The camera's shooting range covers the goods on the shelves.

The number of lower cameras 52 is preferably a plurality, so that the goods selected by the customer are captured. The plurality of lower cameras 52 may be sequentially distributed along the longitudinal direction L of the carrying platform 53 above the carrying platform 53, and the heights may be equal or unequal; the plurality of lower cameras 52 may be sequentially distributed along the width direction W of the carrying platform 53. Above the carrying platform 53, the respective heights may be equal or unequal; a part of the plurality of lower cameras 52 may be sequentially distributed above the carrying platform 53 along the length direction L of the carrying platform, in the plurality of lower cameras 52. The other part may be distributed on the top of the carrying platform 53 in the width direction W of the carrying platform. This embodiment does not limit this. Preferably, the number of the lower cameras is four, two in the longitudinal direction L of the carrying platform 53, and two in the width direction W of the carrying platform 53.

A further embodiment of the present invention provides a settlement system, which includes a client and a settlement device, and the client receives the identity information input by the customer and sends it to the settlement device and the shopping list delivered by the settlement device. The settlement device is the aforementioned settlement device, and the specific content will not be described again.

In summary, the beneficial effects brought by the embodiments of the present invention are as follows:

Good user experience: customers can get relevant information in the first time after taking the goods.

It is apparent from the technical knowledge that the invention can be implemented by other embodiments without departing from the spirit or essential characteristics thereof. Therefore, the above-disclosed embodiments are merely illustrative and not exclusive in all respects. All changes which are within the scope of the invention or equivalent to the scope of the invention are encompassed by the invention.

Claims

A settlement method, characterized in that the settlement method comprises:

Step S1, identifying a pre-registered customer to obtain identity information of the customer, the identity information including the customer's face data and a payment account;

Step S2, tracking the customer who has obtained the identity information in real time in the shopping place to obtain the location of the customer;

In step S3, it is determined whether the location of the customer is consistent with the location of the product in the shopping place. If the location is consistent, the customer is associated with the take-in or put-back action of the product, and the action is taken or retrieved. Or returning the item targeted by the action, generating a shopping list for the customer;

In step S4, settlement is performed according to the shopping list.
The settlement method according to claim 1, wherein in step S3, it is determined whether the location of the customer is consistent with the location of the product in the shopping place, and specifically includes:

The position of the front-facing camera photographed toward the front mounted on the shelf carrying the product indicates the position of the product, and when the forward-facing camera captures the customer identity information indicated by the picture containing the customer, the step S1 is acquired. When the identity information is the same, it is determined that the location of the customer is consistent with the location of the product in the shopping place.
The settlement method according to claim 1 or 2, wherein in step S3, identifying the take-in or put-back action comprises:

Obtaining a multi-frame coherent hand image of the customer before carrying the shelf of the product, and establishing a hand movement trajectory on the time axis for the multi-frame consecutive hand image;

When it is detected that the movement track of the hand is moved inward from the preset virtual action boundary line and the item is in the hand, the action is recognized as a return action;

When it is detected that the movement track of the hand is moved outward from the virtual action boundary line and the product is in the hand, the action is recognized as a take action;

Wherein, the virtual action boundary line is a direction away from the shelf, and the virtual action boundary line is a direction close to the shelf.
The settlement method according to claim 1, wherein in step S3, the identification is taken Take or replace the goods targeted by the action, including:

S31, performing target detection on the acquired multi-frame hand image containing the commodity, corresponding to acquiring a plurality of rectangular area images, the rectangular area image being an image corresponding to the rectangular area including the commodity, the multi-frame hand image and the plurality of cameras One-to-one correspondence;

S32. Acquire a plurality of primary classification results according to the plurality of rectangular area images and the pre-trained first-class classification model, and the pre-trained first-level classification model is an image recognition technology architecture based on a convolutional neural network and passes through all the commodities in the shopping place. The trained model obtains the first-class classification result of the multi-frame hand image according to the plurality of primary classification results and the pre-trained first-order linear regression model;

S33, the first classification result is used as the first classification result;

S34, the first classification result is taken as an item to be identified.
The settlement method according to claim 4, further comprising, after step S32, before step S34, further comprising:

S35. If the primary classification result is a similar commodity, according to the plurality of rectangular region images and the pre-trained secondary classification model, correspondingly obtaining a plurality of secondary classification results, and then according to the plurality of secondary classification results and the pre-trained secondary The linear regression model obtains the secondary classification result of the multi-frame hand image, and uses the secondary classification result as the first classification result. The secondary classification model is the image recognition technology architecture based on the convolutional neural network in advance and is similar in the shopping place. The model of commodity training in the commodity group, otherwise step S33 is performed.
A settlement device, characterized in that the settlement device comprises:

a registration module, configured to receive identity information input by the customer at the time of registration and obtain identity information of the customer who wants to enter the shopping place;

a real-time tracking module, connected to the registration module, for tracking a customer who obtains identity information by the registration module in a shopping place to obtain the location of the customer;

Generating a shopping list module, and connecting with the real-time tracking module, configured to determine whether the location of the customer acquired by the real-time tracking module is consistent with the location of the product in the shopping place, and if the customer is consistent with the product Taking or returning an action association, and generating a shopping list of the customer after identifying the item taken or putting back the action and taking or returning the action;

a settlement module, coupled to the generated shopping list module, for generating a shopping list according to the The shopping list generated by the module is settled.
The settlement device according to claim 6, wherein the generating a shopping list module comprises:

And an associating unit, configured to indicate a position of the product with a forward facing camera mounted on a shelf carrying the product, and when the front camera captures customer identity information represented by a picture containing a customer When the identity information acquired by the registration module is the same, it is determined that the location of the customer is consistent with the location of the product in the shopping place;

a motion recognition unit, configured to acquire a multi-frame coherent hand image of the customer before the shelf carrying the product, and establish a hand motion trajectory on the time axis for the multi-frame consecutive hand image; when the hand is detected When the motion trajectory moves outward from the preset virtual motion boundary line and the product is in the hand, the motion is recognized as a return motion; when the motion trajectory of the hand is detected to be moved outward by the virtual motion boundary line When the product is taken, the action is recognized as a take-up action; wherein, the virtual action boundary line is a direction away from the shelf, and the virtual action boundary line is a direction close to the shelf;

a commodity identification unit for identifying an item for which the taking or putting back action is directed; and

a shopping list generating unit, based on the identity information of the customer confirmed by the association unit, the take-in or put-back action recognized by the action recognition unit, and the product targeted by the take-in or put-back action recognized by the product identification unit, Generate a shopping list for this customer.
The settlement device according to claim 6, wherein the item identification unit comprises:

a target detection subunit, configured to perform target detection according to the multi-frame hand image containing the commodity acquired by the motion recognition unit to acquire a plurality of rectangular region images, where the rectangular region image is an image corresponding to the rectangular region including the commodity a multi-frame hand image corresponding to a plurality of cameras;

a first classification sub-unit, configured to acquire a plurality of primary classification results according to the plurality of rectangular area images and the pre-trained first-level classification model, wherein the pre-trained first-level classification model is an image recognition technology architecture based on a convolutional neural network and A model for training all commodities in a shopping mall, and obtaining a first-class classification result of a multi-frame hand image according to a plurality of primary classification results and a pre-trained first-order linear regression model;

a confirmation unit for using the first-class classification result as the first classification result; and

The result determining unit is configured to use the first classification result as the item to be identified.
A settlement device, characterized in that the settlement device comprises:

a top camera for capturing from the top of the shopping venue to track customers who have obtained identity information in real time in the shopping venue;

a forward facing camera for photographing the front of the shelf to obtain a picture of the customer located in front of the shelf carrying the merchandise;

The lower camera is used to shoot down to get the customer's hand image;

Processor; and

a memory for storing instructions executable by the processor;

Wherein the processor is configured to:

Identifying a pre-registered customer to obtain identity information of the customer, the identity information including the customer's facial data and a payment account; controlling the top camera to track the customer who has acquired the identity information in real time to obtain the location of the customer; determining the customer's location The position is consistent with the position of the product in the shopping place obtained by controlling the front camera, and if it is consistent, the customer is associated with the taking or putting back action of the product, and the hand is obtained according to the lower camera. After the image recognition takes or puts back the action and takes or replaces the product targeted by the action, the customer's shopping list is generated; and the settlement is performed according to the shopping list.
A settlement system, characterized in that the settlement system comprises: a client and a settlement device;

The client is configured to receive the identity information input by the customer and send it to the settlement device and the shopping list delivered by the settlement device;

The settlement device is the settlement device according to any one of claims 6 to 9.