CN112907168A - Dynamic commodity identification method, unmanned sales counter and sales method thereof - Google Patents
Dynamic commodity identification method, unmanned sales counter and sales method thereof Download PDFInfo
- Publication number
- CN112907168A CN112907168A CN202110145060.2A CN202110145060A CN112907168A CN 112907168 A CN112907168 A CN 112907168A CN 202110145060 A CN202110145060 A CN 202110145060A CN 112907168 A CN112907168 A CN 112907168A
- Authority
- CN
- China
- Prior art keywords
- commodity
- hand
- detection frame
- info
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000001514 detection method Methods 0.000 claims abstract description 189
- 238000004364 calculation method Methods 0.000 claims description 17
- 230000005484 gravity Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 235000021443 coca cola Nutrition 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Evolutionary Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Evolutionary Computation (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the technical field of image recognition and detection, and particularly relates to a dynamic commodity recognition method, an unmanned sales counter and a sales method thereof; the dynamic commodity identification method includes S1: collecting a shopping video of a user; s2: detecting shopping videos of users by the dynamic commodity identification method, and marking a hand-held detection frame in each video frame; determining the information h _ info of the commodity being taken in a single video frame according to the commodity detection frame bound by the hand detection frame; the hand-held commodity information h _ info comprises a label of a commodity and a coordinate of a hand-held detection frame; s3: traversing each video frame in the shopping video of the user, and adding or deleting the hand-held commodity information h _ info into or from the shopping cart list according to the motion trend of the hand-held detection frame; s4: and settling the shopping cart list after traversing. The invention binds the commodity detection frame and the hand detection frame into the hand detection frame by calculating the change angle and the distance between the commodity detection frame and the hand detection frame in two continuous video frames, and judges whether the commodity is taken or placed through the hand detection frame.
Description
Technical Field
The invention belongs to the technical field of image recognition and detection, and particularly relates to a dynamic commodity recognition method, an unmanned sales counter and a sales method thereof.
Background
At present, common unmanned sales cabinets include an RFID intelligent cabinet, a static image cabinet, a gravity cabinet and a dynamic image gravity combination cabinet. The unmanned sales counter has advantages and disadvantages, the RFID intelligent counter is adhered with an RFIP wireless label, and the labor cost and the label cost are high; the static image cabinet needs to install a camera on each shelf, so that the hardware cost is high, certain requirements are required for placing goods, and the space utilization rate is low; the gravity cabinet needs to install the gravity sensing device on each goods shelf, the equipment cost is high, the commodity judgment under the condition of close weight cannot be solved, and the error of the equipment is easily caused in the transportation process.
The current dynamic commodity identification method is based on an image target detection technology, commodity ID tracking schemes such as a Deepsort algorithm (including a Kalman filter and a Hungary algorithm) are mostly used, and due to the fact that a commodity detection frame is subjected to false identification and missed identification in target detection, the change angle of a commodity taking process is large, the ID tracking scheme has an unsatisfactory actual effect on commodity tracking, a plurality of IDs often appear on the same commodity, the same ID appears on the same commodity and the like, and the dynamic commodity identification effect is poor. Therefore, there is a need for a dynamic commodity identification method, an unmanned sales counter and a vending method thereof, which can be based on vision without gravity and can effectively avoid misidentification.
Disclosure of Invention
The invention aims to provide a dynamic commodity identification method, an unmanned sales counter and a vending method thereof, which can be separated from gravity, based on vision and can effectively avoid error identification.
The purpose of the invention patent is realized as follows:
a dynamic commodity identification method comprises the following steps
Detecting each video frame in a shopping video of a user through an image classification and identification model, marking a hand image and each commodity image in the video frame by using a commodity detection frame and a hand detection frame respectively, and giving a commodity label corresponding to each commodity detection frame;
respectively calculating the change angle of each commodity detection frame and the change angle of each hand detection frame in two continuous video frames;
if the difference between the change angles of a certain commodity detection frame and the hand detection frame is smaller than a certain threshold value, and the distance between the commodity detection frame and the hand detection frame in a second video frame is smaller than the certain threshold value, binding the commodity detection frame and the hand detection frame as a hand detection frame;
and in the subsequent detection process, judging whether the commodity bound with the hand-held detection frame in the video is taken out or put back according to the movement of the hand-held detection frame.
Further, in two continuous video frames, the coordinates of the hand detection frame and each commodity detection frame are respectively detected, and the commodity detection frames and the hand detection frames are bound to be identified; the coordinates of the bound commodity detection frame/hand detection frame in the previous video frame are (x1, y1), the coordinates in the next video frame are (x2, y2), and the change angle is arctan (x2-x1/y2-y 1).
Further, in two continuous video frames, detecting the coordinate change of the commodity detection frame/hand detection frame and obtaining the moving distance; if the moving distance is smaller than the set threshold, the calculation of the change angle is performed, and if the moving distance is larger than the set threshold, the calculation of the change angle is skipped.
Furthermore, the hand detection frame takes the position of the hand detection frame in the hand detection frame as a judgment point to judge whether the commodity is taken or placed.
A vending method of an unmanned vending cabinet comprises
S1: collecting a shopping video of a user;
s2: detecting shopping videos of users by the dynamic commodity identification method, and marking a hand-held detection frame in each video frame; determining the information h _ info of the commodity being taken in a single video frame according to the commodity detection frame bound by the hand detection frame; the hand-held commodity information h _ info comprises a label of a commodity and a coordinate of a hand-held detection frame;
s3: traversing each video frame in the shopping video of the user, and adding or deleting the hand-held commodity information h _ info into or from the shopping cart list according to the motion trend of the hand-held detection frame;
s4: and settling the shopping cart list after traversing.
Further, the step S3 includes recording hand-held commodity information h _ info frame by frame, forming shopping cart information S _ info for performing motion trend determination;
s3.1, when the shopping cart list is empty, adding the handheld commodity information h _ info of the current video frame to the shopping cart information S _ info;
s3.2 when the shopping cart information is stored, for the handheld commodity information h _ info of the current video frame, searching the shopping cart information S _ info which has the same label as the handheld commodity information h _ info and is closest to the handheld commodity information h _ info in the shopping cart information S _ info; by comparing the position relationship between the hand-held commodity information h _ info and the shopping cart information s _ info, it is determined whether to add the commodity information determined by the current video frame to the shopping cart list.
Further, said S3.2 comprises,
establishing a picking and placing judgment axis according to the picking and placing direction of the commodities in the video;
if the positions of the hand-held commodity information h _ info and the shopping cart information s _ info are respectively located on two sides of the pick-and-place determination axis, determining whether the commodity bound with the hand image is taken out or put back according to the commodity pick-and-place direction, and adding or deleting the shopping cart list according to the taking out or putting back of the commodity.
Further, said S3.2 further comprises,
if the distance between the positions of the handheld commodity information h _ info and the position of the shopping cart information s _ info is larger than a certain threshold value, adding the determined handheld commodity information h _ info to a shopping cart list;
and if the distance between the position of the hand held commodity information h _ info and the position of the shopping cart information s _ info is less than a certain threshold value, replacing the corresponding shopping cart information s _ info with the determined hand held commodity information h _ info.
An unmanned sales counter adopts the selling method of the unmanned sales counter; comprises that
The cabinet body is used for placing commodities;
the shooting unit is fixed in the cabinet body and used for collecting shopping videos of users;
and the processing unit is used for processing and settling the collected shopping videos of the users.
Compared with the prior art, the invention has the outstanding and beneficial technical effects that:
the invention binds the commodity detection frame and the hand detection frame into a hand detection frame by calculating the change angle and the distance between the commodity detection frame and the hand detection frame in two continuous video frames, and judges whether the commodity is taken or placed through the hand detection frame; for the image detection sales counter, the combined hand-held detection frame can effectively reduce the occurrence of false recognition by applying calculation, increase the accuracy of operation and reduce the occurrence of false judgment; the method for binding by calculating the change angle of the detection frame is quick in calculation, small in calculation amount, accurate in calculation binding result and capable of reducing the occurrence of misjudgment and missed judgment.
The invention determines whether to perform binding calculation by calculating the distance between the two commodity detection frames/the hand detection frame before the detection frames are bound, thereby effectively reducing the calculation amount of the processing unit and reducing the occurrence of false identification.
According to the invention, after the commodity detection frame and the hand detection frame are bound into the hand detection frame, the coordinates of the hand detection frame are re-determined, generally, the central coordinates of the hand detection frame before combination is used as the judgment coordinates of the hand detection frame, and the commodity is driven by the hand to move in most states in actual work, so that the hand is used as the center, the calculation logic is better met, and the judgment is more accurate.
The shopping flow is duplicated by adding the shopping cart information s _ info to compare the shopping cart information s _ info with the hand-held commodity information h _ info frame by frame, so that the shopping list can be added or deleted through position change to realize shopping.
Drawings
Fig. 1 is a schematic diagram of binding a hand image and a commodity image.
Fig. 2 is a schematic diagram of the positions of the hand held article information h _ info and the shopping cart information s _ info in the first relationship.
FIG. 3 is a diagram illustrating the positions of the hand held article information h _ info and the shopping cart information s _ info according to the second relationship.
Fig. 4 is a schematic diagram of the positions of the hand held article information h _ info and the shopping cart information s _ info in the third relationship.
Detailed Description
The invention is further described in the following examples:
an unmanned sales counter comprises a cabinet body, a shooting unit and a processing unit; the cabinet body is internally provided with a goods shelf for placing goods, and the shooting unit is arranged in the cabinet body and used for collecting shopping videos of users; the shooting unit is generally a camera, and when the cabinet body is vertical, the shooting unit shoots at the inner top of the cabinet body and shoots downwards. The image of the user shopping video collected by the shooting unit comprises a cabinet body part and an external part, namely, the process that the commodities are required to be shown in the user shopping video and are taken out from the inside of the cabinet body to the outside is needed, and therefore the commodity purchasing judgment is facilitated. The processing unit generally comprises a processor and a storage, wherein a computer program is arranged in the storage, and meanwhile, the video shot by the shooting unit and used for shopping is received, the processor is used for carrying out operation calculation, and a shopping structure is output, so that the charging operation is carried out.
The vending method adopted by the unmanned vending cabinet comprises the following steps,
s1: collecting a shopping video of a user;
s2: detecting shopping videos of users by a dynamic commodity identification method, and marking a hand-held detection frame in each video frame; determining the information h _ info of the commodity being taken in a single video frame according to the commodity detection frame bound by the hand detection frame; the hand-held commodity information h _ info comprises a label of a commodity and a coordinate of a hand-held detection frame;
s3: traversing each video frame in the shopping video of the user, adding or deleting the hand-held commodity information h _ info into or from the shopping cart list according to the motion trend of the hand-held detection frame, and recording the hand-held commodity information h _ info frame by frame to form shopping cart information s _ info;
s4: and settling the shopping cart list after traversing.
Wherein the dynamic commodity identification method comprises the following steps,
detecting each video frame in a shopping video of a user through an image classification and identification model, marking a hand image and each commodity image in the video frame by using a commodity detection frame and a hand detection frame respectively, and giving a commodity label corresponding to each commodity detection frame;
respectively calculating the change angle of each commodity detection frame and the change angle of each hand detection frame in two continuous video frames;
if the difference between the change angles of a certain commodity detection frame and the hand detection frame is smaller than a certain threshold value, and the distance between the commodity detection frame and the hand detection frame in a second video frame is smaller than the certain threshold value, binding the commodity detection frame and the hand detection frame as a hand detection frame;
and in the subsequent detection process, judging whether the commodity bound with the hand-held detection frame in the video is taken out or put back according to the movement of the hand-held detection frame
The image classification and identification model belongs to the field of target detection in computer vision, and the task of target detection is to find out all interested targets in an image and determine the categories and positions of the interested targets; in the invention, the object detection is completed by determining the commodity identification and hand identification on each video frame in the shopping video of the user, namely, an image classification identification model is required to identify various commodities in the video, and the commodity types of different commodities need to be distinguished, for example, different brands and different specifications need to be distinguished. Specifically, the image classification and identification model adopted in the embodiment adopts a PP-YOLO algorithm, and is obtained by inputting audience products of the unmanned sales counter and training the audience products. The commodity image and the hand image after being identified are marked by a commodity detection frame and a hand detection frame which are generally square; for the commodity detection frame, the name of the commodity is marked on one side or the center of the commodity detection frame, namely the commodity label, the commodity label is used for distinguishing other products, different commodities have different labels, and even if the commodities belong to the same category, the commodities are distinguished by the different labels, for example, for coca-cola and pepa-cola, the image classification and identification model is used for distinguishing the commodities.
Specifically referring to fig. 1, the outer square frame represents an image displayed by a video frame, the actual hand image and the commodity image are respectively represented by the characters 'hand' and 'commodity' on the upper and lower sides of the image, and the 'hand' and the 'commodity' in the image are respectively marked by the detection frame; for convenience of illustration, two commodity detection boxes and two hand detection boxes are shown in a single video frame image, which represent the commodity detection box and the hand detection box in the front and rear frames respectively, and the front and rear frames are distinguished from the implementation outer frame by a dotted line outer frame respectively, which is the same as that in other figures. As shown in fig. 1, when the hand image and the commodity image move from the upper portion to the lower portion, if the user holds the commodity, the hand image and the commodity image are both shifted by a certain angle, the distance between the hand image and the commodity image is basically unchanged, and the detection frames of the hand image and the commodity image are arranged in an intersecting manner, so that the binding determination can be performed by calculating the angle. The diagonal lines in the figure indicate the angular direction in which the hand image and the product image are offset.
Further, the respectively calculating the change angles of the commodity detection frame and the hand detection frame in two continuous video frames comprises respectively detecting the coordinates of the hand detection frame and each commodity detection frame in the two continuous video frames, and identifying the binding of the commodity detection frame and the hand detection frame; coordinates of the bound commodity detection frame/hand detection frame in a previous video frame are (x1, y1), coordinates of the bound commodity detection frame/hand detection frame in a next video frame are (x2, y2), and a change angle is arctan (x2-x1/y2-y 1); establishing a coordinate system in a video and calculating the coordinates of the commodity detection frame/hand detection frame can be realized through the image classification and identification model; in this embodiment, the coordinates of the product detection frame/hand detection frame may be the coordinates of any point in the product detection frame/hand detection frame, but it is required to ensure that the coordinates of the same point in the product detection frame/hand detection frame are selected from two consecutive video frames; preferably, the coordinates of the center position of the commodity detection frame/hand detection frame are selected in the embodiment, because the coordinates located in the center can better reflect the commodity and the change condition of the hand, so that the angle calculation is more accurate. In order to prevent calculation errors, a constant of 0.001 is added when y2-y1 is calculated, namely the angle is arctan (x2-x1/y2-y1+ 0.001).
After the change angles of the commodity detection frame and the hand detection frame in two continuous video frames are obtained, the distance between the commodity detection frame and the hand detection frame needs to be calculated, and similarly, the distance between the commodity detection frame and the hand detection frame is calculated through the center coordinates of the commodity detection frame and the hand detection frame; in the camera with the specification of 1920 × 1080, if the distance between the product detection frame and the hand detection frame is less than 120 pixels and the angle difference between the two frames is less than 40 degrees, the two frames are combined, of course, the above threshold has many influence factors, such as camera specification factor, camera installation position factor, product size factor, etc., and the user needs to consider the angle threshold and the distance threshold of the combination of the product detection frame and the hand detection frame according to the above practical factors.
After the commodity detection frame and the hand detection frame are bound into the hand detection frame, the coordinates of the hand detection frame are determined again, generally, the central coordinates of the hand detection frame before combination are used as judgment coordinates of the hand detection frame, that is, in most states of actual work, the commodity is driven by the hand to move, so that the hand is used as the center to better accord with the calculation logic, and the judgment is more accurate. After the commodity detection frame is bound with the hand detection frame, the coordinates of the commodity detection frame are not considered any more, and for the commodity image marked by the commodity detection frame, the processing unit records the label information of the commodity image into a hand commodity list, which indicates that the user holds the commodity at the moment and records the commodity as hand commodity information h _ info, and simultaneously records the central coordinates of the hand detection frame in the hand commodity information h _ info for the next position judgment; there may be a plurality of hand item information h _ info in the hand item list depending on how many items the user takes.
Before calculating the change angles of the commodity detection frame/hand detection frame in two continuous video frames, detecting the coordinate change of the commodity detection frame/hand detection frame and obtaining a moving distance; if the moving distance is smaller than the set threshold, calculating the change angle, and if the moving distance is larger than the set threshold, skipping the calculation of the change angle, namely not calculating the change angle; if all the commodity detection frames and the hand detection frames do not have the combined condition in the current image frame, no hand detection frame exists in the current image frame; the threshold value is generally the maximum distance of hand movement in normal shopping, and if the distance is exceeded, the two commodity detection frames/hand detection frames are not the same detection frame, so that comparison calculation is not needed. Which reduces the amount of computation by the processing unit and reduces the occurrence of misidentification situations.
After the hand detection frame of the current frame and the corresponding hand commodity list are obtained, the shopping cart list is added or deleted by detecting the motion trend of the hand detection frame in the previous frame and the next frame; the shopping cart list is a final output commodity list, namely the condition of commodities actually purchased by a user; after detecting the movement trend of the front and back frames holding the detection frame, the shopping cart information s _ info is generated at the same time. The shopping cart information s _ info is used for recording the motion situation of the hand-held detection frame, so as to perform position judgment on the previous frame image and the next frame image.
Specifically, the motion trend of the hand-held detection frame generally includes the following two cases:
s3.1, when the shopping cart list is empty, adding the handheld commodity information h _ info of the current video frame to the shopping cart information S _ info; this situation occurs when the user has just begun to purchase the product, and a hand detection box may just appear in the video.
S3.2 when the shopping cart information is stored, for the handheld commodity information h _ info of the current video frame, searching the shopping cart information S _ info which has the same label as the handheld commodity information h _ info and is closest to the handheld commodity information h _ info; by comparing the position relationship between the hand-held commodity information h _ info and the shopping cart information s _ info, it is determined whether to add the commodity information determined by the current video frame to the shopping cart list. The shopping cart information s _ info may be multiple, and the list is searched in a traversal mode, so that the position information and the commodity information of the corresponding hand-held detection frame closest to the last frame of image are found as far as possible, and therefore position comparison is facilitated, and commodity addition is performed.
In S3.2, there are several such positional relationships.
In the first relation, as shown in fig. 2, a pick-and-place determining axis is first established according to the pick-and-place direction of the commodity in the video, generally, since the image of the collected user shopping video includes a cabinet portion and an external portion, the pick-and-place determining axis is generally set on a boundary line between the cabinet portion and the external portion, and a debugger generally sets the cabinet portion and the external portion in the image composition on the upper and lower sides of the image, respectively, thereby facilitating the setting of the pick-and-place determining axis. After the pick-and-place determination axis is set, comparing the position information of the hand commodity information h _ info and the position information of the shopping cart information s _ info, if the positions of the hand commodity information h _ info and the position information of the shopping cart information s _ info are respectively positioned at two sides of the pick-and-place determination axis, and the distance between the positions of the hand commodity information h _ info and the position information of the shopping cart information s _ info is less than a certain threshold value, determining whether the commodity bound with the hand image is taken out or put back according to the commodity pick-and-place direction, and adding or deleting a shopping cart list according to the taking out or putting back of the commodity; in this case, if the hand-held article information h _ info is located in the outside area and the shopping cart information s _ info is located in the cabinet area, it indicates that the article is taken out, and at this time, the article information is added to the shopping cart list and the shopping cart information s _ info is described. In this embodiment, the pick-and-place determining axis is a transverse axis, and a transverse coordinate in the transverse axis is fixed, so that the pick-and-place determining axis on which the hand commodity information h _ info is located can be obtained only by determining whether the transverse coordinate in the hand commodity information h _ info is larger or smaller than the transverse coordinate of the pick-and-place determining axis, and similarly, the shopping cart information s _ info is also obtained.
Note that, the distance between the hand held article information h _ info and the shopping cart information s _ info is smaller than a threshold value, which is generally the maximum distance of normal single-frame article movement, and the value is determined according to the pixel size of the camera and the usage environment.
In the second relationship, as shown in fig. 3, if the position distance between the handheld article information h _ info and the shopping cart information s _ info is greater than a certain threshold, the determined handheld article information h _ info is added to the shopping cart list; this situation may occur in a scenario where the product in the previous frame is about to be taken out of the sales counter, and another person in the next frame enters the sales counter to take the same product and is detected, and the handheld products are located at the front and rear ends of the two video frames respectively, and the products are the same, and it should be determined that there are two products, so the newly added handheld product information h _ info is the new shopping cart information s _ info.
In the third relation, if the distance between the positions of the handheld commodity information h _ info and the position of the shopping cart information s _ info is larger than a certain threshold value, the determined handheld commodity information h _ info is added to the shopping cart list; in this case, since the article is moved in the outside portion or in the cabinet portion and is not taken out or put back, only the information is replaced, and the shopping cart list is not updated. The position relationship is also applicable to the hand-held commodity information h _ info and the shopping cart information s _ info located on both sides of the pick-and-place determination axis.
For the above threshold values, which are the maximum distances for normal single-frame product movement, for example, in a camera with a specification of 1920 × 1080, the distance between the positions of the hand-held product information h _ info and the position of the shopping cart information s _ info is 370 pixels at the maximum, 0 pixels at the minimum, and generally about 220 pixels.
It should be noted that, in the shopping video of the user, the movement of the hand and the product is coherent, that is, the shopping cart information s _ info is always in the state of replacing frame by frame, so that the case that the distance between the product information h _ info and the shopping cart information s _ info is too large will not occur in the case of normal movement.
After all the video frames are traversed, shopping is finished; the processing unit calculates and outputs the corresponding commodity price according to the commodity information recorded in the shopping cart list, and then the payment settlement is carried out by the user.
The above embodiments are only preferred embodiments of the present invention, and not intended to limit the scope of the present invention, therefore: all equivalent changes made according to the structure, shape and principle of the invention shall be covered by the protection scope of the invention.
Claims (9)
1. A dynamic commodity identification method comprises the following steps
Detecting each video frame in a shopping video of a user through an image classification and identification model, marking each commodity image and each hand image in the video frames by a commodity detection frame and a hand detection frame respectively, and endowing each commodity detection frame with a corresponding commodity label;
the method is characterized in that:
respectively calculating the change angle of each commodity detection frame and the change angle of each hand detection frame in two continuous video frames;
if the difference between the change angles of a certain commodity detection frame and the hand detection frame is smaller than a certain threshold value, and the distance between the commodity detection frame and the hand detection frame in a second video frame is smaller than the certain threshold value, binding the commodity detection frame and the hand detection frame as a hand detection frame;
and in the subsequent detection process, judging whether the commodity bound with the hand-held detection frame in the video is taken out or put back according to the movement of the hand-held detection frame.
2. The dynamic merchandise identification method of claim 1, wherein: respectively detecting the coordinates of the hand detection frame and each commodity detection frame in two continuous video frames, and identifying the binding of the commodity detection frames and the hand detection frames; the coordinates of the bound commodity detection frame/hand detection frame in the previous video frame are (x1, y1), the coordinates in the next video frame are (x2, y2), and the change angle is arctan (x2-x1/y2-y 1).
3. The dynamic merchandise identification method of claim 2, wherein: detecting the coordinate change of the commodity detection frame/hand detection frame in two continuous video frames and obtaining the moving distance; if the moving distance is smaller than the set threshold, the calculation of the change angle is performed, and if the moving distance is larger than the set threshold, the calculation of the change angle is skipped.
4. The dynamic merchandise identification method of claim 1, wherein: the hand detection frame takes the position of the hand detection frame in the hand detection frame as a judgment point to judge whether the commodity is taken or put.
5. A vending method of an unmanned vending cabinet is characterized in that: comprises that
S1: collecting a shopping video of a user;
s2: detecting a shopping video of a user by the dynamic commodity identification method according to any one of claims 1 to 4, and marking a hand detection box in each video frame; determining the information h _ info of the commodity being taken in a single video frame according to the commodity detection frame bound by the hand detection frame; the hand-held commodity information h _ info comprises a label of a commodity and a coordinate of a hand-held detection frame;
s3: traversing each video frame in the shopping video of the user, and adding or deleting the hand-held commodity information h _ info into or from the shopping cart list according to the motion trend of the hand-held detection frame;
s4: and settling the shopping cart list after traversing.
6. A vending method of an unmanned sales counter according to claim 5, wherein: the S3 includes recording hand-held commodity information h _ info frame by frame to form shopping cart information S _ info for judging movement trend;
s3.1, when the shopping cart list is empty, adding the handheld commodity information h _ info of the current video frame to the shopping cart information S _ info;
s3.2 when the shopping cart information is stored, for the handheld commodity information h _ info of the current video frame, searching the shopping cart information S _ info which has the same label as the handheld commodity information h _ info and is closest to the handheld commodity information h _ info in the shopping cart information S _ info; by comparing the position relationship between the hand-held commodity information h _ info and the shopping cart information s _ info, it is determined whether to add the commodity information determined by the current video frame to the shopping cart list.
7. A vending method of an unmanned sales counter according to claim 6, wherein: said S3.2 comprises the following steps,
establishing a picking and placing judgment axis according to the picking and placing direction of the commodities in the video;
if the positions of the hand commodity information h _ info and the shopping cart information s _ info are respectively located on two sides of the pick-and-place determination axis, and the distance between the positions of the hand commodity information h _ info and the positions of the shopping cart information s _ info is smaller than a certain threshold value, whether the commodity bound with the hand image is taken out or put back is determined according to the commodity pick-and-place direction, and a shopping cart list is added or deleted according to the taking out or putting back of the commodity.
8. A vending method of an unmanned sales counter according to claim 6, wherein: said S3.2 further comprises that,
if the distance between the hand-held commodity information h _ info and the shopping cart information s _ info is larger than a certain threshold value, the newly added hand-held commodity information h _ info is new shopping cart information s _ info;
and if the distance between the position of the hand held commodity information h _ info and the position of the shopping cart information s _ info is less than a certain threshold value, replacing the corresponding shopping cart information s _ info with the determined hand held commodity information h _ info.
9. An unmanned sales counter, its characterized in that: a vending method using an unmanned sales counter according to any one of claims 5 to 8; comprises that
The cabinet body is used for placing commodities;
the shooting unit is fixed in the cabinet body and used for collecting shopping videos of users;
and the processing unit is used for processing and settling the collected shopping videos of the users.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110145060.2A CN112907168A (en) | 2021-02-02 | 2021-02-02 | Dynamic commodity identification method, unmanned sales counter and sales method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110145060.2A CN112907168A (en) | 2021-02-02 | 2021-02-02 | Dynamic commodity identification method, unmanned sales counter and sales method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112907168A true CN112907168A (en) | 2021-06-04 |
Family
ID=76121555
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110145060.2A Pending CN112907168A (en) | 2021-02-02 | 2021-02-02 | Dynamic commodity identification method, unmanned sales counter and sales method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112907168A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113743530A (en) * | 2021-09-16 | 2021-12-03 | 广东佩服科技有限公司 | Automatic vending identification method based on dynamic vision |
| CN114627430A (en) * | 2021-12-30 | 2022-06-14 | 东莞先知大数据有限公司 | Taking takeout detection method and device and storage medium |
| CN115767250A (en) * | 2022-11-15 | 2023-03-07 | 浙江星星冷链集成股份有限公司 | Provincial flow settlement method based on cloud identification image cabinet |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709422A (en) * | 2016-11-16 | 2017-05-24 | 南京亿猫信息技术有限公司 | Supermarket shopping cart hand identification method and identification system thereof |
| CN109353397A (en) * | 2018-09-20 | 2019-02-19 | 北京旷视科技有限公司 | Merchandise control methods, devices and systems and storage medium and shopping cart |
| CN110097715A (en) * | 2018-01-29 | 2019-08-06 | 大黑天物产株式会社 | Merchandise management server, automatic cash register system and merchandise control method |
| CN111127174A (en) * | 2020-01-06 | 2020-05-08 | 鄂尔多斯市东驿科技有限公司 | Intelligent unmanned supermarket control system |
| CN111260685A (en) * | 2018-11-30 | 2020-06-09 | 阿里巴巴集团控股有限公司 | Video processing method and device and electronic equipment |
-
2021
- 2021-02-02 CN CN202110145060.2A patent/CN112907168A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709422A (en) * | 2016-11-16 | 2017-05-24 | 南京亿猫信息技术有限公司 | Supermarket shopping cart hand identification method and identification system thereof |
| CN110097715A (en) * | 2018-01-29 | 2019-08-06 | 大黑天物产株式会社 | Merchandise management server, automatic cash register system and merchandise control method |
| CN109353397A (en) * | 2018-09-20 | 2019-02-19 | 北京旷视科技有限公司 | Merchandise control methods, devices and systems and storage medium and shopping cart |
| CN111260685A (en) * | 2018-11-30 | 2020-06-09 | 阿里巴巴集团控股有限公司 | Video processing method and device and electronic equipment |
| CN111127174A (en) * | 2020-01-06 | 2020-05-08 | 鄂尔多斯市东驿科技有限公司 | Intelligent unmanned supermarket control system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113743530A (en) * | 2021-09-16 | 2021-12-03 | 广东佩服科技有限公司 | Automatic vending identification method based on dynamic vision |
| CN114627430A (en) * | 2021-12-30 | 2022-06-14 | 东莞先知大数据有限公司 | Taking takeout detection method and device and storage medium |
| CN114627430B (en) * | 2021-12-30 | 2023-04-07 | 东莞先知大数据有限公司 | Taking takeout detection method and device and storage medium |
| CN115767250A (en) * | 2022-11-15 | 2023-03-07 | 浙江星星冷链集成股份有限公司 | Provincial flow settlement method based on cloud identification image cabinet |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108846621B (en) | Inventory management system based on strategy module | |
| JP7248689B2 (en) | Vending method and apparatus, and computer readable storage medium | |
| US20200258069A1 (en) | Weight monitoring-based sensing system and sensing method | |
| CN108335408B (en) | Article identification method, device and system for vending machine and storage medium | |
| CN112907168A (en) | Dynamic commodity identification method, unmanned sales counter and sales method thereof | |
| US9916561B2 (en) | Methods, devices and computer readable storage devices for tracking inventory | |
| US9473747B2 (en) | Whole store scanner | |
| US11978014B2 (en) | Dynamic termination zone detection system | |
| CN107403332B (en) | Goods shelf fetching detection system and method | |
| CN109726759B (en) | Unmanned vending method, device, system, electronic equipment and computer readable medium | |
| CN111263224B (en) | Video processing method and device and electronic equipment | |
| CN101149792A (en) | System and method for performing inventory using a mobile inventory robot | |
| CN108557365A (en) | Unmanned shop automatically vending system | |
| US11301684B1 (en) | Vision-based event detection | |
| US20230037427A1 (en) | Identifying barcode-to-product mismatches using point of sale devices and overhead cameras | |
| RU2730112C1 (en) | System and method of identifying objects in composite object | |
| US11790630B1 (en) | Inferring facility planograms | |
| CN111428743B (en) | Commodity identification method, commodity processing device and electronic equipment | |
| CN111260685B (en) | Video processing method and device and electronic equipment | |
| CN111339929A (en) | Retail system of unmanned supermarket | |
| CN115601686A (en) | Method, device and system for confirming delivery of articles | |
| US11488103B1 (en) | Customized retail environments | |
| CN114358881A (en) | Self-service settlement method, device and system | |
| US20240095718A1 (en) | Customized retail environments | |
| CN118097519B (en) | Intelligent shopping cart shopping behavior analysis method and system based on commodity track analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210604 |