WO2021147950A1 - 货品识别方法、货品识别系统及电子设备 - Google Patents

货品识别方法、货品识别系统及电子设备 Download PDF

Info

Publication number
WO2021147950A1
WO2021147950A1 PCT/CN2021/073058 CN2021073058W WO2021147950A1 WO 2021147950 A1 WO2021147950 A1 WO 2021147950A1 CN 2021073058 W CN2021073058 W CN 2021073058W WO 2021147950 A1 WO2021147950 A1 WO 2021147950A1
Authority
WO
WIPO (PCT)
Prior art keywords
goods
user
signal
space
change
Prior art date
Application number
PCT/CN2021/073058
Other languages
English (en)
French (fr)
Inventor
冯立男
张一玫
Original Assignee
上海追月科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海追月科技有限公司 filed Critical 上海追月科技有限公司
Publication of WO2021147950A1 publication Critical patent/WO2021147950A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
    • G06V20/36—Indoor scenes
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47B—TABLES; DESKS; OFFICE FURNITURE; CABINETS; DRAWERS; GENERAL DETAILS OF FURNITURE
    • A47B47/00—Cabinets, racks or shelf units, characterised by features related to dismountability or building-up from elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47B—TABLES; DESKS; OFFICE FURNITURE; CABINETS; DRAWERS; GENERAL DETAILS OF FURNITURE
    • A47B2220/00—General furniture construction, e.g. fittings
    • A47B2220/0091—Electronic or electric devices

Definitions

  • the invention relates to a product identification technology for retail products, and specifically to a product identification method, a product identification system and electronic equipment.
  • the same kind of goods are placed on the same shelf or pallet as much as possible. If a user removes a certain item from one shelf or a pallet and places it on another shelf or pallet, it will cause wrong picking.
  • the user For the problem of random placement, for some solutions in the prior art, during the shopping process, the user must put the picked goods back into the original position when putting them back on the shelf.
  • the computer cannot accurately identify the type and quantity of the returned goods, and cannot update the shopping database in a timely and accurate manner. The user will have to pay after returning the goods to the wrong position on the shelf.
  • the cost of the product causes identification errors and affects user experience. The user misplaced the wrongly picked goods on the wrong shelf or pallet, which caused the user's shopping record to be wrong, which is called the wrong picking and misplacement problem in the industry.
  • multiple cameras are installed on each shelf to monitor the status of the goods; at the same time, multiple cameras are installed on the top of the supermarket to determine the location of users.
  • Using a large number of cameras in the same space will also cause costs.
  • the problem is too high.
  • the video or picture collected by the camera is a relatively large area, and most of its content is background images that have nothing to do with the goods that need to be identified. If you process video or picture and other image data every time, you must process all the information of the entire video or picture.
  • the computing capacity of the computer or server will be very large, and the configuration requirements of the computer will be very high, causing the problem of high hardware cost. If only a set of cameras on the top can be used to realize the user location tracking function and realize the identification of the types of goods, it can further reduce the hardware cost, maintenance cost and operating cost.
  • the purpose of the present invention is to provide a product identification method, a product identification system and electronic equipment to solve the problem of product identification in unmanned convenience stores, and to solve the problem that users make mistakes in their shopping records due to mishandling of goods during shopping , In order to solve the problem of excessive calculation and high hardware cost.
  • the present invention provides a method for identifying goods, including the following steps: a user image collection step, real-time collection of real-time images of each user in a closed space; at least one shelf is set in the closed space, at least A product is placed on at least one pallet of the shelf; the signal acquisition step is to acquire a product change signal; the product change signal is a product decrease signal or a product increase signal; the product change signal includes a product change signal The position of the pallet and the time period of the weight change of the pallet where the goods change; the image interception step is to intercept, from the real-time image of at least one user, the position of the hand of at least one detected user in a detection space during a detection period
  • the spatial image includes consecutive multiple frames of hand space pictures; and a step of determining the type of goods, which determines the type of goods in the hand of the detected user during the detection time period based on the multiple frames of hand space pictures and a product recognition model .
  • the present invention also provides an electronic device, including a memory and a processor; the memory is used to store executable program code; the processor is connected to the memory, and runs with the executable program code by reading the executable program code.
  • the computer program corresponding to the program code can be executed to execute the steps in the above-mentioned goods identification method.
  • the present invention also provides a goods identification system, including the electronic equipment.
  • the goods identification system further includes at least one shelf and two or more three-dimensional cameras; the shelf is arranged in a closed space; the two or more three-dimensional cameras are evenly distributed on the top of the closed space, and the three-dimensional cameras The field of view covers the entire bottom surface of the enclosed space.
  • the beneficial effect of the present invention is to provide a product identification method, a product identification system and electronic equipment, monitor the shelf of the product identification system in real time, obtain a product change signal in real time, record the weight change period of the pallet where the product change occurs, and intercept The space image of the hand is intercepted within a preset time period before or after the weight change period of the pallet of the goods change, and the multi-frame hand space pictures that constitute the image are identified to quickly and accurately determine the category of the goods.
  • the present invention can also accurately identify the quantity of goods that have been taken or put back, so as to adjust the user's shopping record in real time, effectively reduce the amount of computer calculations and reduce the cost of hardware, with fast response speed and low energy consumption. Effectively solve the problem of mishandling and misplacement. Even if a variety of different types of goods are placed on the same tray on a shelf, the type and quantity of the goods that are taken or put back can be accurately identified, which is useful for the user's shopping Information judgment is more accurate, which improves user experience and facilitates promotion and application.
  • Figure 1 is a schematic diagram of the structure of the goods identification system described in Embodiment 1 of the present invention.
  • FIG. 2 is a schematic diagram of the overall structure of the goods identification system described in Embodiment 1 of the present invention.
  • Figure 3 is a schematic diagram of the structure of the shelf described in Example 1 of the present invention.
  • Embodiment 4 is a flowchart of the method for identifying goods in Embodiment 1 of the present invention.
  • FIG. 5 is a flowchart of the model construction steps described in Embodiment 1 of the present invention.
  • FIG. 6 is a flowchart of the steps of acquiring the hand position in Embodiment 1 of the present invention.
  • FIG. 7 is a flowchart of the steps of judging the category of goods described in Embodiment 1 of the present invention.
  • Fig. 8 is a schematic view of the structure when the distance sensor is arranged at the front end of the tray in Embodiment 2 of the present invention.
  • FIG. 9 is a schematic diagram of the structure when the distance sensor is arranged at the rear end of the tray in Embodiment 2 of the present invention.
  • the part When some part is described as being “on” another part, the part may be directly placed on the other part; there may also be an intermediate part on which the part is placed, And the middle part is placed on another part.
  • a component When a component is described as “installed to” or “connected to” another component, both can be understood as directly “installed” or “connected”, or a component is indirectly “mounted to” or “connected to” through an intermediate component “Another component.
  • Embodiment 1 of the present invention provides a goods identification system 100, which includes an electronic device 10, preferably a server or a computer.
  • the electronic device 10 includes a memory 11 and a processor 12; the memory 11 is used to store executable program codes; the processor 12 is connected to the memory 11, and reads the executable program codes. To run the computer program corresponding to the executable program code to execute the steps in the goods identification method.
  • the goods identification system 100 further includes at least one shelf 20 and two or more three-dimensional cameras 30; the shelf 20 is arranged in a closed space 200; the two or more three-dimensional cameras 30 are evenly distributed on the top of the enclosed space 200, and the field of view of the three-dimensional camera 30 covers the entire bottom surface of the enclosed space 200.
  • the goods identification system 100 further includes at least one shelf 20, at least one sensor 40, and at least one processor 12; each shelf 20 includes at least one tray 21, and each tray At least one product is placed on 21; the sensor 40 is set in each shelf 20 to obtain real-time sensing values; the processor 12 is connected to the sensor 40, and according to the real-time sensing value of the sensor 40 The change difference value judges the change of the quantity of goods on the shelf 20, and generates a goods decrease signal or a goods increase signal.
  • the processor 12 After the three-dimensional camera 30 realizes the identification of the product category, the processor 12 obtains the category of the product that is removed or put back from the tray 21, and then uses the change difference of the real-time sensing value of the sensor 40 Calculate the number of goods on the shelf 20 that have changed.
  • the sensor 40 is a weight sensor and is arranged under a tray 21; the real-time sensing value is the real-time weight value of the tray 21 and the goods on the tray 21.
  • the three-dimensional camera 30 is set on the top of the closed space for product identification, and the sensor 40 is set to provide a pre-trigger signal, that is, a signal for product reduction or a signal for product increase.
  • the three-dimensional camera 30 is used to obtain each user The position of the key points of the hand, intercept multiple pictures of the space around the hand during a certain period of time from the video stream, and then determine the type of goods that have been removed or put back. The specific judgment method is described in more detail below statement.
  • this embodiment there is no need to process the complete image collected by the camera 30, and the actual image data that needs to be processed by the computer is less, which can effectively reduce the amount of calculation of the server or the computer and reduce the hardware requirements of the computer.
  • This embodiment can effectively prevent the server or computer from needing to process a large amount of background pixel data that has nothing to do with the goods, avoiding a huge waste of computer computing resources, and avoiding problems such as slow recognition of goods, prone to stalls, long response times, and high error rates. , It can improve the recognition efficiency, reduce the jam phenomenon, reduce the hardware configuration requirements of the computer, and reduce the hardware cost.
  • the pallets 21 can be arranged on the shelf 20 in a manner parallel or flush with each other, and the pallets 21 are detachably connected to the shelf 20.
  • Each tray 21 is an open box, which can be placed with one or more kinds of goods.
  • the same type of goods placed on the same tray 21 have the same weight value, and the different types of goods have different weight values.
  • the judgment of the type of goods is not based on the weight sensor. Therefore, multiple goods can be placed on the same pallet.
  • the change in the value of the weight sensor is only used to determine whether there is an event of taking the goods or placing the goods. If it occurs, provide it to A trigger signal from the server or computer to record the trigger time, so that the server or computer can intercept multiple pictures of the space around the user's hand close to the shelf during a certain period of time from the video stream according to the trigger time, and then determine whether to be taken or put The type of goods returned.
  • the sensor 40 is a weight sensor, which is arranged under a tray 21 to accurately obtain real-time sensing values; the real-time sensing values are real-time sensing values of goods on the tray 21 Weight value; wherein, the processor 12 determines whether the change difference of the real-time sensing value of the weight sensor is a positive number, if it is a positive number, it generates a goods increase signal; if it is a negative number, it generates a goods decrease signal.
  • the weight sensor is connected to the processor 12 of an electronic device 10 (such as a server or a computer), and the processor 12 can obtain the real-time sensing value of the sensor 40 in real time, and according to the change difference of the real-time sensing value of the sensor 40
  • the change of the quantity of goods on the shelf 20 is judged, and a goods change signal is generated, including a goods decrease signal or a goods increase signal.
  • the weight value of each kind of goods is the same or similar.
  • the electronic device judges whether the goods have been taken or put back according to the signal type, records the weight of the goods taken or put back, and records the time point when the goods are taken or put back and the corresponding shelf where the goods change 20 And the position of the tray 21.
  • the processor 12 can further determine the type and quantity of the product to be removed or put back, and to determine based on the video stream. The mutual verification of the types of goods will further improve the accuracy of the identification of the types of goods. If combined with the user's real-time location, the processor 12 can further determine the identity of the user who took or put the goods back.
  • the memory 11 is used to store executable program code; the processor 12 reads the executable program code to run the computer program corresponding to the executable program code to execute a method for goods identification Several steps in, including the following steps S2 ⁇ S10.
  • the goods identification method specifically includes the following steps S1 to S10.
  • the hardware setting step set at least one shelf 20 in a closed space 200, and place at least one kind of goods on at least one tray 21 of the shelf 20; set two or more evenly distributed three-dimensional cameras on the top of the closed space 200 30.
  • the field of view of the three-dimensional camera 30 covers the entire bottom surface of the enclosed space 200.
  • At least one sensor 40 is provided under the tray 21 to obtain real-time sensing values; wherein the sensor 40 and the camera 30 are connected to the processor 12.
  • the processor 12 determines whether the quantity of goods on the shelf 20 is increasing or decreasing according to the change difference of the real-time sensing value of the weight sensor, and generates a goods change signal , Such as a signal for a decrease in goods or a signal for an increase in goods.
  • the model building step is to build a product identification model to identify at least one product.
  • a product recognition model is constructed through a large number of appearance pictures of each product, and the type of the product in the picture can be recognized according to the picture input to the recognition model.
  • the user identification step when a user enters the closed space 200, or before entering the closed space 200, obtains the user's identity information.
  • the computer determines that a certain shopping event has occurred, it can determine the identity of the consumer in the event, so that it is convenient to record their shopping behavior.
  • the user image collection step is to collect real-time images of each user in a closed space 200 in real time; it is understandable that the real-time image collection is realized by the three-dimensional camera 30, and two or more three-dimensional cameras 30 are evenly distributed in At the top of the enclosed space 200, preferably a plurality of three-dimensional cameras 30 are arranged around the shelf 20 and facing the shelf, so that the goods can be photographed when the goods are removed or put back.
  • the signal acquisition step is to acquire a goods change signal;
  • the goods change signal is a goods decrease signal or a goods increase signal;
  • the goods change signal includes the position of the pallet 21 where the goods change and the weight change of the pallet 21 where the goods change Time period.
  • the pick-and-place state judgment step judge the pick-and-place state of the goods according to the goods change signal; when the goods change signal is a goods reduction signal, judge that there are goods on the shelf 20 that have been taken away; when the goods change When the signal is a signal for adding goods, it is judged that there are goods placed on the shelf 20.
  • the detection time period calculation step is to calculate the range of the detection time period, where the detection time period is a preset time period before the time point T1 or after the time T2; when the goods change signal is a goods increase signal When the detection time period is from T1-T3 to T1; when the goods change signal is a goods reduction signal, the detection time period is from T2 to T2+T4; where T3 and T4 are The preset duration.
  • An image interception step intercepting a spatial image of the hand of at least one detected user in a detection space during a detection period from the real-time image of at least one user, including consecutive multiple frames of hand space images.
  • This embodiment acquires a product change signal on the shelf 20 in real time, records the weight change period of the pallet where the product change occurs, and intercepts the detection time period before or after the weight change period of the pallet where the product change occurs.
  • Spatial image identify the multi-frame hand space picture that constitutes the image to quickly and accurately determine the category of goods. Since the shooting speed of 3D images is 10-50 frames per second, the computer can continuously acquire multiple pictures for recognition during the detection period.
  • the types of goods placed on each shelf are pre-stored in the computer, and the electronic device can determine the position of the pallet of the goods to be removed or put back according to the goods change signal, thereby inferring the variety of the goods.
  • a possible conclusion The conclusion of the computer using the video stream recognition can be compared with these possible conclusions, so that the conclusion can be drawn more quickly and accurately, with a small amount of calculation, a fast response speed, and low energy consumption.
  • the large number of video streams collected by the three-dimensional camera 30 are only intercepted from the spatial images of the hands of at least one detected user in a detection space during a detection period, including multiple consecutive frames of hands.
  • Partial space pictures effectively eliminate useless background pictures, effectively reduce the amount of data processing, avoid huge waste of computer computing resources, avoid problems such as slow recognition of goods, prone to jams, long response times, and high error rates, and effectively reduce the computer Hardware configuration requirements, thereby reducing hardware costs.
  • the electronic device can accurately identify the goods that have been taken or put back. kind of. Even if the user puts the goods in the wrong tray position, the electronic device can determine the type of the returned goods in a short time, and then judge the shipped product according to the difference between the single product weight value of the type of goods and the change in the pallet induction weight value. Therefore, the product is deleted from the user’s shopping database, which avoids the problem of mishandling and misplacement from the root cause, and improves the user experience.
  • the model construction step S2 includes steps S21 to S22.
  • the sample collection step is to collect multiple sets of picture samples, each set of picture samples includes multiple sample pictures of a kind of goods at multiple angles; a set of picture samples of the same kind of goods are set with the same group identification, the group The identification is the type of goods corresponding to the set of picture samples.
  • the model training step is to train the convolutional neural network model according to each sample picture in the multiple sets of picture samples and its group identification to obtain the goods recognition model.
  • steps S101-S104 are further included.
  • the detection space range calculation step is to calculate a detection space range according to the position of the shelf 20 where the goods change; the detection space is the internal space of the shelf 20 and/or a preset space in front of the shelf 20
  • the length of the detection space is consistent with the length of the shelf 20
  • the width of the detection space is 0.1 to 1 meter
  • the height of the detection space is 0.1 to 2.5 meters.
  • the shape of the detection space is preferably a rectangular parallelepiped, which facilitates the identification of the goods taken by the hand, and the size of the detection space is smaller than the area of the front end of the shelf 20.
  • the hand position obtaining step is to obtain the position of any key point of each user's two hands; the hand key point is preferably the hand center point.
  • the step of judging the detected user is to compare the two hand positions of each user with the range of the detection space. When at least one hand of a user is located in the detection space within the detection time period, the The user is the detected user.
  • the step of calculating the hand space range is to calculate the hand space range where the hand of the detected user is located in the detection space.
  • the shape of the hand space includes a sphere or a cube; and/or, the center point of the hand space is a key point of the user's hand in the detection space.
  • the hand position acquisition step S12 and the detection space range calculation step S11 are not related to each other, but they are located after the user image acquisition step S4 and before the detected user determination step S13.
  • the hand position acquiring step S102 specifically includes the following steps S1021 to S1022.
  • S1021 a real-time image acquisition step, acquiring a three-dimensional image of the enclosed space 200 in real time, and decomposing the three-dimensional image into a three-dimensional image.
  • S1022 In the key point detection step, at least one frame of three-dimensional image is input to a skeleton tracking model, and at least one key point coordinate of the user's body is output, including the key point coordinate of the user's hand.
  • the skeleton tracking model is a 3D pose estimation model based on deep learning, which can record the movement trajectories of multiple bone key points of each user in a specific space in real time.
  • the 3D pose estimation model is a prior art, which is a 3D model view extraction method based on panoramic images and multi-channel CNN. It is constructed by acquiring the position features of the 3D model surface and the direction features of the 3D model surface in the initial panoramic image.
  • the scale network and multi-channel convolutional neural network take the position feature of the 3D model surface and the direction feature of the 3D model surface as input, and perform network training and the similarity measurement between two different 3D models, so as to obtain the user's body
  • the coordinates of the key points including the key point coordinates of the user's hand.
  • the step S10 of determining the category of goods includes the following steps S1001 to S1002.
  • the group identification obtaining step input the at least one frame of hand space picture into the goods recognition model in sequence, and obtain the group identification corresponding to each frame of hand space picture, and identify at least one group that may appear Don’t mark it as a possible conclusion.
  • an identification credibility calculation step calculating the credibility of each group identification, where the credibility is the number of each group identification in the possibility conclusion and all groups in the possibility conclusion The ratio of the total number of different identifications; the category of the goods corresponding to the group identification with the most credibility is the category of the goods displayed on the hand picture.
  • the reliability is a high probability result of recognition, that is, the recognition result can be accurately obtained when the reliability is greater than 50%.
  • the method further includes:
  • the step of calculating the quantity of goods is to calculate each pallet 21 according to the change difference of the real-time sensing value of the sensor 40 installed at each pallet 21, the single product characteristic value of each type of goods, and the type of goods that change on each pallet 21 The changing quantity of the goods on the pallet 21.
  • the feature value of a single product is the single product weight value; according to the change of the real-time weight sensing value of the weight sensor at the bottom of each tray 21, the difference of each type of product
  • the weight value of a single product and the types of goods that have changed on the pallet 21 can be used to calculate the number of changes in the goods on each pallet 21.
  • the feature value of a single product is the length value of the single product; according to the change difference of the real-time distance sensing value of the distance sensor above each tray 21, the difference of each type of goods
  • the weight value of a single product and the types of goods that have changed on the pallet 21 can be used to calculate the number of changes in the goods on each pallet 21.
  • the processor 12 obtains the product category that has been removed or put back from the tray 21, and then changes the real-time sensing value of the sensor 40 The difference value judges the change quantity of the goods on the shelf 20.
  • the sensor 40 is a weight sensor
  • the change quantity of the goods that are taken or put back can be calculated according to the weight difference; when the sensor 40 is away from the sensor, it can be calculated according to the distance change amount to be taken away or The changed quantity of the returned goods.
  • the method further includes:
  • the shopping database update step when the goods change signal is a goods decrease signal, enter the shopping information of the detected user into the user’s shopping database; when the goods change signal is a goods increase signal, from the The shopping information of the detected user is deleted from the shopping database of the detected user; the shopping information includes the types of goods on hand of the detected user during the detection time period.
  • the server or computer can accurately determine that the user has not consumed the goods, thereby removing the goods from the user’s shopping record. remove. If another user retrieves the product from the wrong location, the server or computer can still accurately determine the user's shopping behavior and add the product to the user's shopping record. Therefore, this embodiment can effectively solve the problem of users randomly placing goods and mishandling them.
  • the wrong location must be on a shelf in a closed space. If an item is placed on the ground by a user, or placed in a location other than the shelf, it will be deemed to have been purchased by the user.
  • the embodiment 2 includes most of the technical features of the embodiment 1, and the difference is that the sensor 40 in the embodiment 2 is a distance sensor instead of the weight sensor in the embodiment 1.
  • the sensor 40 may also include a distance sensor and a weight sensor at the same time.
  • the pallet 21 is closely arranged in a row from the front end of the pallet backward;
  • the sensor 40 It is a distance sensor and is located on the same line as the goods that are arranged in a row.
  • the real-time sensing value is the total length of at least one item; the processor 12 judges the change of the real-time sensing value of the distance sensor Whether the difference is a positive number, if it is positive, a signal for increase in goods is generated; if it is negative, a signal for decrease in goods is generated.
  • the real-time sensing value is the difference between the length of the tray 21 in the front and rear direction and the total length of the at least one item; the processing The device 12 judges whether the change difference of the real-time sensing value of the distance sensor is a positive number, if it is a positive number, it generates a goods decrease signal; if it is a negative number, it generates a goods increase signal.
  • step S1 in the hardware setting step of step S1, at least one item is closely arranged in a row from the front end to the back on a tray 21; the sensor 40 is a distance sensor, and is arranged at the front end of the tray 21 or The rear end is located on the same straight line as the goods arranged in a row; the real-time sensing value is the total length of at least one product; or, the length of the tray 21 in the front and rear direction and the total length of the at least one product Difference.
  • the processor 12 judges whether the quantity of goods on the shelf 20 is increasing or decreasing according to the change difference of the real-time sensing value of the distance sensor, and generates a goods change signal, such as Goods decrease signal or goods increase signal.
  • This embodiment acquires a product change signal on the shelf 20 in real time, records the weight change period of the pallet where the product change occurs, and intercepts the detection time period before or after the weight change period of the pallet where the product change occurs.
  • Spatial image identify the multi-frame hand space pictures that constitute the image to quickly and accurately determine the category of goods. Since the shooting speed of 3D images is 10-50 frames per second, the computer can continuously acquire multiple pictures for recognition during the detection period. After identifying the type of shipment, combined with the length of each item stored in the computer, the number of items of that type that have been removed or put back can be calculated.
  • the server or computer can accurately determine that the user has not consumed the goods, thereby removing the goods from the user’s shopping record. remove. If another user retrieves the product from the wrong location, the server or computer can still accurately determine the user's shopping behavior and add the product to the user's shopping record. Therefore, this embodiment can effectively solve the problem of users randomly placing goods and mishandling them.
  • the beneficial effect of the present invention is to provide a product identification system 100, a product identification method, and an electronic device 10 to monitor the shelf 20 of the product identification system 100 in real time, obtain a product change signal in real time, and record the weight change period of the pallet where the product changes. , Intercept the space image of the hand in a preset time period before or after the weight change period of the pallet where the goods change occurs, and identify the multi-frame hand space image to quickly and accurately determine the type of goods.
  • the present invention can also accurately identify the quantity of goods that have been taken away or put back, so as to adjust and update the user's shopping record in real time. Even if multiple different types of goods are placed in the same tray 21 on a shelf 20, the electronic equipment can accurately identify the type and quantity of the goods that are taken or put back, and effectively solve the problem of mishandling and misplacement from the root cause. The problem has greatly improved the user experience and is conducive to the promotion of applications.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本发明提供一种货品识别方法、货品识别系统及电子设备。

Description

货品识别方法、货品识别系统及电子设备 技术领域
本发明涉及一种用于零售业货品的货品识别技术，具体地说，涉及一种货品识别方法、货品识别系统及电子设备。
背景技术
传统零售业的购物方式，每一家超市货便利店需要有专门的销售人员和收款人员，人力成本较高。随着电子支付技术、身份感知技术及云计算技术的发展，无人超市项目在技术上具备很高的可行性。
在无人超市项目中，急需解决的一个基本问题就是用户选购货品的判断和记录问题，具体地说，计算机或服务器需要准确判定用户从货架上取走或放回至货架的货品的种类、数量及单价等，以便自动为用户实现结算。
现有的无人超市解决方案中，也曾提出采用影像识别技术来判断货品的种类，一般需要采用多个安装在货架上的摄像头实时监控货架前方的空间，获取包含被取走或被放回的货品的多帧图片，在这些图片中，货品可能只占整幅图的3-10％，但是需要将整张图片输入至货品识别模型，计算机需要处理大量与货品无关的背景像素点的数据，造成计算机运算资源的巨大浪费，导致出现货品识别缓慢、容易发生卡顿、响应时间长、出错率高等问题。为了提升识别效率、减少卡顿，经营者只能尽力提升计算机的硬件配置，造成硬件成本过高。
现有技术中，同种类的货品尽量摆放在同一货架或托盘上，如果用户将某货品从一货架或一托盘取下后，将其放置到另一货架或托盘上，就会造成错拿乱放的问题，对于现有技术的一些解决方案来说，在购物过程中，用户将已拾取的货品放回货架时必须放回原位。当用户在将已拾取的货品错放至其他位置时，计算机不能精准识别被放回货品的种类及数量，不能及时准确地更新购物数据库，用户将货品放回至货架的错误位置后还要支付该商品费用，造成识别错误，影响用户体验。用户将错拿的货品乱放至错误的货架或托盘，进而导致 该用户购物记录出错的问题，被业内称为错拿乱放问题。
现有技术的一些方案中，在每一货架上安装多个摄像头，用以监控货品状态；同时，在超市顶部设置多个摄像头，用以判断用户位置，同一空间内使用大量摄像头也会造成成本过高的问题。摄像头采集的视频或图片是覆盖面比较大的区域，其内容上大部分是与需要被识别的货品无关的背景影像，如果每次处理视频或图片等影像资料都要处理整个视频或图片的全部信息，计算机或服务器的运算量就会非常大，对计算机的配置要求会很高，造成硬件成本过高的问题。如果能只用顶部的一组摄像头就能实现用户定位追踪功能和实现货品种类的识别，就可以进一步降低硬件成本、维护成本和运营成本。
因此，市场上需要有一种新的货品识别方案，利用优化的软件方法来解决现有的技术问题。
发明内容
本发明的目的在于，提供一种货品识别方法、货品识别系统及电子设备，以解决无人便利店的货品识别问题，以解决用户在购物中因错拿乱放货品导致其购物记录出错的问题，以解决运算量过大、硬件成本过高的问题。
为实现上述目的，本发明提供一种货品识别方法，包括如下步骤：用户影像采集步骤，实时采集每一用户在一封闭空间内的实时影像；至少一货架被设置于所述封闭空间内，至少一种货品被摆放至所述货架的至少一托盘上；信号获取步骤，获取一货品变动信号；所述货品变动信号为货品减少信号或货品增加信号；所述货品变动信号包括发生货品变动的托盘的位置及发生货品变动的托盘的重量变动时段；影像截取步骤，从至少一用户的实时影像中，截取在一检测时间段内至少一被检测用户在一检测空间内的手部所处的空间影像，包括连续多帧手部空间图片；以及货品种类判断步骤，根据所述多帧手部空间图片及一货品识别模型判断在所述检测时间段内所述被检测用户手上货品的种类。
本发明还提供一种电子设备，包括存储器以及处理器；所述存储器用于存储可执行程序代码；所述处理器连接至所述存储器，通过读取所述可执行程序 代码来运行与所述可执行程序代码对应的计算机程序，以执行上述货品识别方法中的步骤。
本发明还提供一种货品识别系统，包括所述电子设备。
进一步地，所述货品识别系统还包括至少一货架以及两个以上三维摄像头；所述货架设置于一封闭空间内；所述两个以上三维摄像头平均分布于所述封闭空间顶部，所述三维摄像头的视野范围覆盖所述封闭空间的全部底面。
本发明的有益效果在于，提供一种货品识别方法、货品识别系统及电子设备，实时监控货品识别系统的货架，实时获取一货品变动信号，记录发生货品变动的托盘的重量变动时段，截取在发生货品变动的托盘的重量变动时段之前或之后的一预设时间段内截取手部所处的空间影像，识别构成该影像的多帧手部空间图片从而快速准确地判断货品种类。
进一步地，本发明还可以准确识别被取走或被放回的货品的数量，从而实时调整用户的购物记录，有效减少计算机运算量和降低硬件成本，响应速度快且能耗很小，还可以有效解决错拿乱放的问题，即使在一个货架上的同一个托盘内放置多种不同种类的货品，也可准确识别出被拿取或放回的货品的种类及数量，从而对于用户的购物信息判断更加精准，提升了用户体验，便于推广应用。
附图说明
图1为本发明实施例1中所述货品识别系统的结构示意图；
图2为本发明实施例1中所述货品识别系统的整体结构示意图；
图3为本发明实施例1中所述货架的结构示意图；
图4为本发明实施例1中所述货品识别方法的流程图；
图5为本发明实施例1中所述模型构建步骤的流程图；
图6为本发明实施例1中所述手部位置获取步骤的流程图；
图7为本发明实施例1中所述货品种类判断步骤的流程图；
图8为本发明实施例2中当所述距离传感器设置于所述托盘的前端时的结 构示意图；
图9为本发明实施例2中当所述距离传感器设置于所述托盘的后端时的结构示意图。
图中各个部件标号如下：
10、电子设备，11、存储器，12、处理器，20、货架，
21、托盘，30、三维摄像头，40、传感器，
100、货品识别系统，200、封闭空间。
具体实施方式
以下参考说明书附图完整介绍本发明的优选实施例，使其技术内容更加清楚和便于理解。本发明可以通过许多不同形式的实施例来得以体现，其保护范围并非仅限于文中提到的实施例。
在附图中，结构相同的部件以相同数字标号表示，各处结构或功能相似的部件以相似数字标号表示。本发明所提到的方向用语，例如，上、下、前、后、左、右、内、外、上表面、下表面、侧面、顶部、底部、前端、后端、末端等，仅是附图中的方向，只是用来解释和说明本发明，而不是用来限定本发明的保护范围。
当某些部件被描述为“在”另一部件“上”时，所述部件可以直接置于所述另一部件上；也可以存在一中间部件，所述部件置于所述中间部件上，且所述中间部件置于另一部件上。当一个部件被描述为“安装至”或“连接至”另一部件时，二者可以理解为直接“安装”或“连接”，或者一个部件通过一中间部件间接“安装至”或“连接至”另一个部件。
实施例1
如图1所示，本发明实施例1提供一种货品识别系统100，包括电子设备10，优选服务器或计算机。
如图1所示，电子设备10包括存储器11以及处理器12；所述存储器11用于存储可执行程序代码；所述处理器12连接至所述存储器11，通过读取所述可 执行程序代码来运行与所述可执行程序代码对应的计算机程序，以执行货品识别方法中的步骤。
如图2所示，本实施例中，所述货品识别系统100还包括至少一货架20以及两个以上三维摄像头30；所述货架20设置于一封闭空间200内；所述两个以上三维摄像头30平均分布于所述封闭空间200顶部，所述三维摄像头30的视野范围覆盖所述封闭空间200的全部底面。
本实施例中不必同时在所述货架20上以及所述货架20顶端设置多个摄像头，用以分别监控货品状态和判断用户位置，实现了只用顶部的一组摄像头就能实现用户定位追踪功能和实现货品种类的识别，可以降低设置多个摄像头的硬件成本、维护成本和运营成本。
如图1～3所示，本实施例中，所述货品识别系统100还包括至少一货架20、至少一传感器40以及至少一处理器12；每一货架20包括至少一托盘21，每一托盘21上被放置有至少一货品；所述传感器40被设置于每一货架20内，用以获取实时感应值；所述处理器12连接至所述传感器40，根据所述传感器40的实时感应值的变化差值判断所述货架20上的货品数量的变化，生成货品减少信号或货品增加信号。在所述三维摄像头30实现货品种类的识别后，所述处理器12获取从所述托盘21内被取走或放回的货品的种类，再通过所述传感器40的实时感应值的变化差值计算所述货架20上发生变动的货品的数量。本实施例中，传感器40为重量传感器，被设置于一托盘21的下方；所述实时感应值为托盘21及托盘21上货品的实时重量值。
本实施例在封闭空间的顶部设置所述三维摄像头30进行货品识别，并且设置所述传感器40用以提供一个前置触发信号，即货品减少信号或货品增加信号，利用三维摄像头30获取每一用户的手部关键点的位置，从视频流中截取部分时间段内手部周围空间的多张图片，进而判断被取走或被放回的货品的种类，具体的判断方法在下文中有较为详细的陈述。
本实施例无需处理摄像头30采集到的完整影像，实际需要计算机处理的图 片数据较少，可以有效减少服务器或计算机的运算量，降低计算机的硬件需求。本实施例可以有效避免服务器或计算机需要处理大量与货品无关的背景像素点的数据，避免造成计算机运算资源的巨大浪费，避免出现货品识别缓慢、容易发生卡顿、响应时间长、出错率高等问题，可以提升识别效率、减少卡顿现象，降低了计算机的硬件配置要求，降低了硬件成本。
所述托盘21可以多个彼此平行或平齐的方式设置在所述货架20上，所述托盘21可拆卸式连接至所述货架20。每一托盘21皆为一个敞口的盒体，可以被放置有一种或多种货品，放置于同一托盘21的同种类货品具有相同的重量值，不种类货品具有不同的重量值。
本实施例中判断货品的种类并非是基于重量传感器，因此同一个托盘上可以放置多种货品，重量传感器的数值变化仅用于判断是否发生取走货品或放置货品的事件，如果发生，提供给服务器或计算机一个触发信号，记录触发时间，以便服务器或计算机得以根据触发时间从视频流中截取部分时间段内靠近货架的用户的手部周围空间的多张图片，进而判断被取走或被放回的货品的种类。
如图3所示，本实施例中，所述传感器40为重量传感器，被设置于一托盘21的下方，可以精准的获取实时感应值；所述实时感应值为所述托盘21上货品的实时重量值；其中，所述处理器12判断所述重量传感器的实时感应值的变化差值是否为正数，若为正数，生成货品增加信号；若为负数，生成货品减少信号。
在本实施例中，当有某一货品被放回在所述托盘21内时，该货品所处托盘21下方的重量传感器采集的重量值数据就会变大，变化差值为正数；当有某一货品从所述托盘21内被取走时，该货品所处托盘21下方的重量传感器采集的重量值数据就会变小，变化差值为负数。
所述重量传感器连接至一电子设备10(如服务器或计算机)的处理器12，处理器12可以实时获取所述传感器40的实时感应值，并根据所述传感器40的实时感应值的变化差值判断所述货架20上的货品数量的变化，生成一货品变动 信号，包括货品减少信号或货品增加信号。
在本发明的另一实施例中，为了减少数据运算量和降低感知出错率，优选地，在每一托盘21上方仅放置同一种货品，每一种货品的重量值相同或相近似。电子设备根据信号类型判断是否有货品被取走或被放回，记录被取走或被放回的商品重量，并记录货品被取走或被放回的时间点和对应发生货品变动的货架20及托盘21的位置。结合预先存储在电子设备10中的每一货品的重量值和货架20及托盘21的位置编号，处理器12可以进一步判断被取走或被放回货品的种类和数量，与基于视频流判断出的货品种类互相验证，进一步提升货品种类识别的准确率。若结合用户的实时位置，处理器12可以进一步判断取走或放回货品的用户的身份。
货品识别系统100中，存储器11用于存储可执行程序代码；处理器12通过读取所述可执行程序代码来运行与所述可执行程序代码对应的计算机程序，用以执行一种货品识别方法中的若干步骤，包括以下步骤S2～S10。
如图4所示，所述货品识别方法具体包括如下步骤S1～S10。
S1、硬件设置步骤，在一封闭空间200内设置至少一货架20，在货架20的至少一托盘21上摆放至少一种货品；在所述封闭空间200顶部设置平均分布的两个以上三维摄像头30，所述三维摄像头30的视野范围覆盖所述封闭空间200的全部底面。在托盘21下方设置至少一传感器40，用以获取实时感应值；其中，传感器40、摄像头30连接至处理器12。当一货架20的托盘21上有货品被取走或被放回时，处理器12根据重量传感器的实时感应值的变化差值判断货架20上的货品数量是增加或是减少，生成货品变动信号，如货品减少信号或货品增加信号。
S2、模型构建步骤，构建货品识别模型，用以识别至少一种货品。通过每种货品的大量外观图片构建货品识别模型，可以根据输入该识别模型的图片识别出图片中货品的种类。
S3、用户身份识别步骤，一用户在进入所述封闭空间200时，或者，在进 入所述封闭空间200前，获取该用户的身份信息。当计算机判断出某一购物事件发生时，可以判断出该事件中消费者的身份，便于记录其购物行为。
S4、用户影像采集步骤，实时采集每一用户在一封闭空间200内的实时影像；可理解的是，通过所述三维摄像头30实现实时影像的采集，两个以上所述三维摄像头30平均分布于所述封闭空间200顶部，优选多个三维摄像头30设置在所述货架20的周围，并朝向货架，以便在货品被取走或放回时能够拍摄到该货品。
S5、信号获取步骤，获取一货品变动信号；所述货品变动信号为货品减少信号或货品增加信号；所述货品变动信号包括发生货品变动的托盘21的位置及发生货品变动的托盘21的重量变动时段。
S6、取放状态判断步骤，根据所述货品变动信号判断货品的取放状态；当所述货品变动信号为货品减少信号时，判断所述货架20上有货品被取走；当所述货品变动信号为货品增加信号时，判断有货品被放至所述货架20。
S7、变动时间点获取步骤，根据发生货品变动的托盘的重量变动时段，获取托盘重量发生变动的起始时间点T1及托盘重量停止变动的结束时间点T2。
S8、检测时间段计算步骤，计算检测时间段的范围，所述检测时间段为所述时间点T1之前或所述时间T2之后的一预设时间段；当所述货品变动信号为货品增加信号时，所述检测时间段为T1-T3到T1这一时段；当所述货品变动信号为货品减少信号时，所述检测时间段为T2到T2+T4这一时段；其中，T3、T4为预设时长。
S9、影像截取步骤，从至少一用户的实时影像中，截取在一检测时间段内至少一被检测用户在一检测空间内的手部所处的空间影像，包括连续多帧手部空间图片。
S10、货品种类判断步骤，根据所述多帧手部空间图片及一货品识别模型判断在所述检测时间段内所述被检测用户手上货品的种类。
本实施例在货架20上实时获取一货品变动信号，记录发生货品变动的托盘 的重量变动时段，截取在发生货品变动的托盘的重量变动时段之前或之后的检测时间段内截取手部所处的空间影像，识别构成该影像的多帧手部空间图片从而快速准确地判断货品种类。由于三维影像的拍摄速度为10-50帧/秒，在检测时间段内，计算机可以连续获取多张图片进行识别。
优选地，每一货架上被摆放的货品的种类都被预存在计算机中，电子设备可以根据所述货品变动信号判断被取走或被放回货品的托盘位置，从而推测该货品种类的多种可能性结论。计算机利用视频流识别的结论可以与该些可能性结论相对照，进而可以更快速更精准地得出结论，运算量小、响应速度快且能耗低。
本实施例中，所述三维摄像头30采集到的大量视频流，仅从中截取在一检测时间段内至少一被检测用户在一检测空间内的手部所处的空间影像，包括连续多帧手部空间图片，有效排除无用的背景图片，有效减少数据处理量，避免造成计算机运算资源的巨大浪费，避免出现货品识别缓慢、容易发生卡顿、响应时间长、出错率高等问题，有效降低了计算机的硬件配置要求，从而降低硬件成本。
本实施例中，即使在一个货架20上的同一个托盘21内放置多种不同种类的货品，只要重量传感器感应到重量值发生变化，电子设备就可以准确识别出被取走或放回的货品的种类。即使用户将货品错放至错误的托盘位置，电子设备也可以短时间内判断出被放回货品的种类，进而根据该种类货品的单品重量值和托盘感应重量值的变化差值判断出货品的数量，从而将该货品从用户的购物数据库中删除，从根源上避免了错拿乱放的问题，提升了用户体验。
如图5所示，本实施例中，所述模型构建步骤S2包括步骤S21～S22。
S21、样本采集步骤，采集多组图片样本，每一组图片样本包括一种货品在多角度下的多张样本图片；同一种类货品的一组图片样本被设有相同的组别标识，该组别标识即为该组图片样本对应的货品的种类。
S22、模型训练步骤，根据多组图片样本中的每一样本图片及其组别标识训 练卷积神经网络模型，获取所述货品识别模型。
如图4所示，本实施例中，在所述信号获取步骤S5与所述影像截取步骤S9之间，还包括步骤S101-S104。
S101、检测空间范围计算步骤，根据所述发生货品变动的货架20的位置计算一检测空间的范围；所述检测空间为所述货架20内部空间和/或所述货架20前方一预设的空间；本实施例中，所述检测空间的长度与所述货架20长度一致，所述检测空间的宽度为0.1～1米，所述检测空间的高度为0.1～2.5米。所述检测空间的形状优选长方体，这样便于手部拿取的货品识别，且所述检测空间的尺寸小于所述货架20前端的面积。
S102、手部位置获取步骤，获取每一用户的两个手部的任一关键点的位置；所述手部关键点优选为手部中心点。
S103、被检测用户判断步骤，将每一用户的两个手部位置与所述检测空间的范围对比，当一用户的至少一手部在所述检测时间段内位于所述检测空间内时，该用户即为被检测用户。
S104、手部空间范围计算步骤，计算所述被检测用户位于所述检测空间内的手部所处的手部空间的范围。所述手部空间的形状包括球体或正方体；和/或，所述手部空间的中心点为用户位于检测空间内的手部的关键点。
其中,所述手部位置获取步骤S12与所述检测空间范围计算步骤S11互不关联，但其位于所述用户影像采集步骤S4之后且位于所述被检测用户判断步骤S13之前。
如图6所示，本实施例中，所述手部位置获取步骤S102，具体包括如下步骤S1021～S1022。
S1021、实时影像采集步骤，实时获取所述封闭空间200的三维影像，将所述三维影像分解为三维图。
S1022、关键点检测步骤，将至少一帧三维图输入至一骨骼追踪模型，输出至少一用户身体的关键点的坐标，包括用户手部的关键点坐标。本实施例优选 输入一帧三维图至一骨骼追踪模型即可实现获取用户手部的关键点坐标的功能。如果连续N次执行所述关键点检测步骤，连续输入N帧三维图至骨骼追踪模型，可以得到连续N个用户手部关键点坐标形成的运动轨迹。所述骨骼追踪模型为基于深度学习的3D pose estimation模型，该模型可以实时记录特定空间内的每一个用户的多个骨骼关键点的运动轨迹。所述3D pose estimation模型为现有技术，是基于全景图和多通道CNN的三维模型视图提取方法，通过获取初始全景图中的3D模型表面的位置特征以及3D模型表面的方向特征，来构建多尺度网络与多通道卷积神经网络，将3D模型表面的位置特征以及3D模型表面的方向特征作为输入，进行网络的训练和两种不同3D模型之间的相似性度量，从而能够获取用户身体的关键点的坐标，包括用户手部的关键点坐标。
如图7所示，本实施例中，所述货品种类判断步骤S10包括如下步骤S1001～S1002。
S1001、组别标识获取步骤，将所述至少一帧手部空间图片依次输入至所述货品识别模型，获取每一帧手部空间图片所对应的组别标识，将可能出现的至少一种组别标识作为可能性结论。
S1002、标识可信度计算步骤，计算每一种组别标识的可信度，所述可信度为所述可能性结论中每一种组别标识的数量与所述可能性结论中全部组别标识总数的比值；可信度最大的组别标识所对应的货品的种类即为所述手部图片上显示的货品的种类。
可以理解的是，所述可信度为识别的大概率结果，即在所述可信度大于50％时即可准确获取识别结果。
如图4所示，在所述货品种类判断步骤S10之后，还包括：
S11、货品数量计算步骤，根据安装在每一托盘21处的传感器40的实时感应值的变化差值、每一种类货品的单品特征值以及每一托盘21上发生变动的货品种类计算每一托盘21上的货品的变动数量。
当所述传感器40为位于一托盘21底部的重量传感器时，单品特征值为单 品重量值；根据每一托盘21底部的重量传感器的实时重量感应值的变化差值、每一种类货品的单品重量值以及托盘21上发生变动的货品种类，可以计算出每一托盘21上的货品的变动数量。
当所述传感器40为位于一托盘21上方的距离传感器时，单品特征值为单品长度值；根据每一托盘21上方的距离传感器的实时距离感应值的变化差值、每一种类货品的单品重量值以及托盘21上发生变动的货品种类，可以计算出每一托盘21上的货品的变动数量。
可理解的是，利用三维摄像头30实现货品种类的识别后，处理器12获取从所述托盘21内被取走或被放回的该货品种类，再通过所述传感器40的实时感应值的变化差值判断所述货架20上货品的变化数量。当所述传感器40为重量传感器时，可以根据重量差计算出被取走或被放回的货品的变化数量；当所述传感器40距离传感器时，可以根据距离变化量计算出被取走或被放回的货品的变化数量。
如图4所示，在所述货品种类判断步骤S10之后，还包括：
S12、购物数据库更新步骤，当所述货品变动信号为货品减少信号时，录入所述被检测用户的购物信息至该用户的购物数据库；当所述货品变动信号为货品增加信号时，从所述被检测用户的购物数据库中删除该用户的购物信息；所述购物信息包括在所述检测时间段内所述被检测用户手上货品的种类。
在本实施例中，即使用户将货品放置在错误位置，如其他货架或其他托盘上，服务器或计算机也可以准确判断出用户并未消费该货品，从而将该货品从该用户的购物记录中移除。如果有另一用户从错误的位置将该货品重新取走，服务器或计算机还是可以准确判断出此用户的购物行为，将该商品添加至此用户的购物记录中。因此，本实施例可以有效解决用户将货品胡乱摆放、错拿乱放的问题。但是，该错误位置必须是封闭空间内的货架上，如果有一货品被一用户放在地上，或者放在货架以外的其他位置，都会被视为被该用户购买了此货品。
实施例2
如图8、图9所示，实施例2中包括实施例1中大部分的技术特征，其区别在于，实施例2中的传感器40为距离传感器，而不是实施例1中的重量传感器。在其他实施例中，所述传感器40也可同时包括距离传感器及重量传感器。
具体的，实施例2所述货品识别系统100的所述货架20上，每一托盘上只放置同一种类的至少一货品，在一托盘21上从托盘前端向后被紧密排列为一列；传感器40为距离传感器，且与被排成一列的货品位于同一直线上。
如图8所示，当所述距离传感器设置于所述托盘21的前端时，所述实时感应值为至少一货品的总长度；所述处理器12判断所述距离传感器的实时感应值的变化差值是否为正数，若为正数，生成货品增加信号；若为负数，生成货品减少信号。
如图9所示，当所述距离传感器设置于所述托盘21的后端时，所述实时感应值为托盘21前后方向的长度与所述至少一货品的总长度的差值；所述处理器12判断所述距离传感器的实时感应值的变化差值是否为正数，若为正数，生成货品减少信号；若为负数，生成货品增加信号。
在所述货品识别方法中，在步骤S1的硬件设置步骤中，至少一货品在一托盘21上从前端向后被紧密排列为一列；传感器40为距离传感器，设置于所述托盘21的前端或后端，且与被排成一列的货品位于同一直线上；所述实时感应值为至少一货品的总长度；或者，为所述托盘21前后方向的长度与所述至少一货品的总长度的差值。
当一货架的托盘上有货品被取走或被放回时，处理器12根据距离传感器的实时感应值的变化差值判断货架20上的货品数量是增加或是减少，生成货品变动信号，如货品减少信号或货品增加信号。
本实施例在货架20上实时获取一货品变动信号，记录发生货品变动的托盘的重量变动时段，截取在发生货品变动的托盘的重量变动时段之前或之后的检测时间段内截取手部所处的空间影像，识别构成该影像的多帧手部空间图片从 而快速准确地判断货品种类。由于三维影像的拍摄速度为10-50帧/秒，在检测时间段内，计算机可以连续获取多张图片进行识别。在识别出货品种类后，结合计算机中预存的每一种类货品的单品长度值，可以计算出被取走或被放回的该种类货品的数量。
在本实施例中，即使用户将货品放置在错误位置，如其他货架或其他托盘上，服务器或计算机也可以准确判断出用户并未消费该货品，从而将该货品从该用户的购物记录中移除。如果有另一用户从错误的位置将该货品重新取走，服务器或计算机还是可以准确判断出此用户的购物行为，将该商品添加至此用户的购物记录中。因此，本实施例可以有效解决用户将货品胡乱摆放、错拿乱放的问题。
本发明的有益效果在于，提供一种货品识别系统100、货品识别方法及电子设备10，实时监控货品识别系统100的货架20，实时获取一货品变动信号，记录发生货品变动的托盘的重量变动时段，截取在发生货品变动的托盘的重量变动时段之前或之后的一预设时间段内截取手部所处的空间影像，识别多帧手部空间图片从而快速准确地判断货品种类。
进一步地，本发明还可以准确识别被取走或被放回的货品的数量，从而实时调整更新用户的购物记录。即使在一个货架20上的同一个托盘21内放置多种不同种类的货品，电子设备也可准确识别出被拿取或放回的货品的种类及数量，从根源上有效解决了错拿乱放的问题，很好的提升了用户体验，有利于推广应用。
以上所述仅是本发明的优选实施方式，使本领域的技术人员更清楚地理解如何实践本发明，这些实施方案并不是限制本发明的范围。对于本技术领域的普通技术人员，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。

Claims (15)

  1. 一种货品识别方法，其特征在于，包括如下步骤：
    用户影像采集步骤，实时采集每一用户在一封闭空间内的实时影像；至少一货架被设置于所述封闭空间内，至少一种货品被摆放至所述货架的至少一托盘上；
    信号获取步骤，获取一货品变动信号；所述货品变动信号为货品减少信号或货品增加信号；所述货品变动信号包括发生货品变动的托盘的位置及发生货品变动的托盘的重量变动时段；
    影像截取步骤，从至少一用户的实时影像中，截取在一检测时间段内至少一被检测用户在一检测空间内的手部所处的空间影像，包括连续多帧手部空间图片；以及
    货品种类判断步骤，根据所述多帧手部空间图片及一货品识别模型判断在所述检测时间段内所述被检测用户手上货品的种类。
  2. 如权利要求1所述的货品识别方法，其特征在于，
    在所述用户影像采集步骤之前，还包括：
    用户身份识别步骤，一用户在进入所述封闭空间时，或者，在进入所述封闭空间前，获取该用户的身份信息；
    在所述信号获取步骤之后，还包括：
    取放状态判断步骤，根据所述货品变动信号判断货品的取放状态；当所述货品变动信号为货品减少信号时，判断所述货架上有货品被取走；当所述货品变动信号为货品增加信号时，判断有货品被放至所述货架。
  3. 如权利要求1所述的货品识别方法，其特征在于，
    在所述信号获取步骤与所述影像截取步骤之间，还包括：
    变动时间点获取步骤，根据发生货品变动的托盘的重量变动时段，获取托盘重量发生变动的起始时间点T1及托盘重量停止变动的结束时间点T2；以及
    检测时间段计算步骤，计算检测时间段的范围，所述检测时间段为所述时间点T1之前或所述时间T2之后的一预设时间段；
    当所述货品变动信号为货品增加信号时，所述检测时间段为T1-T3到T1这一时段；
    当所述货品变动信号为货品减少信号时，所述检测时间段为T2到T2+T4这一时段；其中，T3、T4为预设时长。
  4. 如权利要求1所述的货品识别方法，其特征在于，
    在所述信号获取步骤与所述影像截取步骤之间，还包括：
    检测空间范围计算步骤，根据所述发生货品变动的货架的位置计算一检测空间的范围；所述检测空间为所述货架内部空间和/或所述货架前方一预设的空间；
    手部位置获取步骤，获取每一用户的两个手部的任一关键点的位置；
    被检测用户判断步骤，将每一用户的两个手部位置与所述检测空间的范围对比，当一用户的至少一手部在所述检测时间段内位于所述检测空间内时，该用户即为被检测用户；以及
    手部空间范围计算步骤，计算所述被检测用户位于所述检测空间内的手部所处的手部空间的范围。
  5. 如权利要求4所述的货品识别方法，其特征在于，
    所述检测空间的长度与所述货架长度一致，
    所述检测空间的宽度为0.1～1米，所述检测空间的高度为0.1～2.5米；
    所述手部空间的形状包括球体或正方体；和/或，
    所述手部空间的中心点为用户位于所述检测空间内的手部的一关键点。
  6. 如权利要求5所述的货品识别方法，其特征在于，
    所述手部位置获取步骤，具体包括如下步骤：
    实时影像采集步骤，实时获取所述封闭空间的三维影像，将所述三维影像分解为三维图；以及
    关键点检测步骤，将至少一帧三维图输入至一骨骼追踪模型，所述骨骼追踪模型输出至少一用户身体的关键点的坐标，包括用户手部的关键点坐标。
  7. 如权利要求1所述的货品识别方法，其特征在于，
    所述货品种类判断步骤包括如下步骤：
    组别标识获取步骤，将所述至少一帧手部空间图片依次输入至所述货品识别模型，获取每一帧手部空间图片所对应的组别标识，将可能出现的至少一种组别标识作为可能性结论；以及
    标识可信度计算步骤，计算每一种组别标识的可信度，所述可信度为所述可能性结论中每一种组别标识的数量与所述可能性结论中全部组别标识总数的比值；可信度最大的组别标识所对应的 货品的种类即为所述手部图片上显示的货品的种类。
  8. 如权利要求1所述的货品识别方法，其特征在于，
    在所述货品种类判断步骤之后，还包括：
    货品数量计算步骤，根据安装在每一托盘处的传感器的实时感应值的变化差值、每一种类货品的单品特征值以及每一托盘上发生变动的货品种类计算每一托盘上的货品的变动数量；和/或，
    购物数据库更新步骤，当所述货品变动信号为货品减少信号时，录入所述被检测用户的购物信息至该用户的购物数据库；当所述货品变动信号为货品增加信号时，从所述被检测用户的购物数据库中删除该用户的购物信息；所述购物信息包括在所述检测时间段内所述被检测用户手上货品的种类及数量。
  9. 一种电子设备，包括：
    存储器，用于存储可执行程序代码；以及
    处理器，连接至所述存储器，通过读取所述可执行程序代码来运行与所述可执行程序代码对应的计算机程序，以执行如权利要求1-8中任一项所述的货品识别方法中的步骤。
  10. 一种货品识别系统，包括权利要求9所述的电子设备。
  11. 如权利要求10所述的货品识别系统，其特征在于，还包括
    至少一货架，设置于一封闭空间内；以及
    两个以上三维摄像头，平均分布于所述封闭空间顶部，所述三维摄像头的视野范围覆盖所述封闭空间的全部底面。
  12. 如权利要求10所述的货品识别系统，其特征在于，还包括
    至少一货架，每一货架包括至少一托盘，每一托盘上被放置有至少一货品；以及
    至少一传感器，被设置于每一货架内部或外部，用以获取实时感应值；
    其中，所述处理器连接至所述传感器，根据所述传感器的实时感应值的变化差值判断所述货架上的货品数量的变化，生成货品减少信号或货品增加信号。
  13. 如权利要求12所述的货品识别系统，其特征在于，
    所述传感器为重量传感器，被设置于一托盘的下方；
    所述实时感应值为所述托盘及托盘上货品的实时重量值；
    其中，所述处理器判断所述重量传感器的实时感应值的变化差值是否为正数，若为正数，生成货品增加信号；若为负数，生成货品减少信号。
  14. 如权利要求12所述的货品识别系统，其特征在于，
    至少一货品在一托盘上从所述托盘前端向后被紧密排列为一列；
    所述传感器为距离传感器，设置于所述托盘的前端，且与被排成一列的货品位于同一直线上；所述实时感应值为至少一货品的总长度；
    其中，所述处理器判断所述距离传感器的实时感应值的变化差值是否为正数，若为正数，生成货品增加信号；若为负数，生成货品减少信号。
  15. 如权利要求14所述的货品识别系统，其特征在于，
    至少一货品在一托盘上从所述托盘前端向后被紧密排列为一列；
    所述传感器为距离传感器，设置于所述托盘的后端，且与被排成一列的货品位于同一直线上；所述实时感应值为托盘前后方向的长度与所述至少一货品的总长度的差值；
    其中，所述处理器判断所述距离传感器的实时感应值的变化差值是否为正数，若为正数，生成货品减少信号；若为负数，生成货品增加信号。
PCT/CN2021/073058 2020-01-22 2021-01-21 货品识别方法、货品识别系统及电子设备 WO2021147950A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010075247.5 2020-01-22
CN202010075247.5A CN111310610B (zh) 2020-01-22 2020-01-22 货品识别方法、货品识别系统及电子设备

Publications (1)

Publication Number Publication Date
WO2021147950A1 true WO2021147950A1 (zh) 2021-07-29

Family

ID=71147071

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073058 WO2021147950A1 (zh) 2020-01-22 2021-01-21 货品识别方法、货品识别系统及电子设备

Country Status (2)

Country Link
CN (1) CN111310610B (zh)
WO (1) WO2021147950A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310610B (zh) * 2020-01-22 2023-02-10 上海追月科技有限公司 货品识别方法、货品识别系统及电子设备
CN111815852A (zh) * 2020-07-07 2020-10-23 武汉马克到家科技有限公司 一种用于开门自取式售货柜的图像和重力双模自动商品识别系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9830485B1 (en) * 2014-08-25 2017-11-28 Amazon Technologies, Inc. Pick verification using moving RFID tags
CN108985861A (zh) * 2018-08-23 2018-12-11 深圳码隆科技有限公司 一种基于开放式购物环境的购物结算控制方法和装置
CN109002780A (zh) * 2018-07-02 2018-12-14 深圳码隆科技有限公司 一种购物流程控制方法、装置和用户终端
CN109243112A (zh) * 2018-08-23 2019-01-18 深圳码隆科技有限公司 一种开放式环境购物控制方法和装置
CN110347772A (zh) * 2019-07-16 2019-10-18 北京百度网讯科技有限公司 物品状态检测方法、装置及计算机可读存储介质
CN111310610A (zh) * 2020-01-22 2020-06-19 上海追月科技有限公司 货品识别方法、货品识别系统及电子设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108460908A (zh) * 2018-03-02 2018-08-28 北京京东尚科信息技术有限公司 自动售货方法和系统以及自动售货装置和自动贩售机
CN109754209B (zh) * 2019-01-02 2022-06-17 京东方科技集团股份有限公司 货品摆放区域确定方法及装置
CN110472515B (zh) * 2019-07-23 2021-04-13 创新先进技术有限公司 货架商品检测方法及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9830485B1 (en) * 2014-08-25 2017-11-28 Amazon Technologies, Inc. Pick verification using moving RFID tags
CN109002780A (zh) * 2018-07-02 2018-12-14 深圳码隆科技有限公司 一种购物流程控制方法、装置和用户终端
CN108985861A (zh) * 2018-08-23 2018-12-11 深圳码隆科技有限公司 一种基于开放式购物环境的购物结算控制方法和装置
CN109243112A (zh) * 2018-08-23 2019-01-18 深圳码隆科技有限公司 一种开放式环境购物控制方法和装置
CN110347772A (zh) * 2019-07-16 2019-10-18 北京百度网讯科技有限公司 物品状态检测方法、装置及计算机可读存储介质
CN111310610A (zh) * 2020-01-22 2020-06-19 上海追月科技有限公司 货品识别方法、货品识别系统及电子设备

Also Published As

Publication number Publication date
CN111310610B (zh) 2023-02-10
CN111310610A (zh) 2020-06-19

Similar Documents

Publication Publication Date Title
WO2019120040A9 (zh) 目标物定位系统及定位方法
US11393213B2 (en) Tracking persons in an automated-checkout store
CN109446883B (zh) 商品状态识别方法、装置、电子设备及可读存储介质
US11521248B2 (en) Method and system for tracking objects in an automated-checkout store based on distributed computing
JP6194777B2 (ja) 動作判定方法、動作判定装置および動作判定プログラム
WO2021147950A1 (zh) 货品识别方法、货品识别系统及电子设备
US20210174299A1 (en) Method, system and device for collecting sales information of commodities in cabinet, and storage medium
US20190318491A1 (en) System and method for gathering data related to quality service in a customer service environment
CN108470255A (zh) 工作量统计方法及装置、存储介质、计算设备
CN112819001B (zh) 基于深度学习的复杂场景卷烟烟包识别方法和装置
CN113888254A (zh) 货架商品管理方法和电子设备
US11615430B1 (en) Method and system for measuring in-store location effectiveness based on shopper response and behavior analysis
CN111428743B (zh) 商品识别方法、商品处理方法、装置及电子设备
CN104718560B (zh) 用于估计单体的大小的图像监测装置及其方法
CN115601686A (zh) 物品交付确认的方法、装置和系统
US20220129821A1 (en) Retail traffic analysis statistics to actionable intelligence
CN208027373U (zh) 图像处理设备
US20170278112A1 (en) Information processing apparatus, information processing method, and non-transitory computer readable medium
CN110895722B (zh) 信息处理方法、陈列系统及智能陈列架
KR20230165153A (ko) 3차원 기반 행동 패턴 분석 장치 및 방법
CN111861670A (zh) 商品推荐方法、装置、设备及存储介质
CN117934088A (zh) 商品推荐方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21744884

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21744884

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21744884

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 25.01.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21744884

Country of ref document: EP

Kind code of ref document: A1