WO2019120040A9 - 目标物定位系统及定位方法 - Google Patents

目标物定位系统及定位方法 Download PDF

Info

Publication number
WO2019120040A9
WO2019120040A9 PCT/CN2018/117325 CN2018117325W WO2019120040A9 WO 2019120040 A9 WO2019120040 A9 WO 2019120040A9 CN 2018117325 W CN2018117325 W CN 2018117325W WO 2019120040 A9 WO2019120040 A9 WO 2019120040A9
Authority
WO
WIPO (PCT)
Prior art keywords
user
goods
shelf
dimensional
shopping
Prior art date
Application number
PCT/CN2018/117325
Other languages
English (en)
French (fr)
Other versions
WO2019120040A1 (zh
Inventor
冯立男
夏鼎
马捷昱
李庭涛
邬文尧
张一玫
Original Assignee
上海云拿智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海云拿智能科技有限公司 filed Critical 上海云拿智能科技有限公司
Priority to KR1020207002600A priority Critical patent/KR102510679B1/ko
Priority to EP18891454.3A priority patent/EP3731546A4/en
Priority to AU2018386790A priority patent/AU2018386790A1/en
Priority to SG11202003732YA priority patent/SG11202003732YA/en
Publication of WO2019120040A1 publication Critical patent/WO2019120040A1/zh
Publication of WO2019120040A9 publication Critical patent/WO2019120040A9/zh
Priority to US16/812,041 priority patent/US20200202163A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01G—WEIGHING
    • G01G19/00—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0639—Item locations
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F5/00—Show stands, hangers, or shelves characterised by their constructional features
    • A47F5/0018—Display racks with shelves or receptables
    • A47F5/0025—Display racks with shelves or receptables having separate display containers or trays on shelves or on racks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01G—WEIGHING
    • G01G19/00—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups
    • G01G19/52—Weighing apparatus combined with other objects, e.g. furniture
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01V—GEOPHYSICS; GRAVITATIONAL MEASUREMENTS; DETECTING MASSES OR OBJECTS; TAGS
    • G01V7/00—Measuring gravitational fields or waves; Gravimetric prospecting or detecting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • G06Q10/0875—Itemisation or classification of parts, supplies or services, e.g. bill of materials
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • G06Q20/206—Point-of-sale [POS] network systems comprising security or operator identification provisions, e.g. password entry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • G06Q20/208—Input by product or record sensing, e.g. weighing or scanner processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C9/00—Individual registration on entry or exit
    • G07C9/30—Individual registration on entry or exit not involving the use of a pass
    • G07C9/32—Individual registration on entry or exit not involving the use of a pass in combination with an identity check
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
    • G07C9/00—Individual registration on entry or exit
    • G07C9/30—Individual registration on entry or exit not involving the use of a pass
    • G07C9/38—Individual registration on entry or exit not involving the use of a pass with central registration
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0009—Details of the software in the checkout register, electronic cash register [ECR] or point of sale terminal [POS]
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0018—Constructional details, e.g. of drawer, printing means, input means
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/0054—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles
    • G07G1/0063—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/009—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader the reader being an RFID reader
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/12—Cash registers electronically operated
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/12—Cash registers electronically operated
    • G07G1/14—Systems including one or more distant stations co-operating with a central processing unit
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/33—Services specially adapted for particular environments, situations or purposes for indoor environments, e.g. buildings
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/35—Services specially adapted for particular environments, situations or purposes for the management of goods or merchandise
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F10/00—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for
    • A47F10/02—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for for self-service type systems, e.g. supermarkets
    • A47F2010/025—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for for self-service type systems, e.g. supermarkets using stock management systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01G—WEIGHING
    • G01G19/00—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups
    • G01G19/40—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight
    • G01G19/413—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight using electromechanical or electronic computing means
    • G01G19/414—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight using electromechanical or electronic computing means using electronic computing means only
    • G01G19/4144—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight using electromechanical or electronic computing means using electronic computing means only for controlling weight of goods in commercial establishments, e.g. supermarket, P.O.S. systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01G—WEIGHING
    • G01G19/00—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups
    • G01G19/40—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight
    • G01G19/42—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight for counting by weighing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/10—Recognition assisted with metadata
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/12—Acquisition of 3D measurements of objects

Definitions

  • the invention relates to a real-time location positioning and tracking technology for users in the retail industry, and in particular, to a target positioning system and a positioning method.
  • each supermarket and convenience store needs a dedicated salesperson and a cashier, which has a high labor cost.
  • the unmanned supermarket project is technically very feasible.
  • a basic problem that needs to be solved urgently is the user's judgment and record of purchasing goods.
  • the server needs to accurately determine which goods each user has selected, which goods have been put back, and which are taken from the supermarket. What goods are gone.
  • the applicant of the present invention proposes a real-time judgment based on the weight monitoring technology or the image monitoring technology to determine whether any goods on the shelf have been removed or put back. At the same time, it is also necessary to solve the problem of determining the identity of a user who takes or returns the goods.
  • the object of the present invention is to provide a target positioning system and a positioning method, which effectively solve the existing technology that cannot locate and track the target in real time, and that has an ambiguous identity of a user who takes or returns the goods in an unmanned supermarket. problem.
  • the present invention provides a target positioning system, including: a closed space; a three-dimensional image acquisition device for acquiring at least one frame of three-dimensional images in real time, and the three-dimensional image includes all or part of at least one target An image; and a target coordinate acquisition unit provided in a data processing device for establishing a three-dimensional coordinate system in the closed space, and acquiring the target in the three-dimensional coordinate system in real time based on the at least one frame of three-dimensional image The coordinate set or coordinates.
  • the present invention provides a method for positioning an object, including the following steps: a space setting step for setting a closed space; a three-dimensional image acquisition step for acquiring at least one frame of three-dimensional images in real time, wherein An image including all or part of at least one target; and a target coordinate obtaining step for establishing a three-dimensional coordinate system in the closed space, and acquiring a coordinate set or coordinates of a target in real time according to the at least one frame of three-dimensional image .
  • the advantage of the present invention is to provide a target positioning system and a positioning method, which can obtain the real-time position of a specific target in a specific area, and in particular, can obtain the real-time position of each user in an unmanned supermarket, combined with each board Or the location of the product and the status of each item on the shelf, you can accurately determine the identity of the user who removed or returned the item on the shelf, so as to update the user ’s shopping record so that the user can automatically settle after the purchase is completed .
  • FIG. 1 is a top view of an unmanned supermarket according to an embodiment of the present invention.
  • FIG. 2 is a schematic structural diagram of a tray and a shelf plate according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of an overall structure of a shelf according to an embodiment of the present invention.
  • FIG. 4 is a structural block diagram of a user identity recognition system according to an embodiment of the present invention.
  • FIG. 5 is a structural block diagram of a target positioning system according to an embodiment of the present invention.
  • FIG. 6 is a distribution diagram of an image sensor in a closed space according to an embodiment of the present invention.
  • FIG. 7 is a structural block diagram of a goods sensing system based on weight monitoring according to an embodiment of the present invention.
  • FIG. 8 is a structural block diagram of a goods sensing system based on image monitoring according to an embodiment of the present invention.
  • FIG. 9 is a positional relationship diagram of the second camera and the shelf according to an embodiment of the present invention.
  • FIG. 10 is a structural block diagram of a shopping user judgment system according to an embodiment of the present invention.
  • 11 is a structural block diagram of a shopping information recording unit according to an embodiment of the present invention.
  • FIG. 12 is a structural block diagram of a settlement system according to an embodiment of the present invention.
  • FIG. 13 is a flowchart of a method for positioning an object in an embodiment of the present invention.
  • FIG. 15 is a flowchart of steps for obtaining coordinate of an object in an embodiment of the present invention.
  • FIG. 16 is a flowchart of steps for acquiring a position parameter according to an embodiment of the present invention.
  • FIG. 17 is a flowchart of another target positioning method according to an embodiment of the present invention.
  • FIG. 18 is a flowchart of a judgment process for a shopping user in an embodiment of the present invention.
  • FIG. 19 is a flowchart of a shopping information recording step in an embodiment of the present invention.
  • 100 user identification system 101 access control device, 102 identification device; 1021 code scanning device, 1022 identity acquisition unit, 103 user entrance, 104 user exit;
  • 300 goods sensing system based on weight monitoring 301 goods database generation unit, 302 weight value acquisition unit, 303 pick and place status judgment unit, 304 goods database update unit;
  • 400 image sensing-based goods perception system 401 sample acquisition unit, 402 model training unit, 403 real-time picture acquisition unit, 404 goods category acquisition unit, 405 first camera, 406 second camera;
  • 500 shopping user judgment system 501 goods information storage unit, 502 shelf board coordinate storage unit, 503 shelf board and user matching judgment unit, 504 goods and user matching judgment unit;
  • 700 settlement system 701 total amount calculation unit, 702 payment unit.
  • the component When certain components are described as being “on” another component, the component may be placed directly on the other component; there may also be an intermediate component on which the component is placed, And the intermediate component is placed on another component.
  • the two When a component is described as “mounted to” or “connected to” another component, the two can be understood as directly “installed” or “connected” or a component is indirectly “mounted to” or “connected to” through an intermediate component “Another part.
  • This embodiment relates to a target positioning system, which is part of an unmanned sales system for a user unmanned supermarket.
  • a plurality of image acquisition devices are set on the top of the unmanned supermarket space to obtain a target (shopping user). Real-time location and effective tracking in unmanned supermarket space.
  • the unmanned sales system includes an enclosed space 1 in which a plurality of shelves 2 are arranged, and each shelf 2 includes a bracket 3 and a plurality of trays detachably mounted on the bracket 3. 4. Multiple trays 4 are parallel to each other at different heights or flush with each other at the same height. Each tray 4 is provided with a plurality of side-by-side rack plates 5, and at least one kind of goods is placed on each of the rack plates 5. In this embodiment, the goods placed on the shelf plate 5 need to be convenient for the user to remove or put back. Therefore, the end of the shelf plate 5 facing the user is used as the front end of the shelf plate 5.
  • a weight sensing device 6 preferably a rectangular parallelepiped weight sensor, is provided between each shelf plate 5 and the tray 4. A lower surface of one end thereof is connected to the pallet 4, and an upper surface of the other end thereof is connected to the shelf plate 5.
  • each shelf plate 5 is an open box, and one or more kinds of goods can be placed thereon.
  • the goods are standard goods, and the appearance and weight of the same kind of goods are the same or similar.
  • the same kind of goods placed on the same shelf plate 5 have the same weight value, and different kinds of goods have different weight values, and each weight value corresponds to only one type of goods.
  • the weight sensing device 6 can accurately obtain the real-time weight sensing value of the shelf plate 5 and the goods on the upper surface thereof, and accurately sense each change in the weight value of each shelf plate 5, including an increase amount or a decrease amount.
  • This embodiment also includes a data processing device 7, such as a server or a computer.
  • the data processing device 7 is provided with multiple data processing software and has multiple functional modules. It can be connected to multiple hardwares through data lines and implemented in a combination of software and hardware. A variety of functions.
  • this embodiment further includes a user identity recognition system 100 for identifying identity information of each user.
  • the user identification system 100 includes an access control device 101 and an identification device 102.
  • the closed space 1 described in this embodiment is not an absolutely sealed space, but a relatively closed space.
  • the closed space 1 is provided with an entrance and exit, preferably a user entrance 103 and a user Exit 104. All users enter the closed space 1 from the user entrance 103, and exit the closed space 1 from the user exit 104.
  • each entrance and exit of the closed space 1 is provided with an access control device 101, preferably an automatic brake.
  • the identity identification device 102 is configured to obtain identity information of a user, and includes a code scanning device 1021 connected to the data processing device 7 and an identity obtaining unit 1022 in the data processing device 7.
  • the code scanning device 1021 is provided inside or outside the access control device 101 at the user entrance 103, preferably on the outer surface of the automatic gate, for scanning an identification code, preferably a two-dimensional code; the identity acquisition unit 1022 is a data processing device A function module in 7 may obtain the user's identity information according to the identity identification code.
  • the access control device 101 at the user exit 104 does not need to be provided with the identification device 102.
  • each user downloads a special application software (APP) used with unmanned supermarkets to a mobile communication terminal (mobile phone, tablet computer, etc.), registers an account in the application software (APP), and associates it with the payment software.
  • APP application software
  • the payment software contains user registration information and electronic payment information, including user identity information, bank account information, and payment password. After the registration is completed, the user identity information is stored in the user database of the data processing device 7.
  • the application software (APP) in the mobile communication terminal can generate a two-dimensional code.
  • the two-dimensional code stores the user's identity information.
  • the two-dimensional code generated by the application software is used.
  • the code is facing the scanning end of the scanning device 1021. After scanning, the scanning device 1021 decodes the two-dimensional code and transmits the decoding result to the data processing device 7. If the two-dimensional code is identifiable and recognized, The identity information matches the identity information stored in the user database in advance. It is determined that the user's identity is legal.
  • the access control device 101 is opened to allow the user to enter the closed space 1.
  • the access control device 101 at the user entrance 103 is provided with a sensing device, such as an infrared sensor.
  • the access control device 101 detects that someone passes through the access control and then automatically closes.
  • the access control device 101 at the user exit 104 senses that someone is approaching the access control device 101 from inside the closed space 1, the access control device will automatically open. After the user leaves the closed space 1, the access control device 101 perceives someone walking through the gate and then automatically closes.
  • the data processing device 7 may generate a shopping database of the user, and update the shopping database according to each shopping behavior of the user during the user shopping process. Since the mobile communication terminal carried by the user carries out real-time data exchange with the data processing device 7 through the application software (APP), the user's shopping database can also be displayed in the application software (APP) in the mobile communication terminal to form a shopping cart interface so that Users understand their shopping history and subsequent settlement.
  • APP application software
  • the target positioning system 200 includes a closed space 1, a three-dimensional image acquisition device 201, a three-dimensional image acquisition unit, and a target coordinate acquisition unit 202 to acquire each target object. Real-time location in enclosed space 1.
  • the target object described in this embodiment is all or part of the user and its extension.
  • the target positioning system 200 is a user positioning system for obtaining the position of the user as a whole or in part (such as the head, hands, etc.), that is, A coordinate set in the three-dimensional coordinate system.
  • the three-dimensional image acquisition device 201 includes at least one image sensor 2011 for acquiring at least one frame of three-dimensional images in real time.
  • the plurality of image sensors 2011 are evenly distributed on the top of the closed space 1 with the lens facing downward, and the central axis of the lens may be perpendicular to the horizontal plane or may have a certain inclination angle.
  • the field of view of the lens of the image sensor 2011 covers the entire bottom surface of the closed space 1.
  • the three-dimensional image collected by the image sensor includes the user image.
  • the user image refers to all or Partial picture. If there is no one in the closed space, the three-dimensional image at each moment is the same as the previous one. It can be determined that the three-dimensional image at that moment is the background and does not include any user images.
  • Each image sensor 2011 includes a depth image sensor 2012 and an RGB image sensor 2013 and a three-dimensional image integration unit 2014 arranged in parallel.
  • the depth image sensor 2012 continuously acquires multiple frames of depth images; the RGB image sensor 2013 continuously acquires multiple frames of RGB images and three-dimensional image integration.
  • the unit 2014 combines a frame of depth image and a frame of RGB image acquired by the same image sensor 2011 at the same time into a frame of three-dimensional image.
  • the above two sensors acquire synchronously (simultaneous acquisition and the same acquisition frequency), the image sensor 2011 can acquire the same frame number of RGB images and depth images per second, and the three-dimensional image integration unit 2014 can continuously obtain multiple frames of three-dimensional images and transmit them to the data.
  • the target coordinate acquisition unit 202 is a functional module in the data processing device 7.
  • a three-dimensional coordinate system is established in the closed space, and the user is acquired in real-time from the three-dimensional image including the user image in the three-dimensional coordinate system.
  • the target coordinate acquisition unit 202 includes a coordinate system establishment unit 2021, a parameter acquisition unit 2022, a background removal unit 2023, and a target coordinate calculation unit 2024.
  • the coordinate system establishing unit 2021 establishes a three-dimensional coordinate system in the closed space.
  • the center point of the bottom surface of the closed space (the ground of the unmanned supermarket) is selected as the origin of the coordinate system, and the X-axis and Y-axis are set in the horizontal direction. Set the Z axis in the straight direction.
  • a user can use a coordinate set to represent the user ’s position. If precise control and convenient calculation of the position are considered, a specific point in the coordinate set may also be used.
  • the position of the user represents the position of the user. For example, the position of the user may be represented by the coordinates of the highest point (the point with the largest Z-axis value) in the user coordinate set.
  • the parameter acquisition unit 2022 processes the continuous multiple frames of the three-dimensional image including the user image, and obtains a position parameter and a color parameter of each pixel point of each frame of the three-dimensional image;
  • the position parameters are x, y, and z, which represent the pixel point Position coordinates in the three-dimensional coordinate system;
  • the color parameters are r, g, and b, which respectively represent the three primary color intensities of the pixel point.
  • the pixel parameters representing the same position of the user's body and its extensions have the same color parameters r, g, and b. Because the distance between the image sensor at different positions and the user is different, the primary position parameters directly collected by each image sensor are the position coordinates of a point on the user's body and its extension with respect to the image sensor. The primary position parameters collected by the position image sensor are converted into position parameters in a three-dimensional coordinate system established in the closed space.
  • the parameter acquisition unit 2022 includes a sensor coordinate acquisition unit 20221, a relative coordinate acquisition unit 20222, and a coordinate correction unit 20223.
  • the sensor coordinate acquisition unit 20221 acquires the center point of the image sensor that collects the three-dimensional image of the frame (i.e., the depth image sensor 2012 and RGB arranged side by side).
  • each frame of 3D images includes and includes only one user's image. If the color parameters of N pixels that belong to different 3D images and have the same position parameters are the same, and N is greater than 0.9 * M is less than or equal to M, the background removal unit 2023 determines that the N pixels are background pixels, and removes the N background pixels from the M frame 3D image to obtain the M frame without background 3D image, that is, the User image. In successively acquired 3D images, if the color parameters of pixels with the same position parameters belonging to different 3D images are the same, or most of them (such as 90%) are the same, the position of the pixels can be regarded as the background, so that the The pixel is removed from the corresponding three-dimensional image.
  • the target coordinate calculation unit 2024 if the target is all of the user and its extension, the set of position parameters of all pixels in the M-frame backgroundless three-dimensional image is all the coordinates of the user and its extension Set; in the coordinate set, the position parameter of the pixel with the largest parameter z is defined as the coordinates of the user.
  • the remaining pixels can represent the overall trajectory of the user. If the three-dimensional images of M frames that are continuously acquired include each user's three-dimensional image, it is necessary to firstly capture all or part of three-dimensional images of only one user in each M-frame three-dimensional image.
  • a coordinate set of the user's part can be obtained, such as the head, shoulders, elbows, wrists, hands, and so on.
  • the depth image sensor 2012 and the RGB image sensor 2013 are provided with a lens, respectively, and the lens of the depth image sensor 2012 and the RGB image sensor 2013 are arranged side by side and adjacent to each other. If the center axis of the two lenses is set perpendicular to the horizontal plane, the two lenses Will look down on the goods and users in the enclosed space. Under normal circumstances, the two lenses can capture the position coordinate set of the user's head and shoulders. When the user extends his hand, the position coordinate set of the user's arms, elbows, wrists, and hands can also be captured.
  • the user's hands and head position can be mapped, that is, The position of a hand can be obtained in real time, and it can be determined which user the hand belongs to.
  • the field of view of the image sensor 2011 may also cover part of the space outside the entrance and exit.
  • the user's image can be acquired by the image sensor 2011.
  • the entire process of the user using the unmanned sales system, including the identification process at the entrance and exit, the process of entering the enclosed space 1, the process of walking or staying in the enclosed space 1, and the process of leaving the enclosed space 1, are all in the image sensor 2011.
  • the real-time position of a user with a known identity and a part of his body in the closed space 1 can be monitored in real time.
  • the code scanning device 1021 reads the user's two-dimensional code
  • the data processing device 7 can obtain its identity information.
  • the image sensor 2011 starts to locate and track the user's position in real time when the code scanning device 1021 reads the code, and monitors whether the user is connected with a certain user. One shelf matches. When the image sensor 2011 cannot obtain the real-time three-dimensional image of the user, it can be determined that the user's shopping is over, so as to settle the user.
  • This embodiment also includes a goods sensing system to sense the pick-and-place status of each kind of goods in real time. When any kind of goods is removed or put back, the type and quantity of the goods taken out or put back are obtained. .
  • the pick-and-place state includes a stationary state of the goods, a removed state, and a returned state.
  • the goods perception system includes two different technical solutions.
  • this embodiment further includes a goods sensing system 300 based on weight monitoring, which is used to sense the pick-and-place status of each kind of goods in real time. And put back.
  • the goods sensing system 300 based on weight monitoring includes a goods database generation unit 301, a weight value acquisition unit 302, a pick-and-place state determination unit 303, and a goods database update unit 304.
  • the above four units are functional modules in the data processing device 7 and work in cooperation with the shelf 2 provided with the weight sensing device 6 to monitor the real-time weight sensing value of each shelf plate 5 to determine whether any goods have been removed or placed. return. When any kind of goods is taken away or put back, the goods sensing system 300 obtains the kind and quantity of the goods taken out or put back.
  • the goods database generating unit 301 is used to generate a goods database; the goods database includes the goods information of each goods and the weight sensing value of each shelf for placing the goods, and the goods information includes the type of the goods, the weight value of the single product, and The corresponding shelf board number and shelf number of the product, including the product number, product name, model, net content and unit price, etc.
  • the goods database generation unit 301 includes an initialization unit 3011, an information entry unit 3012, and a sensing value initialization unit 3013.
  • the initialization unit 3011 is used for performing initialization processing on a goods database, and establishing a goods database in the memory of the data processing device 7.
  • the information input unit 3012 is used to input the weight value of each item and the item information, and store it into the item database, and enter the weight value of each item on the unmanned supermarket shelf using a keyboard or a scanner.
  • the sensing value initialization unit 3013 is configured to collect a weight sensing value of each shelf board after the goods are placed, and store the weight sensing values in the goods database.
  • the goods information is entered into the data processing device 7 and stored in the goods database.
  • the weight of the shelf board is 100 grams, and the weight of each bottle is 200 grams.
  • the weight value of the shelf board is initialized in the product database after initialization.
  • the product name corresponding to the brand beverage (a kind of herbal tea), net content (195ml), place of origin (Guangdong), unit price (5 yuan), single product weight value (200 grams), shelf number (1), shelf plate Information such as serial number (1-12) and product number (025) is also stored in the goods database.
  • the weight value collecting unit 302 is respectively connected to the weight sensing device 6 in each of the boards 5 through a data line, so as to collect the real-time weight sensing value of each board 5 in real time.
  • the collection time interval is 0.1-0.2 seconds.
  • the real-time weight sensing value is the sensing value of the weight sensor, which represents the weight of each shelf board before the goods are placed on the shelf board 5; after the goods are placed on the shelf board 5, it represents the shelf board and the shelf The total weight value of the goods on the board; the real-time weight sensing value will change when some goods are removed or put back on the shelf board 5.
  • the shelf board is left empty to obtain the sensing value X0 of the weight sensing device 6 (weight sensor).
  • Y (goods weight value) k * X (sensor value) + b
  • the sensor value collected by the weight sensing device 6 (weight sensor) in real time each time, combined with the values of the parameters k and b, can calculate the total weight of the existing goods on each board.
  • the pick-and-place state determination unit 303 is used to determine whether the weight sensing value of each shelf board has changed. If it is smaller, it is determined that there are goods on the shelf board. If it is larger, it is determined that there are items placed on the shelf board. If it is completely unchanged, it means that the goods on the shelf board have not changed at all, and the weight value acquisition unit 302 performs real-time acquisition again.
  • the pick-and-place state determination unit 303 includes a weight difference calculation unit 3031, a weight difference determination unit 3032, and a shelf board information recording unit 3033.
  • the weight difference calculation unit 3031 calculates the difference between the real-time weight sensing value of each board collected in real time and the weight sensing value of the same board stored in the goods database, and records it as the weight difference of each board. For example, in the foregoing example, if the weight of the shelf plate on which a certain brand of beverage is placed changes to 1300 g or 1900 g, the weight difference is recorded as -400 g or 200 g, respectively.
  • the weight difference judging unit 3032 compares the weight difference of at least one shelf board with 0; when the weight difference of a shelf board is less than 0, it is determined that there are goods on the shelf board; when the weight difference of a shelf board When it is greater than 0, it is determined that an item is placed on the shelf board. At this time, it is not determined whether the item is a product that the user has taken off the shelf before, or it may be a user's carry-on item. For example, in the previous example, if the weight difference is -400 grams, it can be considered that the goods are taken away; if the weight difference is 200 grams, it can be considered that there are items placed on the shelf.
  • the shelf board information recording unit 3033 records the shelf board number of the shelf board and the weight difference of the shelf board. For example, if the weight difference in the foregoing example is -400 grams, it is known that the weight of the shelf plate is reduced, and the number (1-12) of the shelf plate is recorded. If the weight difference in the previous example is 200 grams, it is known that the initial weight sensing value of the shelf board is 1700 grams. At this time, the items placed on the shelf board are not necessarily the goods on the original shelf, so it is likely that Products originally belonging to other shelves or personal belongings of the user can optionally generate an alarm signal at this time to remind the management personnel or the user. If necessary, the shelf plate number of the shelf can be displayed on a certain display. So that managers or users can deal with it in a timely manner.
  • the pick-and-place state judgment unit 303 further includes a goods category judgment unit 3034 and a goods quantity calculation unit 3035.
  • the cargo category determination unit 3034 judges the type of the removed product according to the shelf board number and the cargo information corresponding to the shelf board stored in the cargo database. For example, given the number of the shelf board (1-12), if only one product is placed on each shelf board, you can determine the type of the product is a herbal tea, and you can also find out other product information, such as the weight value of a single product. (200 grams), net content (195ml), place of origin (Guangdong), unit price (5 yuan) and so on. If the shelf is placed with a variety of goods, only the possible types and quantities of the removed goods can be initially determined based on the weight difference.
  • the quantity calculation unit 3035 calculates the absolute value of the weight difference of a board and the rack stored in the goods database.
  • the ratio is rounded using the rounding method.
  • the integer obtained is the quantity of the removed product.
  • the weight difference in the previous example is -400 grams
  • its absolute value is 400 grams
  • the ratio of the weight to the weight of a single product (200 grams) is 2, so the ratio is the number of goods removed. Because there may be a small weight difference between multiple goods of the same type, the ratio after direct calculation may not be an integer, and may be closer to an integer, so the ratio needs to be rounded by rounding. , So that you can determine the type and quantity of goods taken away.
  • the cargo category determination unit 3034 judges the type of the returned product according to the shelf board number and the product information corresponding to the shelf board.
  • the quantity calculation unit 3035 calculates the ratio between the absolute value of the weight difference of the shelf plate and the weight value of the single product of the goods corresponding to the shelf plate. The ratio is rounded by the rounding method, and the obtained integer is returned. The quantity of the goods.
  • the goods database update unit 304 is configured to store the real-time weight sensing value to the goods database, and form a new weight sensing value to update the weight sensing value of each shelf in the goods database for the next call and judgment .
  • the beneficial effect of the goods sensing system 300 based on weight monitoring described in this embodiment is to provide a goods sensing solution based on weight monitoring, which can monitor the real-time weight sensing value of the goods on the shelf in real time, and sense the weight change of each board in real time. From the weight changes of all the shelves on the shelf, it can be inferred which kind of goods have been taken away or put back, and the type and quantity of goods taken away or put back can also be judged.
  • this embodiment further includes a goods monitoring system system 400 based on image monitoring, which includes a sample acquisition unit 401, a model training unit 402, a real-time picture acquisition unit 403, and a cargo category acquisition unit 404.
  • the unit is a functional module in the data processing device 7.
  • the goods sensing system 400 based on image monitoring can monitor the real-time image of the area in front of the shelf to determine the type of goods that have been taken or returned.
  • the goods monitoring system 400 based on image monitoring further includes a first camera 405 and a second camera 406.
  • the first camera 405 is connected to the sample collection unit 401 and is used to take pictures of multiple angles and multiple distances of each product.
  • the second camera 406 is connected to the real-time picture acquisition unit 403 and is used to take a real-time picture of the space in front of the shelf.
  • the number of the second cameras 406 is two or four, and the second cameras 406 are disposed outside the shelf 2, and each of the second cameras 406 faces a corner of the shelf 2.
  • the front ends of the plurality of shelf plates 5 of the shelf 2 are located on the same plane, which is called a shelf plane.
  • the second camera 406 is provided with a lens, and the field of view of the lens covers the space in front of the shelf. When the shelf is removed or placed on the shelf, an image of the product being removed or returned is captured by the second camera.
  • the front space of the shelf refers to the space area corresponding to the plane of the shelf in front of the shelf.
  • the front space of the shelf generally refers to a range of 30-50 cm width in front of the shelf.
  • the lens of each second camera 406 faces the space in front of the shelf. Central region.
  • the angle between the central axis of the lens of the second camera 406 and the horizontal plane is 30-60 degrees; and / or, the distance between the lens of the second camera 406 and the upper or lower end of the shelf 2 is 0.8-1.2 meters; and / or, The distance between the lens of the second camera 406 and one side of the shelf 2 is 0.8-1.2 meters, to ensure that the field of vision of the second camera 406 can completely cover the space in front of the shelf.
  • the image of the removal process or the placement process is captured by the second camera 406.
  • the sample collection unit 401 is used to collect at least one set of picture samples, each set of picture samples includes multiple sample pictures of a product at multiple angles; a group of picture samples of the same kind of goods is provided with the same group identifier, and The group identifier represents the type of goods corresponding to the group of picture samples.
  • the first camera 405 needs to take 3,000 to 5000 pictures of different angles and different distances for each kind of goods on the shelf 2 and transmit the pictures to the sample collection unit 401 of the data processing device 7.
  • the six-sided drawings of the appearance of the same kind of goods are the same or similar, so as long as one or several products are selected in the same kind of goods, After taking photos for many times, the training samples of this kind of goods can be sampled.
  • the model training unit 402 is configured to train a convolutional neural network (CNN) model according to each sample picture and a group identifier of each sample picture in a plurality of groups of picture samples to obtain a product identification model.
  • CNN convolutional neural network
  • the convolutional neural network model in this embodiment is a Faster RCNN network model with the smallest amount of calculation and the fastest response speed.
  • the fastest response speed of this model is only about 0.2 seconds, which can be accurately identified in a very short time The type and number of items in the picture.
  • the real-time picture acquisition unit 403 is connected to a plurality of second cameras 406 for continuously acquiring at least one real-time picture of the space in front of the shelf, and each real-time picture includes part or all of one or more pictures of the goods.
  • the second camera 406 can capture the goods in front of the shelf from different angles. Full or partial photo showing the shape, pattern and color of the item.
  • the goods category acquisition unit 404 is configured to obtain the types of goods displayed in the real-time picture according to the real-time picture and the goods identification model.
  • the real-time picture collection unit 403 collects multiple real-time pictures in a certain period of time, and then inputs them to the goods identification model after preprocessing to determine the group identifier corresponding to the picture in the period, and judges within the period according to the group identifier. The kind of goods photographed.
  • the beneficial effect of the product monitoring system 400 based on image monitoring is that it can monitor the image of the space in front of the shelf in real time to determine whether any product has been removed from the shelf or returned to the shelf.
  • the convolution algorithm in machine learning is used to infer the shipment. The conclusion of the kind and quantity of the possibility is selected, and the result with the highest credibility is selected as the final conclusion.
  • this embodiment further includes a shopping user judgment system 500, which is a functional module in the data processing device 7.
  • a shopping user judgment system 500 which is a functional module in the data processing device 7.
  • the shopping user determination system 500 includes an item information storage unit 501, a shelf coordinate storage unit 502, a shelf plate and user matching determination unit 503, and a product and user matching determination unit 504.
  • the goods database generating unit 301 and the goods database updating unit 304 in the goods sensing system 300 based on weight monitoring are stored in the goods information storage unit 501, and the goods database includes each goods information; the goods The information includes the name, model, net content, and unit price of each item, and also includes the shelf number where the item is placed, the shelf number where the item is placed, and the item number.
  • the object positioning system 200 establishes a three-dimensional coordinate system in the closed space. Since the positions of the shelves 2 and the shelf plates 5 are determined, the coordinates of each shelf 2 and each shelf plate 5 can be obtained after the coordinate system is established.
  • the shelf board coordinate set is stored in the shelf board coordinate storage unit 502, and the height of the shelf board space (for example, 30 cm) above the shelf board for placing goods can be set to obtain the coordinate set of the shelf board space.
  • the target coordinate acquisition unit 202 can acquire the real-time coordinate set of each user's hand.
  • the shelf and user matching determination unit 503 Judging that the shelf board 5 matches the user, it can be considered that the user extends his hand into the shelf board space above the shelf board 5, and the user's behavior may be to take the goods or put them back.
  • the goods sensing system 300 based on weight monitoring monitors the real-time weight sensing value of each shelf 5 and the goods sensing system 400 based on image monitoring monitors the real-time images of the area in front of each shelf.
  • the two goods sensing systems work together to determine whether there is The goods are taken or put back, and the type and quantity of the goods taken or put back are judged.
  • the product-user matching determination unit 504 determines that the product matches the user, and the product At this moment, the user is removed from the shelf board or placed on the shelf board to determine the identity of the user.
  • this embodiment further includes a shopping information recording unit 600, which is a functional module in the data processing device 7, and generates at least one shopping database according to the identity information of each user to record that each user takes away at least The type and quantity of a product.
  • the shopping information recording unit 600 includes a shopping database generating unit 601 and a shopping database updating unit 602.
  • the identity obtaining unit 1022 obtains the identity information of the user, and the shopping database generating unit 601 generates the user's shopping database in the data processing device 7 according to the identity information of the user. There is no shopping information in the shopping database under the status.
  • any item matches a user it means that the user has made a shopping behavior, taking the item from a shelf or placing the item on a shelf.
  • the goods sensing system 300 based on weight monitoring detects that the real-time weight sensing value of a certain shelf board 5 is reduced, it means that there is a product removed from the shelf board, and the goods category determination unit 3034 stores it according to the goods database.
  • the product information and the shelf number of the weight difference less than 0 make a preliminary judgment of the possible types and quantities of the removed products. If in addition to the original product, other types of goods are misplaced on the shelf, the user cannot determine the specific type and quantity of the removed goods using only the weight difference, so only the initial difference can be determined based on the weight difference. Possible types and quantities of goods.
  • the weight difference of the shelf board is -100 grams, and the weight of the original product P on the shelf board is 50 grams, but the user has misplaced the other product Q on the shelf board and its weight is 100 grams, then this The result of the preliminary judgment is that the goods taken away are two goods P or one goods Q.
  • the image sensing-based product sensing system 400 monitors the real-time image of the user taking the product from the shelf and judges the type of the removed product again. If the judgment result is one of the preliminary judgment results If they match, you can confirm the type and quantity of the removed goods.
  • the shopping database update unit 602 generates the pickup information, including the types and quantities of the removed goods, as well as the information of the goods. The shopping information is written into the shopping database of the user, so that the types and quantities of the goods in the shopping database are consistent with the types and quantities of the goods actually purchased by the user.
  • the shopping database update unit 602 generates a set of shopping information according to the type and quantity of the removed goods and the identity information of the user who removed the goods, and stores the set of shopping information in the user's shopping database.
  • the shopping information includes the type of the goods removed at the moment. And quantity, as well as the product information of the product, such as product name, model, net content and unit price, etc. After the user has taken the goods multiple times in the closed space 1, the shopping database includes multiple sets of shopping information. Since the mobile communication terminal carried by the user and the data processing device 7 are connected by wireless communication and exchange data, the shopping database The shopping information can also be displayed on the APP interface of the user's mobile communication terminal.
  • the real-time weight sensing value of a certain shelf board 5 is increased by the goods sensing system 300 based on weight monitoring, the weight difference of the shelf board is greater than 0, indicating that an item is placed on the shelf On the board, you can determine whether the item is purchased.
  • each shopping information in the user's shopping database to determine whether the weight value of the purchased product matches the weight difference of the shelf board, that is, determine whether there is one or more of the total weight of the purchased product and the The weight difference between the shelves is the same. If so, you can make a preliminary judgment on the possible types and quantities of the item. For example, if the weight difference of the shelf board is 200 grams, and there are two 100 grams of goods A and four 50 grams of goods B among the purchased goods, it can be initially determined that there are 2 items returned to the shelves. Good A, or 1 good A and 2 good B, or 4 good B.
  • the product sensing system 400 based on image monitoring monitors the user's real-time image of returning the product to the shelf and judges the type of the returned product again. If the judgment result matches the preliminary judgment result, The type and quantity of the returned product can be confirmed.
  • the shopping database update unit 602 generates return information, including the type and quantity of the returned product, and the product information of the product, and deletes the information from the user's shopping database.
  • the shopping information corresponding to the return information makes the types and quantities of the goods in the shopping database consistent with the types and quantities of the goods purchased by the user.
  • the goods sensing system 300 based on weight monitoring and the goods sensing system 400 based on image monitoring can also determine the type and quantity of the returned goods. Further, the goods sensing system can also determine the quality of the returned goods. Whether the type is consistent with the original type of goods on the shelf with the increase in real-time weight sensing value. If they are not the same, an alarm signal can optionally be generated to remind the manager or user. At the same time, the goods sensing system records the number of the shelf board, and records the type and weight information of the misplaced goods.
  • the goods sensing system 300 based on weight monitoring detects that the weight sensing value of the shelf board decreases, According to the weight difference, the type and weight information of the misplaced goods, and the type and weight information of the original goods on the shelf, a preliminary determination of the possible types and quantities of the removed goods is made. The real-time image is judged again to confirm the type and quantity of the removed goods.
  • the judgment result of the goods perception system 400 based on image monitoring does not match the judgment result of the goods perception system 300 based on weight monitoring, or it is impossible to judge the type of goods returned, it can be confirmed that the item returned to the shelf is not the unmanned supermarket
  • the existing goods may be the items brought by the user, such as umbrellas, mobile phones, etc.
  • an alarm signal can be optionally generated to remind the management personnel or the user.
  • the shelf plate number of the shelf can be displayed. On a display for management or users to deal with in a timely manner.
  • this embodiment further includes a settlement system 700, which is a functional module in the data processing device 7 and is configured to settle charges based on the types and quantities of all the goods in the user's shopping database.
  • a settlement system 700 is a functional module in the data processing device 7 and is configured to settle charges based on the types and quantities of all the goods in the user's shopping database. After the user finishes the shopping process, he / she can leave the closed space 1 from the access control device at the entrance and exit by himself.
  • the image sensor 2011 of the target positioning system 200 cannot obtain the real-time three-dimensional image of the user, it can be determined that the user has finished shopping, and the settlement system 700 settles the fee for the user.
  • the settlement system 700 includes a total amount calculation unit 701 and a payment unit 702.
  • the total amount calculation unit 701 calculates the total amount according to the types and quantities of all the goods in the shopping database of the user. Since the unit price of each type of goods is pre-existed as data information in the data processing device 7 Therefore, the sum of the product of the unit price and quantity of multiple goods is the total amount that the user needs to pay. Further, in other embodiments, users can enjoy discounts on goods or use coupons, vouchers, etc.
  • the total amount that the user needs to pay is the sum of the product of the unit price and quantity of multiple goods minus the coupon and And / or the amount of the voucher and / or the amount of the discount.
  • the payment unit 702 is the payment software or third-party payment software that comes with the settlement system 700. The payment unit 702 can be deducted from the user's bank account or electronic account, and the amount of the deduction is the same as the total amount that the user needs to pay.
  • This embodiment also provides a target positioning method, that is, a method for implementing the foregoing target positioning system. As shown in FIG. 13, the method includes the following steps S301) to S303).
  • the technical effect is to use multiple image sensors to monitor the entire process.
  • the user's moving image in the closed space real-time positioning of the target (user) 's position in the closed space, tracking the user's trajectory in the closed space, and obtaining the user's body and a part of its extension (such as the head, hands, etc.) Set of three-dimensional coordinates.
  • Step S301 A space setting step, in which one or more entrances and exits are set in the closed space, and the user must enter and exit the unmanned supermarket through the entrances and exits.
  • Step S302) A three-dimensional image acquisition step is used to acquire at least one frame of three-dimensional images in real time, and the three-dimensional images include images of all or part of at least one target.
  • the three-dimensional image acquisition step includes the following steps: step S3021) an image sensor setting step, a plurality of image sensors are set on the top of the closed space, the lens of the image sensor faces downward, and a plurality of image sensors The field of view covers the entire bottom surface of the enclosed space.
  • Each image sensor 2011 includes a depth image sensor 2012 and an RGB image sensor 2013 arranged in parallel.
  • Step S3022) The original image acquisition step is used to acquire at least one frame depth image and at least one frame RGB image simultaneously in real time.
  • Step S3023) The three-dimensional image integration step combines the depth image and RGB image collected by the same image sensor at the same time into one frame of three-dimensional image; repeats step S3022) the original image acquisition step and step S3023) the three-dimensional image integration step to continuously integrate multiple frames Three-dimensional image.
  • the image sensor 2011 can acquire multiple frames of three-dimensional images of the user as a whole and the user's movement trajectory in a closed space.
  • the user When the identity of a user is identified, the user is located at the gate of the entrance and exit of the closed space, and an image sensor facing the entrance and exit starts to collect real-time three-dimensional images near the gate, including the entire three-dimensional image of the user.
  • an image sensor facing the entrance and exit starts to collect real-time three-dimensional images near the gate, including the entire three-dimensional image of the user.
  • the user travels or stays in the closed space, no matter where the user goes, there will be at least one lens of an image sensor located on the top of the closed space that can point at the user. The entire monitoring process continues until the user moves from a certain location.
  • the gates at the entrance and exit leave the enclosed space.
  • Multiple image sensors simultaneously capture real-time 3D images, and each real-time 3D image includes the 3D image of the user.
  • Step S303) A target coordinate obtaining step.
  • the target is a certain user.
  • a three-dimensional coordinate system is established in the closed space, and a user's coordinate set or coordinates are obtained in real time based on the at least one frame of three-dimensional images.
  • any point in the closed space can be used as the origin of the coordinate system.
  • a three-dimensional coordinate system can be established in the closed space, and a user's coordinate set or coordinate set can be obtained in real time according to the at least one frame of three-dimensional images.
  • the step S303) target coordinate acquisition step includes a step S3031) coordinate system establishment step, step S3032) parameter acquisition step, step S3033) background removal step, and step S3034) target coordinate calculation step.
  • Step S303) can be used to obtain a three-dimensional coordinate set of the user's body and a part of its extension (such as the head, hands, etc.).
  • Step S3031) A coordinate system establishing step is used to establish a three-dimensional coordinate system in the closed space; preferably, the center point of the bottom surface of the closed space (the ground of the unmanned supermarket) is selected as the origin of the coordinate system.
  • the center point of the bottom surface of the closed space (the ground of the unmanned supermarket) is selected as the origin of the coordinate system.
  • Step S3032 A parameter obtaining step of obtaining a position parameter and a color parameter of each pixel point of each frame of the three-dimensional image;
  • the position parameters are x, y, and z, and represent position coordinates of the pixel point in the three-dimensional coordinate system.
  • the color parameters are r, g, and b, which respectively represent the three primary color intensities of the pixel.
  • step S3032 the position parameters of each pixel point of a frame of three-dimensional image are obtained, and specifically include the following steps S30321) to S30323).
  • Step S30321) The sensor coordinate acquisition step is used to acquire the center point of the image sensor (the midpoint of the line connecting the center points of the depth sensor and the RGB sensor lens in parallel) that captures the three-dimensional image of the frame under the three-dimensional coordinate system. coordinate of.
  • Step S30322) a relative coordinate acquisition step for establishing a second three-dimensional coordinate system using the center point of the image sensor as a second origin, and acquiring the coordinates of each pixel point in the second three-dimensional coordinate system from the three-dimensional image 0.
  • Step S30323 A coordinate correction step for calculating and correcting the coordinates of the center point of the image sensor in the three-dimensional coordinate system and the coordinates of each pixel point in the three-dimensional image in the second three-dimensional coordinate system.
  • the coordinates of each pixel point of the three-dimensional image in the three-dimensional coordinate system so as to obtain a position parameter of each pixel point.
  • Step S3033) The background removal step.
  • N pixels are background pixels
  • N of the background pixels are removed from the M frames of the three-dimensional image to obtain M frames of backgroundless three-dimensional image, which is an image of a user.
  • the position of the pixels can be regarded as the background, so that the The pixel is removed from the corresponding three-dimensional image.
  • the remaining set of pixels can represent the overall trajectory of the user.
  • Step S3034) The step of calculating the coordinates of the target.
  • the set of position parameters of all pixels in the M-frame backgroundless three-dimensional image is the overall coordinate set of the user; in this coordinate set, the position parameters of the pixel with the largest position parameter z Defined as the coordinates of the target. If the target is further defined as the user's hand, a real-time coordinate set of the user's hand can be obtained.
  • the target positioning method may further include the following steps: step S401) shelf setting step, step S402) user identification step, step S403) goods perception step, step S404) shopping user judgment step, and step S405 ) Shopping information recording steps.
  • Step S401) shelf setting step, step S402) user identification step occurs after step S301) space setting step, and occurs before step S302) three-dimensional image acquisition step.
  • Step S401) The shelf setting step sets at least one shelf in the enclosed space; each shelf includes at least one shelf board, and at least one kind of goods are placed on each shelf board.
  • Step S402) The user identity recognition step identifies the identity information of each user entering the enclosed space.
  • Step S403) The goods sensing step is used to sense the pick-and-place status of each product in real time. When any one of the goods is removed or returned, the type and quantity of the removed or returned goods are obtained. Step S403) the goods sensing step and step S302) the three-dimensional image acquisition step and step S303) the target coordinate acquisition step are performed separately without interference. Both the real-time goods pick-up state and the real-time coordinates of the target are sent to the data processing device 7 for the next shopping user judgment.
  • Step S404 The judgment step of the shopping user is used to judge the identity of the user who has taken or returned the goods according to the identity information of the user and the real-time location of the user when any of the goods has been taken or put back; step S405
  • the shopping information recording step generates at least one shopping database according to the identity information of each user, and is used to record the type and quantity of each item taken by each user.
  • step S404) shopping user judgment step includes step S4041) goods information storage step, step S4042) shelf board coordinate storage step, step S4043) shelf board and user matching judgment step, and step S4044) goods and user matching judgment step.
  • Step S4041) a step of storing information of the goods, for storing a database of the goods, including each item of information; a step of storing the coordinates of the shelves in a step S4042), establishing a three-dimensional coordinate system in the closed space, for storing the set of shelves coordinates and the coordinates of the shelves Set the height of the shelf board space above the shelf board, and obtain the coordinate set of the shelf board space; step S4043) judging step of matching the shelf board with the user, when the coordinate set of the shelf board space above a shelf board and a user's hand When there is an intersection of the coordinate sets, it is determined that the shelf board matches the user; step S4044) The step of determining the matching of the goods with the user, when there are goods removed from a shelf board or placed on a shelf board, and at the same time The next user matches the shelf, and it is determined that the product matches the user.
  • the coordinate set of the shelf space for placing goods above each shelf plate can be determined, and each shelf plate number corresponds to a coordinate set of the shelf space.
  • each shelf plate number corresponds to a coordinate set of the shelf space.
  • step S405) shopping information recording step includes step S4051) shopping database generation step and step S4052) shopping database update step.
  • Step S4051) A shopping database generating step is used to generate a user's shopping database based on the user's identity information when the user's identity is identified;
  • step S4052) a shopping database updating step, when the goods are removed, according to the Take out the type and quantity of the goods and the identity information of the user who took the goods to generate shopping information and store it in the user's shopping database; when the goods are returned, according to the types and quantities of the goods and the goods
  • the return information is generated from the identity information of the user, and the shopping information corresponding to the return information is deleted from the shopping database of the user.
  • Steps S301) to S303) can obtain the overall and local real-time location of the user, and in combination with steps S401) to S405), when a pick-and-place event occurs on the goods on the shelf, the identity of the user who removes or returns the goods , And timely correct the shopping record in the user's shopping database.
  • the goods perception system finds that a product has been removed from a shelf by a user, and determines the type and quantity of the removed product, it can write the information, quantity, unit price and other information of the removed product.
  • the return information of the returned goods can be deleted from the user's shopping database.
  • the advantage of the present invention is to provide a target positioning system and a positioning method, which can obtain the real-time position of a specific target in a specific area, and in particular, can obtain the real-time position of each user in an unmanned supermarket, combined with each board Or the location of the product and the status of each item on the shelf, you can accurately determine the identity of the user who removed or returned the item on the shelf, so as to update the user ’s shopping record so that the user can automatically settle after the purchase is completed .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • Multimedia (AREA)
  • General Business, Economics & Management (AREA)
  • Economics (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Mathematical Physics (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geophysics (AREA)
  • General Life Sciences & Earth Sciences (AREA)
  • Computer Security & Cryptography (AREA)

Abstract

一种目标物定位系统（200）及定位方法，目标物定位系统（200）包括封闭空间（1）、三维影像采集装置（201）以及目标物坐标获取单元（203），目标物定位方法包括如下步骤：空间设置步骤（S301）、三维影像采集步骤（S302）以及目标物坐标获取步骤（S304），通过获取特定目标物在特定区域内实时位置，特别是可以获取每一用户在无人超市内的实时位置，结合每一架板（5）或货品的位置以及货架（2）上每一货品取放状态，可以准确判断出货架（2）上取走或放回的货品的用户的身份，从而更新该用户的购物记录，以便用户在购物完成后实现自动结算。

Description

目标物定位系统及定位方法 技术领域
本发明涉及一种用于零售业的用户实时位置定位追踪技术，具体地说，涉及一种目标物定位系统及定位方法。
背景技术
传统零售业的购物方式，每一家超市货便利店需要有专门的销售人员和收款人员，人力成本较高。随着电子支付技术、身份识别技术及云计算技术的发展，无人超市项目在技术上具备很高的可行性。在无人超市项目中，急需解决的一个基本问题就是用户选购货品的判断和记录问题，具体地说，服务器需要准确判断每一个用户选择了哪些货品、放回了哪些货品以及从超市中带走了哪些货品。
现有技术中，有人采用人脸技术识别用户身份，根据每一用户离开超市时，身上携带货品的RFID标签确定该用户已购买商品的种类和数量，这种方式需要在每一个货品上设置RFID标签，在门禁处设置RFID读写器。这种方案不需要定位跟踪用户的实时位置，其不足之处在于，首先，硬件成本较高，每一RFID标签的价格大约在0.5-1元左右，标签会提升每一种商品的成本，降低超市的竞争力，对于成本为5元的货品来说，加装RFID标签会提升其成本的10-20％；其次，货品感知存在被屏蔽、被去除的可能性，从而出现用户蒙蔽RFID阅读器的现象，导致货品丢失；再次，该方案中只有在超市门禁处才能实现结算，如果有用户在离店前将可食用的货品吃掉，将包装留在超市里，RFID阅读器也无法感知和确定用户的真实消费金额。也就是说，这一方案高度依靠市民用户的自律性和道德水平，而不是用技术手段加以约束，这样的无人超市营业过程中的风险较大。
本发明的申请人提出一种基于重量监测技术或影像监测技术实时判断货架上是否有货品被取走或被放回，同时，还需要解决取走或放回货品的用户身份判断问题。
发明内容
本发明的目的在于，提供一种目标物定位系统及定位方法，有效解决现有技术存在的无法实时定位及跟踪目标物、在无人超市中取走或放回货品的用户身份不明确的技术问题。
为实现上述目的，本发明提供一种目标物定位系统，包括：封闭空间；三维影像采集装置，用以实时采集至少一帧三维影像，所述三维影像中包括至少一目标物的全部或部分的影像；以及目标物坐标获取单元，设于一数据处理设备内，用以在所述封闭空间内建立三维坐标系，根据所述至少一帧三维影像实时获取所述目标物在所述三维坐标系的坐标集或坐标。
为实现上述目的，本发明提供一种目标物定位方法，包括如下步骤：空间设置步骤，用以设置一封闭空间；三维影像采集步骤，用以实时采集至少一帧三维影像，所述三维影像中包括至少一目标物的全部或部分的图像；以及目标物坐标获取步骤，用以在所述封闭空间内建立三维坐标系，根据所述至少一帧三维影像实时获取一目标物的坐标集或坐标。
本发明的优点在于，提供一种目标物定位系统及定位方法，可以获取特定目标物在特定区域内实时位置，特别是可以获取每一用户在无人超市内的实时位置，结合每一架板或货品的位置以及货架上每一的货品取放状态，可以准确判断出货架上取走或放回的货品的用户的身份，从而更新该用户的购物记录，以便用户在购物完成后实现自动结算。
附图说明
图1为本发明实施例所述的无人超市俯视图；
图2为本发明实施例所述的托盘及架板的结构示意图；
图3为本发明实施例所述货架的整体结构示意图；
图4为本发明实施例所述用户身份识别系统的结构框图；
图5为本发明实施例所述目标物定位系统的结构框图；
图6为本发明实施例所述影像传感器在封闭空间内的分布图；
图7为本发明实施例所述基于重量监测的货品感知系统的结构框图；
图8为本发明实施例所述基于影像监测的货品感知系统的结构框图；
图9为本发明实施例所述第二摄像头与所述货架的位置关系图；
图10为本发明实施例所述购物用户判断系统的结构框图；
[根据细则91更正 06.06.2019]　
图11为本发明实施例所述购物信息记录单元的结构框图；
图12为本发明实施例所述结算系统的结构框图。
图13为本发明实施例中目标物定位方法的流程图；
图14为本发明实施例中三维影像采集步骤的流程图；
图15为本发明实施例中目标物坐标获取步骤的流程图；
图16为本发明实施例中位置参数获取步骤的流程图；
图17为本发明实施例中另一种目标物定位方法的流程图；
图18为本发明实施例中购物用户判断步骤的流程图；
图19为本发明实施例中购物信息记录步骤的流程图。
图中各个部件标号如下：
1封闭空间，2货架，3支架，4托盘，5架板，6重量感应装置，7数据处理设备；
100用户身份识别系统，101门禁装置，102身份识别装置；1021扫码装置，1022身份获取单元，103用户入口，104用户出口；
[根据细则91更正 06.06.2019]　
200目标物定位系统，201三维影像采集装置，202目标物坐标获取单元；
[根据细则91更正 06.06.2019]　
2011影像传感器，2012深度图像传感器，2013RGB图像传感器，2014三维影像整合单元；
300基于重量监测的货品感知系统，301货品数据库生成单元，302重量值采集单元，303取放状态判断单元，304货品数据库更新单元；
3011初始化单元，3012信息录入单元，3013感应值初始化单元；
3031重量差值计算单元，3032重量差值判断单元，3033架板信息记录单元；3034货品种类判断单元，3035货品数量计算单元；
400基于影像监测的货品感知系统，401样本采集单元，402模型训练单元，403实时图片采集单元，404货品种类获取单元，405第一摄像头，406第二摄像头；
500购物用户判断系统，501货品信息存储单元，502架板坐标存储单元，503架板与用户匹配判断单元，504货品与用户匹配判断单元；
[根据细则91更正 06.06.2019]　
600购物信息记录单元，601购物数据库生成单元，602购物数据库更新单元；
700结算系统，701总金额计算单元，702支付单元。
具体实施方式
以下参考说明书附图完整介绍本发明的优选实施例，使其技术内容更加清楚和便于理解。本发明可以通过许多不同形式的实施例来得以体现，其保护范围并非仅限于文中提到的实施例。在附图中，结构相同的部件以相同数字标号表示，各处结构或功能相似的部件以相似数字标号表示。
当某些部件被描述为“在”另一部件“上”时，所述部件可以直接置于所述另一部件上；也可以存在一中间部件，所述部件置于所述中间部件上，且所述中间部件置于另一部件上。当一个部件被描述为“安装至”或“连接至”另一部件时，二者可以理解为直接“安装”或“连接”，或者一个部件通过一中间部件间接“安装至”或“连接至”另一个部件。
本实施例涉及一种目标物定位系统，是一种用户无人超市的无人售货系统中的一部分，通过在无人超市空间顶部设置多个影像采集装置，从而获取目标物(购物用户)在无人超市空间内的实时位置并实现有效跟踪。
如图1～3所示，所述无人售货系统包括一封闭空间1，其内设有多个货架2，每一货架2包括支架3及可拆卸式安装在支架3上的多个托盘4，多个托盘4在不同高度彼此平行或者在同一高度彼此平齐。每一托盘4上设有多个并列设置的架板5，每一架板5上放置有至少一种货品。本实施例架板5上放置的货品需要便于用户取走或放回，因此，以架板5朝向用户的一端作为架板5的前端。
每一架板5与托盘4之间都设置有一个重量感应装置6，优选长方体形状重量传感器，其一端的下表面连接至托盘4，其另一端的上表面连接至架板5。本实施例中，每一架板5皆为一个敞口的盒体，可以被放置有一种或多种货品，所述货品为标准货品，同一种类的货品的外观和重量都相同或近似。放置于同一架板5的同种类货品具有相同的重量值，不同种类货品具有不同的重量值，每一重量值仅对应一类货品。重量感应装置6可以精准的获取架板5及其上表面货品的实时重量感应值，精确感知每个架板5的每一次重量值变化量，包括增加量或减小量。
本实施例还包括数据处理设备7，如服务器或计算机等，数据处理设备7内部设有多个数据处理软件，具有多个功能模块，可以通过数据线连接至多个硬件，以软硬件结合方式实现多种功能。
如图1、图4所示，本实施例还包括用户身份识别系统100，用以识别每一用户的身份信息。用户身份识别系统100包括门禁装置101及身份识别装置102，本实施例所述封闭空间1不是绝对密封空间，而是相对封闭的空间，封闭空间1设有出入口，优选一个用户入口103及一个用户出口104，所有用户由用户入口103进入此封闭空间1，由用户出口104离开此封闭空间1。
如图1、图4所示，封闭空间1的每一出入口处都设置有门禁装置101，优选一自动闸机。身份识别装置102用以获取用户的身份信息，包括连接至数据处理设备7的扫码装置1021以及数据处理设备7内的身份获取单元1022。扫码装置1021设置于用户入口103处的门禁装置101的内部或外部，优选设于自动闸机的外表面，用以扫描身份识别码，优选一二维码；身份获取单元1022为数据处理设备7中的一个功能模块，可以根据所述身份识别码获取用户的身份信息。用户出口104处的门禁装置101无需设置身份识别装置102。
本实施例中，每个用户将与无人超市配套使用的专用应用软件(APP)下载到移动通信终端(手机、平板电脑等)中，在应用软件(APP)中注册账号，关联至支付软件；或者，每个用户将支付软件(如微信/支付宝)下载到移动通信终端中，在支付软件中嵌入与无人超市配套使用的小程序，在支付软件中注册账号，专用应用软件(APP)或支付软件内设有用户注册信息及电子支付信息，包括用户身份信息、银行账号信息、支付密码等。注册完成后，用户身份信息会存储于数据处理设备7的用户数据库中。
移动通信终端中的应用软件(APP)可以生成一二维码，该二维码内存储有用户的身份信息等，当某一用户需要从外部进入封闭空间1时，将应用软件生成的二维码正对扫码装置1021的扫描端，扫码后扫码装置1021对此二维码进行解码处理，将解码结果传送至数据处理设备7，如果二维码是可识别的，且识别出的身份信息与预先存储在用户数据库的身份信息相匹配，确定该用户身份合法，门禁装置101打开允许该用户进入封闭空间1，用户入口103处的门禁装置101设有感应装置，如红外传感器等，当该用户进入封闭空间1之后， 门禁装置101感知到有人走过门禁，然后自动关闭。当用户购物结束需要离开封闭空间1时，用户出口104处的门禁装置101感应到有人从封闭空间1内部靠近门禁装置101时，该门禁装置会自动打开，待用户离开封闭空间1后，门禁装置101感知到有人走过门禁，然后自动关闭。
身份获取单元1022根据所述身份识别码获取用户的身份信息后，数据处理设备7可以生成该用户的购物数据库，在用户购物过程中根据用户每一次购物行为获取购物信息更新该购物数据库。由于用户随身携带的移动通信终端通过应用软件(APP)与数据处理设备7进行实时数据交换，用户的购物数据库也可以显示在移动通信终端中的应用软件(APP)中，形成购物车界面，以便用户了解自己的购物记录及后续结算。
[根据细则91更正 06.06.2019]　
如图4、图5所示，本实施例所述目标物定位系统200，包括封闭空间1、三维影像采集装置201、三维影像获取单元及目标物坐标获取单元202，用以获取每一目标物在封闭空间1的实时位置。本实施例所述目标物为用户及其延伸部的全部或局部，目标物定位系统200即为用户定位系统，用以获取用户整体或局部(如头部、手部等)的位置，也即在所述三维坐标系下坐标集。
三维影像采集装置201包括至少一影像传感器2011，用以实时采集至少一帧三维影像。多个影像传感器2011平均分布于封闭空间1顶部，其镜头朝向下方，镜头中轴线可以与水平面垂直也可以有一定的倾角。影像传感器2011的镜头的视野范围覆盖封闭空间1的全部底面。用户在无人超市内行动或购物时，始终处于影像传感器的监视之下，此时影像传感器采集到的三维影像中包括用户影像，所述用户影像是指该用户身体及其延伸部的全部或局部的图片。如果该封闭空间内空无一人，每一时刻下的三维影像都与前一时刻下相同，可以判断该时刻下的三维影像都为背景，不包括任何用户影像。
[根据细则91更正 06.06.2019]　
每一影像传感器2011包括并列设置的深度图像传感器2012及RGB图像传感器2013以及三维影像整合单元2014，深度图像传感器2012连续采集多帧深度图像；RGB图像传感器2013连续采集多帧RGB图像，三维影像整合单元2014将同一影像传感器2011同一时刻采集到的一帧深度图像及一帧RGB图像结合为一帧三维影像。
上述两个传感器同步采集(同时采集且采集频率相同)，影像传感器2011每秒可以获取同样帧数的RGB图像及深度图像，三维影像整合单元2014每秒可以连续获得多帧三维影像并传送至数据处理设备7的目标物坐标获取单元202。
目标物坐标获取单元202为数据处理设备7中的一个功能模块，在所述封闭空间内建立三维坐标系，根据连续多帧包括用户影像的三维影像实时获取所述用户在所述三维坐标系下的坐标集或坐标。目标物坐标获取单元202包括坐标系建立单元2021、参数获取单元2022、背景去除单元2023以及目标物坐标计算单元2024。
坐标系建立单元2021在所述封闭空间内建立三维坐标系，优选地，选择封闭空间底面(无人超市的地面)的中心点作为坐标系原点，在水平方向设置X轴、Y轴，在竖直方向设置Z轴。
由于用户身体的全部或部分在三维坐标系下占用较大的空间，因此可以用一个坐标集代表用户的位置，如果考虑到位置精确控制和计算方便，也可以用该坐标集中某一个特定的点的坐标代表用户的位置，例如可以采用该用户坐标集中最高的一个点(Z轴数值最大的点)的坐标来表示用户位置。
参数获取单元2022对连续多帧包括用户影像的三维影像进行处理，获取每一帧三维影像的每一像素点的位置参数和色彩参数；所述位置参数为x、y、z，代表该像素点在所述三维坐标系下的位置坐标；所述色彩参数为r、g、b，分别代表该像素点的三原色强度。当某一用户行进至任一影像传感器的视野内之后，数据处理设备7可以获取多帧三维影像，每一帧三维影像中都包括用户影像和背景影像，每一像素点可能是用户的一部分也可以是背景的一部分。
在不同的影像传感器采集到三维影像中，表示用户身体及其延伸部的同样位置的像素点，其色彩参数r、g、b都是相同的。由于不同位置的影像传感器与用户的距离不同，每一影像传感器直接采集的初级位置参数都是用户身体及其延伸部上的一点相对于该影像传感器的位置坐标，因此要进行坐标变换，将不同位置的影像传感器采集的初级位置参数都转换为在所述封闭空间内建立的三维坐标系下的位置参数。
参数获取单元2022包括传感器坐标获取单元20221、相对坐标获取单元20222以及坐标修正单元20223，传感器坐标获取单元20221获取采集该帧三维影像的影像传感器的中心点(即并列设置的深度图像传感器2012及RGB图像传感器2013的镜头中心点连线的中分点)在所述封闭空间内建立的所述三维坐标系下的坐标；相对坐标获取单元20222根据所述影像传感器的中心点为第二原点建立第二三维坐标系，其X轴、Y轴及Z轴的方向与所述三维坐标系相同，从所述三维影像获取每一像素点在所述第二三维坐标系下的坐标；坐标修正单元20223用以根据所述影像传感器中心点在所述三维坐标系下的坐标及所述三维影像中每一像素点在第二三维坐标系下的坐标，计算并修正所述三维影像的每一像素点在所述三维坐标系下的坐标，从而获得用户及其延伸部的每一像素点的位置参数。
在连续采集的M帧三维影像中，每一帧三维影像包括且仅包括一个用户的影像，若分属于不同三维影像、位置参数相同的N个像素点的色彩参数相同时，且N大于0.9*M且小于或等于M，背景去除单元2023判定该N个像素点为背景像素点，从所述M帧三维影像中去除N个所述背景像素点，获得M帧无背景三维影像，即为该用户的影像。在连续获取的三维影像中，如果分属于不同三维影像的、位置参数相同的像素点的色彩参数相同，或者大部分(如90％)相同，即可认定像素点的位置为背景，从而可以将该像素点从相应的三维影像中去除。
在目标物坐标计算单元2024中，若目标物为用户及其延伸部全部，所述M帧无背景三维影像中所有像素点的位置参数的集合即为所述用户及其延伸部的全部的坐标集；在所述坐标集中，参数z最大的像素点的位置参数被定义为用户的坐标。在连续获取的三维影像中，去除背景像素点后，剩下的像素点即可代表该用户整体的行进轨迹。若连续采集的M帧三维影像中，每一帧三维影像包括多个用户的影像，需要先在每一M帧三维影像截取只含一个用户全部或部分的三维影像。
若目标物为用户及其延伸部的局部，可以获取所述用户局部的坐标集，如头部、肩部、肘部、腕部、手部等。深度图像传感器2012及RGB图像传感器2013分别设有一个镜头，深度图像传感器2012的镜头与RGB图像传感器2013的镜头并列设置且彼此相邻，如果将两个镜头的中轴线垂直水平面设置，两个镜头就会俯视封闭空间内的货品及用户。正常情况下， 两个镜头可以捕捉到用户头部及肩部的位置坐标集，当用户伸出手时，也可以捕捉到用户臂部、肘部、腕部、手部的位置坐标集。如果将某一时刻下该用户的头部、肩部、肘部、腕部、手部都连成一条折线或曲线，即可将用户的手部与头部位置建立对应关系，也就是说，可以实时获取到某一手部的位置，同时可以判断出该手部属于哪一用户。
进一步地，影像传感器2011的视野范围也可以覆盖出入口外部的部分空间，当用户在出入口外部时，该用户的影像就可以被影像传感器2011获取到。用户使用所述无人售货系统的全部过程，包括出入口处身份识别过程、进入封闭空间1的过程、在封闭空间1内行走或驻留过程、离开封闭空间1过程，全部处于影像传感器2011的监控下，可以实时监控已知身份的某一用户及其身体的一部分在封闭空间1内的实时位置。扫码装置1021读取用户的二维码时，数据处理设备7即可获取其身份信息，影像传感器2011从扫码装置1021读码时开始定位及实时跟踪该用户位置，监控该用户是否与某一货架匹配。当影像传感器2011无法获取该用户的实时三维影像时，可以认定该用户购物结束，从而对其进行结算。
本实施例还包括货品感知系统，用以实时感知每一种货品的取放状态，当任一种类货品被取走或被放回时，获取被取走或被放回的货品的种类及数量。所述取放状态包括货品静置状态、被取走状态及被放回状态。本实施例中，所述货品感知系统包括两种不同的技术方案。
如图7所示，本实施例还包括一种基于重量监测的货品感知系统300，用以实时感知每一种货品的取放状态，所述取放状态包括货品静置状态、被取走状态及被放回状态。
基于重量监测的货品感知系统300包括货品数据库生成单元301、重量值采集单元302、取放状态判断单元303以及货品数据库更新单元304。上述四个单元为数据处理设备7中的功能模块，与设有重量感应装置6的货架2配合工作，可以监控每一架板5的实时重量感应值，判断是否有货品被取走或被放回。当任一种类货品被取走或被放回时，货品感知系统300获取被取走或被放回的货品的种类及数量。
货品数据库生成单元301用以生成一货品数据库；货品数据库包括每一货品的货品信息及用以放置货品的每一架板的重量感应值，所述货品信息包括货品的种类、单品重量值以及货品对应的架板编号与货架编号，还包括货品的编号货品名、型号、净含量及单价等。
货品数据库生成单元301包括初始化单元3011、信息录入单元3012以及感应值初始化单元3013。初始化单元3011用以对一货品数据库进行初始化处理，在数据处理设备7的存储器中建立货品数据库。信息录入单元3012用以录入每一货品的重量值及货品信息，并将其存储至所述货品数据库，利用键盘或扫码器将无人超市货架上的每一种货品的重量值都录入所述货品数据库。感应值初始化单元3013用以采集每一架板被放置货品后的重量感应值，并将其存储至所述货品数据库。
在无人超市布置过程中，优选地，在每一架板5上摆放种类相同、重量相同的多个货品之后，货品信息被录入至数据处理设备7，并存储于货品数据库中。以某品牌饮料为示例，某一架板上摆放有8瓶某品牌饮料，架板的重量为100克，每瓶饮料重量为200克，在初始化后货品数据库中该架板的感应重量值为1700克，该品牌饮料对应的产品名称(某凉茶)、净含量(195ml)、产地(广东)、单价(5元)、单品重量值(200克)、货架编号(1)、架板编号(1-12)、商品编号(025)等信息也被存储在所述货品数据库内。
重量值采集单元302通过数据线分别连接至每一架板5内的重量感应装置6，用以实时采集每一架板5的实时重量感应值，优选地，采集时间间隔为0.1-0.2秒。所述实时重量感应值为重量传感器的感应数值，在货品被摆放至架板5之前，代表每一架板的重量；在货品被摆放至架板5之后，代表该架板及该架板上货品的总重量值；当有货品被取走或被放回至架板5时，实时重量感应值会发生变化。
为了减小误差，在架板5上正式摆放货品之前，需要先进行校准处理，用多种不同重量的标准砝码计算重量传感器的感应值X与架板上方放置物品的实际重量值Y之间的对应关系。例如，先将架板空置，获取重量感应装置6(重量传感器)的感应值X0，此时架板上货品重量Y0＝0克；再用重量值为500克、1000克的标准砝码分别放置在架板上，获取重量感应装置6(重量传感器)的感应值X1、X2，对应的架板上货品重量Y1＝500克，Y2＝1000克。利用公式Y(货品重量值)＝k*X(传感器值)+b，计算并存储三组参数k、b的数值，选择其中偏差较小的参数组。在后续的实时监控过程中，重量感应装置6(重量传感器)每次实时采集到的传感器值，结合参数k、b的数值，即可以计算出每一架板上既有货品的总重量。
取放状态判断单元303用以判断每一架板的重量感应值是否发生变化，若变小，判定该架板上有货品被取走；若变大，判定有物品被放置于该架板上；若完全不变，说明该架板上的货品完全没有变化，重量值采集单元302重新进行实时采集。取放状态判断单元303包括重量差值计算单元3031、重量差值判断单元3032以及架板信息记录单元3033。
重量差值计算单元3031计算实时采集的每一架板的实时重量感应值与所述货品数据库内存储的同一架板的重量感应值的差值，记录为每一架板的重量差值。例如，前述示例中，若放置前述某品牌饮料的架板重量发生变化，变为1300克或1900克，分别记录重量差值为-400克或200克。
重量差值判断单元3032将至少一架板的重量差值与0对比；当一架板的重量差值小于0时，判定该架板上有货品被取走；当一架板的重量差值大于0时，判定该架板上有物品被放置，此时不能确定该物品是否为用户之前从货架上取走的货品，也可能是用户的随身物品。例如，前述示例中，重量差值为-400克，即可认定有货品被取走；重量差值为200克，即可以认定有物品被放置在货架上。
当一架板的重量差值大于0或小于0时，架板信息记录单元3033记录该架板的架板编号以及该架板的重量差值。例如，若前述示例中的重量差值-400克，已知该架板重量减少，记录该架板的编号(1-12)。若前述示例中的重量差值200克，已知该架板初始状态下的重量感应值为1700克，此时被放置在该架板上的物品必然不是原货架上的商品，因此很可能是原属于其他架板上的商品或者是用户的随身物品，此时可选择地生成一个报警信号，提醒管理人员或用户，必要时，可以将该架板的架板编号显示在某一显示器上，以便管理人员或用户及时处理。
取放状态判断单元303还包括货品种类判断单元3034及货品数量计算单元3035。当一架板的重量差值小于0时，货品种类判断单元3034根据该架板编号及所述货品数据库内存储的该架板对应的货品信息判断被取走货品的种类。例如，已知该架板的编号(1-12)，如果每一架板上只放置一种货品，可以判断该货品种类为某凉茶，也可以对应找出其他货品信息，如单品重量值(200克)、净含量(195ml)、产地(广东)、单价(5元)等。如果该架板被放置多种货品，只能根据重量差值初步判断出被取走货品可能的种类及数量。
当一架板的重量差值小于0时，如果每一架板上只放置一种货品，货品数量计算单元3035计算一架板的重量差值的绝对值与所述货品数据库内存储的该架板上单一货品重量值的比值，利用四舍五入法对该比值进行取整处理，获得的整数即为被取走货品的数量。例如，前述示例中的重量差值-400克，其绝对值为400克，与单品重量值(200克)的比值为2，因此该比值即为被取走货品的数量。由于同种类的多个货品之间也可能存在较小的重量差值，直接计算之后的比值不一定为整数，可能为趋近于某个整数，因此需要利用四舍五入法对该比值进行取整处理，从而可以判断出被取走货品的种类及数量。
在理想状态下，如果用户素质较高，每一次放回货品时都能将该货品正确放回至该货品原属货架。当一架板的重量差值大于0时，货品种类判断单元3034根据该架板编号及该架板对应的货品信息判断被放回货品的种类。货品数量计算单元3035计算该架板的重量差值的绝对值与该架板对应的货品的单品重量值的比值，利用四舍五入法对该比值进行取整处理，获得的整数即为被放回货品的数量。
货品数据库更新单元304用以存储所述实时重量感应值至所述货品数据库，形成新的重量感应值，以更新所述货品数据库中每一架板的重量感应值，以待下次调用和判断。
本实施例所述基于重量监测的货品感知系统300的有益效果在于，提供一种基于重量监测的货品感知方案，可以实时监测货架上货品的实时重量感应值，实时感知每一架板的重量变化，由货架上所有架板的重量变化推断出有哪一种货品被取走或被放回，还可以判断被取走或被放回货品的种类和数量。
如图8所示，本实施例还包括一种基于影像监测的货品感知系统系统400，其包括样本采集单元401、模型训练单元402、实时图片采集单元403以及货品种类获取单元404，上述四个单元为数据处理设备7中的功能模块，基于影像监测的货品感知系统400可以监控货架前方区域的实时影像，判断被取走或被放回的货品的种类。
基于影像监测的货品感知系统400还包括第一摄像头405及第二摄像头406，第一摄像头405连接至样本采集单元401，用以拍摄每一货品多个角度多个距离的图片。第二摄像头406连接至实时图片采集单元403，用以拍摄一货架前方空间的实时图片。
如图9所示，优选地，第二摄像头406的数量为两个或四个，设置于货架2的外部，每一第二摄像头406朝向货架2的一个角落处。货架2的多个架板5的最前端位于同一平面上，该平面称之为货架平面，第二摄像头406设有镜头，该镜头的视野范围覆盖所述货架前方空间；当货品被从所述货架上被取下或者被放置在货架上时，所述货品被取下过程或被放回过程的影像被所述第二摄像头拍摄到。所述货架前方空间是指货架前方对应货架平面的空间区域，所述货架前方空间一般是指货架前方30～50厘米宽度的区域范围，每一个第二摄像头406的镜头朝向所述货架前方空间的中心区域。
优选地，第二摄像头406的镜头的中轴线与水平面夹角为30-60度；和/或，第二摄像头406的镜头与货架2上端或下端的距离为0.8-1.2米；和/或，第二摄像头406的镜头与货架2一侧边的距离为0.8-1.2米，确保第二摄像头406的视野范围可以完全覆盖货架前方空间，当货品被从货架2上被取下或者被放置在货架2上时，取下过程或放置过程的影像被第二摄像头406拍摄到。
样本采集单元401用以采集至少一组图片样本，每一组图片样本包括一种货品在多角度下的多张样本图片；同一种类货品的一组图片样本被设有相同的组别标识，该组别标识代表该组图片样本对应的货品的种类。优选地，第一摄像头405对货架2上每种货品需要拍摄不同角度不同距离的3000～5000张图片，并传送至数据处理设备7的样本采集单元401。由于本实施例涉及的无人超市中所销售的货品为标准货品，同一种类货品的外观六面图都是相同或相近似的，因此在同一种类货品中只要选择一个或几个产品，对其进行多次拍照处理即可完成该类货品训练样本的采样。
模型训练单元402用以根据多组图片样本中的每一样本图片及每一样本图片的组别标识训练卷积神经网络(CNN)模型，获取货品识别模型。优选地，本实施例中的卷积神经网络模型是目前运算量最小、响应速度最快的Faster RCNN网络模型，该模型的最快响应速度只要0.2秒左右，可以在极短的时间准确识别出图片的物品的种类及数量。
实时图片采集单元403连接至多个第二摄像头406，用以连续采集货架前方空间的至少一实时图片，每一实时图片包括一个或多个货品图片的部分或全部。当有用户从货架的某一架板上取走货品时，或者，当有用户放置货品或物品至货架的某一架板时，第二摄像头 406可以会从不同角度拍摄到货品在货架前的整体或局部照片，显示该货品的形状、图案及色彩。
货品种类获取单元404用以根据所述实时图片及所述货品识别模型获取所述实时图片中显示的货品的种类。实时图片采集单元403在某一时段采集到的多张实时图片，在预处理后输入至所述货品识别模型，从而判断该时段内的图片对应的组别标识，根据组别标识判断该时段内拍到的货品的种类。
基于影像监控的货品感知系统400的有益效果在于，可以实时监测货架前方空间的影像，判断是否有货品被从货架上取走或被放回至货架，利用机器学习中的卷积算法推断出货品的种类及数量的可能性结论，并选择其中可信度最高的结果作为最后结论。
如图10所示，本实施例还包括购物用户判断系统500，其为数据处理设备7中的功能模块，当任一种类货品被取走或被放回时，根据所述用户的身份信息及所述用户的实时位置判断取走或放回货品的用户身份。购物用户判断系统500包括货品信息存储单元501、架板坐标存储单元502、架板与用户匹配判断单元503以及货品与用户匹配判断单元504。
基于重量监测的货品感知系统300中的货品数据库生成单元301及货品数据库更新单元304生成或更新的货品数据库皆存储于货品信息存储单元501中，所述货品数据库包括每一货品信息；所述货品信息包括每一货品的货品名、型号、净含量及单价等，还包括放置该货品的货架编号、放置该货品的架板编号及货品编号。
目标物定位系统200在所述封闭空间内建立三维坐标系，由于货架2及架板5的位置确定，因此建立坐标系后即可获取各个货架2及各个架板5的坐标，货架坐标集及架板坐标集存储于架板坐标存储单元502中，设置架板上方的用以放置货品的架板空间的高度(如30CM)，即可获取所述架板空间的坐标集。
[根据细则91更正 06.06.2019]　
目标物坐标获取单元202可以获取每一用户手部的实时坐标集，当一架板5上方的架板空间的坐标集与一用户手部坐标集有交集时，架板与用户匹配判断单元503判定该架板5与该用户匹配，可以认为该用户将手部伸入至该架板5上方的架板空间，用户的行为可能是取走货品或者放回货品。
基于重量监测的货品感知系统300监控每一架板5的实时重量感应值，同时基于影像监测的货品感知系统400监控每一货架前方区域的实时影像，两个货品感知系统配合工作，判断是否有货品被取走或被放回，并判断被取走或被放回的货品的种类及数量。当有货品从一架板上被取走或被放置到一架板上时，且同一时刻下有一用户与该架板匹配，货品与用户匹配判断单元504判定该货品与该用户匹配，该货品在这一时刻被该用户从该架板上取走或放置到该架板上，从而确定该用户的身份。
如图11所示，本实施例还包括购物信息记录单元600，其为数据处理设备7中的功能模块，根据每一用户的身份信息生成至少一购物数据库，用以记录每一用户取走至少一货品的种类及数量。购物信息记录单元600包括购物数据库生成单元601以及购物数据库更新单元602。
当一用户的身份被用户身份识别系统100识别时，身份获取单元1022获取用户的身份信息，购物数据库生成单元601根据所述用户的身份信息在数据处理设备7中生成该用户的购物数据库，初始状态下的购物数据库无任何购物信息。当任一货品与一用户匹配时，说明该用户发生一次购物行为，将货品从一架板上取走或将货品放置到一架板上。
如果此时基于重量监测的货品感知系统300监控到某一架板5的实时重量感应值减小，说明有货品从该架板上被取走，货品种类判断单元3034根据所述货品数据库内存储的货品信息及重量差值小于0的架板编号初步判断被取走货品可能的种类及数量。如果该架板上除了原有货品外，还被错放了其他种类货品，用户只用重量差值无法判断被取走货品具体的种类和数量，因此只能根据重量差值初步判断出被取走货品可能的种类及数量。例如，若该架板的重量差值为-100克，且该架板上原有货品P的重量为50克，但是被用户误放了其他架板上货品Q，其重量为100克，那么此时初步判断的结果是被取走的货品为两个货品P或者一个货品Q。
当货品与一用户匹配时，基于影像监测的货品感知系统400监控到用户从架板上将货品取走的实时影像，再次判断被取走的货品的种类，如果判断结果与初步判断结果之一相符，即可确认被取走的货品的种类及数量，购物数据库更新单元602生成取货信息，包括被取走货品的种类及数量，以及该货品的货品信息，并且将该取货信息相应的购物信息写入至 该用户的购物数据库中，使得购物数据库中的货品的种类及数量与用户实际购买的货品的种类及数量一致。
购物数据库更新单元602根据被取走货品的种类及数量以及取走货品的用户的身份信息生成一组购物信息，并存储至该用户的购物数据库，该购物信息中包括此刻被取走货品的种类及数量，以及该货品的货品信息，如货品名、型号、净含量及单价，等等。用户在封闭空间1内多次取走货品之后，其购物数据库内包括多组购物信息，由于用户随身携带的移动通信终端与数据处理设备7以无线通信方式连接并进行数据交换，因此，购物数据库中的购物信息也可以显示在用户的移动通信终端的APP界面上。
当货品与一用户匹配时，若基于重量监测的货品感知系统300监控到某一架板5的实时重量感应值增大，该架板的重量差值大于0，说明有物品被放置于该架板上，可以判断该物品是否为已购货品。
在该用户的购物数据库中查询每一购物信息，判断是否有已购货品的重量值与该架板的重量差值相匹配，也即判断是否有一个或多个已购货品的总重量与该架板的重量差值相同。如果是，可以初步判断该物品可能的种类及数量。例如，若该架板的重量差值为200克，且已购货品中有两个100克的货品A及四个50克的货品B，可以初步判断被放回该架板的物品为2个货品A，或者1个货品A及两个货品B，或者4个货品B。
当货品与一用户匹配时，基于影像监测的货品感知系统400监控到用户将货品放回至架板上的实时影像，再次判断被放回的货品的种类，如果判断结果与初步判断结果相符，即可确认被放回的货品的种类及数量，购物数据库更新单元602生成归还信息，包括被放回货品的种类及数量，以及该货品的货品信息，并且从该用户的购物数据库中删除与所述归还信息相应的购物信息，使得购物数据库中的货品的种类及数量与用户购买的货品的种类及数量一致。
同理，基于重量监测的货品感知系统300、基于影像监测的货品感知系统400还可以判断被放回的货品的种类及数量，进一步地，所述货品感知系统还可以判断被放回的货品的种类与实时重量感应值增大的架板上原有货品种类是否一致，如果不一致，可选择地生成一个报警信号，提醒管理人员或用户。同时，所述货品感知系统记录该架板的编号，并记录 被误放的货品的种类及重量信息，之后，基于重量监测的货品感知系统300若监测到该架板的重量感应值减小，根据重量差值、被误放的货品的种类及重量信息、该架板上原有货品的种类及重量信息，初步判断出被取走货品可能的种类及数量，基于影像监测的货品感知系统400利用实时影像再次判断，即可确认被取走的货品的种类及数量。
如果基于影像监测的货品感知系统400的判断结果与基于重量监测的货品感知系统300的判断结果不符，或者无法判断被放回货品种类，即可确认被放回架板的物品并非该无人超市中的既有货品，有可能是用户自带的物品，如雨伞、手机等，此时可选择地生成一个报警信号，提醒管理人员或用户，必要时，可以将该架板的架板编号显示在某一显示器上，以便管理人员或用户及时处理。
如图12所示，本实施例还包括结算系统700，其为数据处理设备7中的功能模块，用以根据所述用户的购物数据库中所有货品的种类及数量结算费用。用户购物过程结束后，可以自行从出入口的门禁装置处离开离开封闭空间1。当目标物定位系统200的影像传感器2011无法获取该用户的实时三维影像时，可以认定该用户购物结束，结算系统700为该用户结算费用。
结算系统700包括总金额计算单元701及支付单元702。当所述用户离开所述封闭空间时，总金额计算单元701根据所述用户的购物数据库中全部货品的种类及数量计算总金额，由于每一种类货品的单价作为货品信息预存在数据处理设备7中，因此多种货品单价与数量的乘积的总和的金额即为该用户需要支付的总金额。进一步地，在其他实施例中，用户可以享受到货品折扣或使用优惠券、抵用券等，用户需要支付的总金额为多种货品单价与数量的乘积的总和的金额内减去优惠券和/或抵用券金额和/或折扣金额。支付单元702为结算系统700自带的支付软件或第三方支付软件，可以从所述用户的银行账户或电子账户上扣款，扣除的款项金额与该用户需要支付的总金额相同。
本实施例还提供一种目标物定位方法，也就是前述目标物定位系统的实现方法，如图13所示，包括如下步骤S301)～S303)，其技术效果在于，利用多个影像传感器全程监测用户在封闭空间内的活动影像，实时定位目标物(用户)的在封闭空间的位置，跟踪用户 在封闭空间的行动轨迹，获取用户身体及其延伸部的一部分(如头部、手部等)的三维坐标集。
步骤S301)空间设置步骤，在该封闭空间设置一个或过个出入口，用户必须由此出入口进出该无人超市。
[根据细则91更正 06.06.2019]　
步骤S302)三维影像采集步骤，用以实时采集至少一帧三维影像，所述三维影像中包括至少一目标物全部或部分的图像。如图14所示，步骤S302)三维影像采集步骤包括如下步骤：步骤S3021)影像传感器设置步骤，在所述封闭空间顶部设置多个影像传感器，所述影像传感器的镜头朝向下方，多个影像传感器的视野范围覆盖所述封闭空间的全部底面。每一影像传感器2011包括并列设置的深度图像传感器2012及RGB图像传感器2013。步骤S3022)原始图像采集步骤，用以实时同步采集至少一帧深度图像及至少一帧RGB图像。步骤S3023)三维影像整合步骤，将同一影像传感器同一时刻采集到的深度图像及RGB图像结合为一帧三维影像；重复步骤S3022)原始图像采集步骤及步骤S3023)三维影像整合步骤，连续整合多帧三维影像。在用户在无人超市的购物过程中，影像传感器2011可以获取该用户整体的多帧三维影像及该用户在封闭空间的运动轨迹。当一用户的身份被识别时，该用户是位于封闭空间出入口的闸机处，正对所述出入口的一影像传感器开始采集闸机附近的实时三维影像，包括该用户整体的三维影像。当用户在所述封闭空间内行进或驻留时，无论用户走到哪一个位置，都会有至少一个位于封闭空间顶部的影像传感器的镜头能对着该用户，整个监控过程持续到用户从某一出入口的闸机处离开封闭空间为止。多个影像传感器同时采集实时三维影像，每个实时三维影像中都包括该用户的三维影像。
步骤S303)目标物坐标获取步骤，所述目标物为某一用户，在所述封闭空间内建立三维坐标系，根据所述至少一帧三维影像实时获取一用户的坐标集或坐标。建立三维坐标系时，理论上可以以封闭空间的任意一点做为坐标系原点，封闭空间内建立三维坐标系，根据所述至少一帧三维影像实时获取一用户的坐标集或坐标集。
如图15所示，步骤S303)目标物坐标获取步骤包括步骤S3031)坐标系建立步骤、步骤S3032)参数获取步骤、步骤S3033)背景去除步骤以及步骤S3034)目标物坐标计算 步骤。步骤S303)可用以获取用户身体及其延伸部的一部分(如头部、手部等)的三维坐标集。
步骤S3031)坐标系建立步骤，用以在所述封闭空间内建立三维坐标系；优选地，选择封闭空间底面(无人超市的地面)的中心点点作为坐标系原点。在水平方向设置X轴、Y轴，在竖直方向设置Z轴。
步骤S3032)参数获取步骤，获取每一帧三维影像的每一像素点的位置参数和色彩参数；所述位置参数为x、y、z，代表该像素点在所述三维坐标系下的位置坐标；所述色彩参数为r、g、b，分别代表该像素点的三原色强度。当某一用户行进至任一影像传感器的视野内之后，数据处理设备可以获取多帧三维影像，每一帧三维影像中都包括用户影像和背景影像，每一像素点可能是用户的一部分也可以是背景的一部分。
如图16所示，在步骤S3032)参数获取步骤中，获取一帧三维影像的每一像素点的位置参数，具体包括如下步骤步骤S30321)～S30323)。步骤S30321)传感器坐标获取步骤，用以获取采集该帧三维影像的影像传感器的中心点(并列设置的深度传感器及RGB传感器的镜头中心点的连线的中分点)在所述三维坐标系下的坐标。步骤S30322)相对坐标获取步骤，用以以所述影像传感器的中心点为第二原点建立第二三维坐标系，从所述三维影像获取每一像素点在所述第二三维坐标系下的坐标0。步骤S30323)坐标修正步骤，用以根据所述影像传感器中心点在所述三维坐标系下的坐标及所述三维影像中每一像素点在第二三维坐标系下的坐标，计算并修正所述三维影像的每一像素点在所述三维坐标系下的坐标，从而获得每一像素点的位置参数。
步骤S3033)背景去除步骤，在连续采集的M帧三维影像中，若分属于不同三维影像、位置参数相同的N个像素点的色彩参数相同时，且N大于0.9*M且小于或等于M，判定该些N个像素点为背景像素点，从所述M帧三维影像中去除N个所述背景像素点，获得M帧无背景三维影像，即为某用户的影像。在连续获取的三维影像中，如果分属于不同三维影像的、位置参数相同的像素点的色彩参数相同，或者大部分(如90％)相同，即可认定像素点的位置为背景，从而可以将该像素点从相应的三维影像中去除。在连续获取的三维影像中， 去除每一个三维影像中的背景像素点后，剩下的像素点的集合即可代表该用户整体的行进轨迹。
步骤S3034)目标物坐标计算步骤，所述M帧无背景三维影像中所有像素点的位置参数的集合即为该用户整体的坐标集；在该坐标集中，位置参数z最大的像素点的位置参数被定义为目标物的坐标。如果将目标物进一步定义为用户的手部，就可以获取用户手部的实时坐标集。
如图17所示，所述目标物定位方法还可以包括如下步骤：步骤S401)货架设置步骤、步骤S402)用户身份识别步骤、步骤S403)货品感知步骤、步骤S404)购物用户判断步骤以及步骤S405)购物信息记录步骤。
步骤S401)货架设置步骤、步骤S402)用户身份识别步骤发生在步骤S301)空间设置步骤之后，且发生在步骤S302)三维影像采集步骤之前。步骤S401)货架设置步骤在所述封闭空间内设置至少一货架；每一货架包括至少一架板，每一架板上被放置有至少一个种类的货品。步骤S402)用户身份识别步骤识别进入所述封闭空间内的每一用户的身份信息。
步骤S403)货品感知步骤，用以实时感知每一货品的取放状态，当任一货品被取走或被放回时，获取被取走或被放回的货品的种类及数量。步骤S403)货品感知步骤与步骤S302)三维影像采集步骤及步骤S303)目标物坐标获取步骤分别执行，互不干涉。实时货品取放状态与目标物实时坐标皆发送至数据处理设备7以进行下一步的购物用户判断。
步骤S404)购物用户判断步骤，当任一货品被取走或被放回时，用以根据所述用户的身份信息及所述用户的实时位置判断取走或放回货品的用户身份；步骤S405)购物信息记录步骤，根据每一用户的身份信息生成至少一购物数据库，用以记录每一用户取走至少一货品的种类及数量。
如图18所示，步骤S404)购物用户判断步骤包括步骤S4041)货品信息存储步骤、步骤S4042)架板坐标存储步骤、步骤S4043)架板与用户匹配判断步骤以及步骤S4044)货品与用户匹配判断步骤。
步骤S4041)货品信息存储步骤，用以存储货品数据库，包括每一货品信息；步骤S4042)架板坐标存储步骤，在所述封闭空间内建立三维坐标系，用以存储货架坐标集及架板坐标集，设置架板上方的架板空间的高度，获取所述架板空间的坐标集；步骤S4043)架板与用户匹配判断步骤，当一架板上方的架板空间的坐标集与一用户手部坐标集有交集时，判定该架板与该用户匹配；步骤S4044)货品与用户匹配判断步骤，当有货品从一架板上被取走或被放置到一架板上时，且同一时刻下有一用户与该架板匹配，判定该货品与该用户匹配。在所述封闭空间内建立的三维坐标系下，每一架板上方用以放置货品的架板空间的坐标集是可以确定的，每一架板编号对应一个架板空间的坐标集。当用户手部的坐标集与所述架板空间的坐标集有交集时，说明用户手部伸入至一架板上方空间，也即该架板与该用户匹配。如果同时所述货品感知系统发现有货品从一架板上被取走或被放置到一架板上，即可以认定该用户发生了取货事件或者还货事件。
如图19所示，步骤S405)购物信息记录步骤包括步骤S4051)购物数据库生成步骤以及步骤S4052)购物数据库更新步骤。步骤S4051)购物数据库生成步骤，当一用户的身份被识别时，用以根据所述用户的身份信息生成该用户的购物数据库；步骤S4052)购物数据库更新步骤，当货品被取走时，根据被取走货品的种类及数量以及取走货品的用户的身份信息生成购物信息，且存储至该用户的购物数据库中；当货品被放回时，根据被放回货品的种类及数量以及放回货品的用户的身份信息生成归还信息，从该用户的购物数据库中删除与所述归还信息相应的购物信息。
步骤步骤S301)～S303)可以获取用户整体及局部的实时位置，再结合步骤步骤S401)～S405)，在货架上的货品发生取放事件时，可以判断取走或放回货品的用户的身份，并及时修正该用户购物数据库中的购物记录。如果所述货品感知系统发现有货品被某用户从一架板上被取走，且判定出被取走货品的种类和数量，即可将被取走货品的货品信息、数量、单价等信息写入该用户的购物数据库中。如果所述货品感知系统发现有货品被某用户放置到一架板上，且判定出被放回货品的种类和数量，即可将被放回货品的归还信息从该用户的购物数据库中删除。
用户在封闭空间内进行的购物活动中，可能会发生多次取货事件及多次归还事件，每次事件发生后，用户的购物数据库都会发生相应的变动，使其购物数据库中记载的购物信息与该用户实际购物内容保持一致，当用户离开封闭空间时，结算系统可以自动为其完成结算。
本发明的优点在于，提供一种目标物定位系统及定位方法，可以获取特定目标物在特定区域内实时位置，特别是可以获取每一用户在无人超市内的实时位置，结合每一架板或货品的位置以及货架上每一的货品取放状态，可以准确判断出货架上取走或放回的货品的用户的身份，从而更新该用户的购物记录，以便用户在购物完成后实现自动结算。
以上所述仅是本发明的优选实施方式，使本领域的技术人员更清楚地理解如何实践本发明，这些实施方案并不是限制本发明的范围。对于本技术领域的普通技术人员，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。

Claims (14)

  1. 一种目标物定位系统，其特征在于，包括：
    封闭空间；
    三维影像采集装置，用以实时采集至少一帧三维影像，所述三维影像中包括至少一目标物的全部或部分的影像；以及
    目标物坐标获取单元，设于一数据处理设备内，用以在所述封闭空间内建立三维坐标系，根据所述至少一帧三维影像实时获取所述目标物在所述三维坐标系的坐标集或坐标。
  2. 如权利要求1所述的目标物定位系统，其特征在于，
    所述三维影像采集装置包括
    至少一影像传感器，设于所述封闭空间顶部，所述影像传感器的视野范围覆盖所述封闭空间的全部底面；
    其中，每一影像传感器包括
    深度图像传感器，用以采集一目标物的至少一帧深度图像；
    RGB图像传感器，用以采集一目标物的至少一帧RGB图像；以及
    三维影像整合单元，用以将同一影像传感器同一时刻采集到的一帧深度图像及一帧RGB图像结合为一帧三维影像。
  3. 如权利要求1所述的目标物定位系统，其特征在于，
    所述目标物坐标获取单元包括
    坐标系建立单元，用以在所述封闭空间内建立三维坐标系；在所述封闭空间内任选一点作为原点，在水平方向设置X轴、Y轴，在竖直方向设置Z轴；
    参数获取单元，用以获取每一帧三维影像的每一像素点的位置参数和色彩参数；所述位置参数为x、y、z，代表该像素点在所述三维坐标系下的位置坐标；所述色彩参数为r、g、b，分别代表该像素点的三原色强度；
    背景去除单元，在连续采集的M帧三维影像中，若分属于不同三维影像、 位置参数相同的N个像素点的色彩参数相同时，且N大于0.9*M且小于或等于M，用以判定该N个像素点为背景像素点，从所述M帧三维影像中去除N个所述背景像素点，获得M帧无背景三维影像；以及
    目标物坐标计算单元，所述M帧无背景三维影像中所有像素点的位置参数的集合即为所述目标物的坐标集。
  4. 如权利要求3所述的目标物定位系统，其特征在于，
    所述参数获取单元包括
    传感器坐标获取单元，用以获取采集该帧三维影像的影像传感器的中心点在所述三维坐标系下的坐标；
    相对坐标获取单元，用以根据所述影像传感器的中心点为第二原点建立第二三维坐标系，从所述三维影像获取每一像素点在所述第二三维坐标系下的坐标；以及
    坐标修正单元，用以根据所述影像传感器中心点在所述三维坐标系下的坐标及所述三维影像中每一像素点在第二三维坐标系下的坐标，计算并修正所述三维影像的每一像素点在所述三维坐标系下的坐标，从而获得每一像素点的位置参数。
  5. 如权利要求1所述的目标物定位系统，其特征在于，还包括
    至少一货架，设置于所述封闭空间内；每一货架包括至少一架板，每一架板上被放置有至少一货品；
    用户身份识别系统，用以识别进入所述封闭空间内的每一用户的身份信息；
    货品感知系统，用以实时感知每一货品的取放状态，当任一货品被取走或被放回时，获取被取走或被放回的货品的种类及数量；
    购物用户判断系统，当任一货品被取走或被放回时，用以根据每一用户的身份信息及每一用户的实时位置判断取走或放回货品的用户身份；以及
    购物信息记录系统，根据每一用户的身份信息生成至少一购物数据库，用 以记录每一用户取走至少一货品的种类及数量。
  6. 如权利要求5所述的目标物定位系统，其特征在于，
    所述购物用户判断系统包括
    货品信息存储单元，用以存储货品数据库，包括每一货品信息；
    架板坐标存储单元，在所述封闭空间内建立三维坐标系，用以存储货架坐标集及架板坐标集，设置架板上方的架板空间的高度，获取所述架板空间的坐标集；
    架板与用户匹配判断单元，当一架板上方的架板空间的坐标集与一用户手部坐标集有交集时，判定该架板与该用户匹配；以及
    货品与用户匹配判断单元，当有货品从一架板上被取走或被放置到一架板上时，且同一时刻下有一用户与该架板匹配，判定该货品与该用户匹配。
  7. 如权利要求5所述的目标物定位系统，其特征在于，
    所述购物信息记录系统包括
    购物数据库生成单元，当一用户的身份被识别时，用以根据所述用户的身份信息生成该用户的购物数据库；以及
    购物数据库更新单元，当货品被取走时，根据被取走货品的种类及数量以及取走货品的用户的身份信息生成购物信息，且存储至该用户的购物数据库中；当货品被放回时，根据被放回货品的种类及数量以及放回货品的用户的身份信息生成归还信息，从该用户的购物数据库中删除与所述归还信息相应的购物信息。
  8. 一种目标物定位方法，其特征在于，包括如下步骤
    空间设置步骤，用以设置一封闭空间；
    三维影像采集步骤，用以实时采集至少一帧三维影像，所述三维影像中包括至少一目标物的全部或部分的图像；以及
    目标物坐标获取步骤，用以在所述封闭空间内建立三维坐标系，根据所述 至少一帧三维影像实时获取一目标物的坐标集或坐标。
  9. 如权利要求8所述的目标物定位方法，其特征在于，
    所述三维影像采集步骤包括
    影像传感器设置步骤，在所述封闭空间顶部设置至少一影像传感器，所述影像传感器的视野范围覆盖所述封闭空间的全部底面；
    原始图像采集步骤，用以实时同步采集至少一帧深度图像及至少一帧RGB图像；以及
    三维影像整合步骤，用以将同一影像传感器同一时刻采集到的一帧深度图像及一帧RGB图像结合为一帧三维影像。
  10. 如权利要求8所述的目标物定位方法，其特征在于，
    所述目标物坐标获取步骤包括
    坐标系建立步骤，用以在所述封闭空间内建立三维坐标系；在所述封闭空间内任选一点作为原点，在水平方向设置X轴、Y轴，在竖直方向设置Z轴；
    参数获取步骤，获取每一帧三维影像的每一像素点的位置参数和色彩参数；
    所述位置参数为x、y、z，代表该像素点在所述三维坐标系下的位置坐标；
    所述色彩参数为r、g、b，分别代表该像素点的三原色强度；
    背景去除步骤，在连续采集的M帧三维影像中，若分属于不同三维影像、位置参数相同的N个像素点的色彩参数相同时，且N大于0.9*M且小于或等于M，判定该些N个像素点为背景像素点，从所述M帧三维影像中去除N个所述背景像素点，获得M帧无背景三维影像；以及
    目标物坐标计算步骤，所述M帧无背景三维影像中所有像素点的位置参数的集合即为所述目标物的坐标集。
  11. 如权利要求10所述的目标物定位方法，其特征在于，
    在所述参数获取步骤中，获取一帧三维影像的每一像素点的位置参数，具体包括如下步骤：
    传感器坐标获取步骤，用以获取采集该帧三维影像的影像传感器的中心点在所述三维坐标系下的坐标；
    相对坐标获取步骤，用以根据所述影像传感器的中心点为第二原点建立第二三维坐标系，从所述三维影像获取每一像素点在所述第二三维坐标系下的坐标；以及
    坐标修正步骤，用以根据所述影像传感器中心点在所述三维坐标系下的坐标及所述三维影像中每一像素点在第二三维坐标系下的坐标，计算并修正所述三维影像的每一像素点在所述三维坐标系下的坐标，从而获得每一像素点的位置参数。
  12. 如权利要求8所述的目标物定位方法，其特征在于，还包括
    货架设置步骤，在所述封闭空间内设置至少一货架；每一货架包括至少一架板，每一架板上被放置有至少一货品；
    用户身份识别步骤，用以识别进入所述封闭空间内的每一用户的身份信息；
    货品感知步骤，用以实时感知每一货品的取放状态，当任一货品被取走或被放回时，获取被取走或被放回的货品的种类及数量；
    购物用户判断步骤，当任一货品被取走或被放回时，用以根据每一用户的身份信息及每一用户的实时位置判断取走或放回货品的用户身份；以及
    购物信息记录步骤，根据每一用户的身份信息生成至少一购物数据库，用以记录每一用户取走至少一货品的种类及数量。
  13. 如权利要求12所述的目标物定位方法，其特征在于，
    所述购物用户判断步骤包括
    货品信息存储步骤，用以存储货品数据库，包括每一货品信息；
    架板坐标存储步骤，在所述封闭空间内建立三维坐标系，用以存储货架坐标集及架板坐标集，设置架板上方的架板空间的高度，获取所述架板空间的坐标集；
    架板与用户匹配判断步骤，当一架板上方的架板空间的坐标集与一用户手部坐标集有交集时，判定该架板与该用户匹配；以及
    货品与用户匹配判断步骤，当有货品从一架板上被取走或被放置到一架板上时，且同一时刻下有一用户与该架板匹配，判定该货品与该用户匹配。
  14. 如权利要求12所述的目标物定位方法，其特征在于，
    所述购物信息记录步骤包括
    购物数据库生成步骤，当一用户的身份被识别时，用以根据所述用户的身份信息生成该用户的购物数据库；
    购物数据库更新步骤，当货品被取走时，根据被取走货品的种类及数量以及取走货品的用户的身份信息生成购物信息，且存储至该用户的购物数据库中；当货品被放回时，根据被放回货品的种类及数量以及放回货品的用户的身份信息生成归还信息，从该用户的购物数据库中删除与所述归还信息相应的购物信息。
PCT/CN2018/117325 2017-12-18 2018-11-23 目标物定位系统及定位方法 WO2019120040A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
KR1020207002600A KR102510679B1 (ko) 2017-12-18 2018-11-23 목표물 위치 결정 시스템 및 위치 결정 방법
EP18891454.3A EP3731546A4 (en) 2017-12-18 2018-11-23 SYSTEM FOR POSITIONING A TARGET OBJECT AND POSITIONING METHOD
AU2018386790A AU2018386790A1 (en) 2017-12-18 2018-11-23 Target positioning system and target positioning method
SG11202003732YA SG11202003732YA (en) 2017-12-18 2018-11-23 Target object positioning system and positioning method
US16/812,041 US20200202163A1 (en) 2017-12-18 2020-03-06 Target positioning system and target positioning method

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201711366390.4 2017-12-18
CN201711366390 2017-12-18
CN201810079535.0 2018-01-26
CN201810079535.0A CN108551658B (zh) 2017-12-18 2018-01-26 目标物定位系统及定位方法

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/812,041 Continuation US20200202163A1 (en) 2017-12-18 2020-03-06 Target positioning system and target positioning method

Publications (2)

Publication Number Publication Date
WO2019120040A1 WO2019120040A1 (zh) 2019-06-27
WO2019120040A9 true WO2019120040A9 (zh) 2019-12-12

Family

ID=62926467

Family Applications (4)

Application Number Title Priority Date Filing Date
PCT/CN2018/117326 WO2019144690A1 (zh) 2017-12-18 2018-11-23 基于影像监测的货品感知系统及货品感知方法
PCT/CN2018/117324 WO2019120039A1 (zh) 2017-12-18 2018-11-23 无人售货系统及无人售货方法
PCT/CN2018/117327 WO2019144691A1 (zh) 2017-12-18 2018-11-23 基于重量监测的货品感知系统及货品感知方法
PCT/CN2018/117325 WO2019120040A1 (zh) 2017-12-18 2018-11-23 目标物定位系统及定位方法

Family Applications Before (3)

Application Number Title Priority Date Filing Date
PCT/CN2018/117326 WO2019144690A1 (zh) 2017-12-18 2018-11-23 基于影像监测的货品感知系统及货品感知方法
PCT/CN2018/117324 WO2019120039A1 (zh) 2017-12-18 2018-11-23 无人售货系统及无人售货方法
PCT/CN2018/117327 WO2019144691A1 (zh) 2017-12-18 2018-11-23 基于重量监测的货品感知系统及货品感知方法

Country Status (8)

Country Link
US (3) US20200202163A1 (zh)
EP (3) EP3731546A4 (zh)
JP (4) JP7016187B2 (zh)
KR (3) KR102454854B1 (zh)
CN (10) CN208044756U (zh)
AU (5) AU2018405072A1 (zh)
SG (3) SG11202003731PA (zh)
WO (4) WO2019144690A1 (zh)

Families Citing this family (109)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN208044756U (zh) * 2017-12-18 2018-11-02 上海云拿智能科技有限公司 无人售货系统
WO2019152503A1 (en) * 2018-01-30 2019-08-08 PAR Excellence Systems, Inc. Supply storage holder and system
CN108652332A (zh) 2018-04-19 2018-10-16 上海云拿智能科技有限公司 悬挂式货架
WO2020003221A1 (en) * 2018-06-28 2020-01-02 Shekel Scales (2008) Ltd Systems and methods for weighing products on a shelf
CN108921540A (zh) * 2018-07-09 2018-11-30 南宁市安普康商贸有限公司 基于购买者位置定位的开放式自助销售方法及系统
EP3613638A1 (en) 2018-08-10 2020-02-26 Lg Electronics Inc. Vehicle display system for vehicle
CN109448237B (zh) * 2018-08-29 2019-10-11 拉卡拉汇积天下技术服务（北京）有限公司 无人售货装置及其上货方法和控制系统
CN109461255A (zh) * 2018-09-12 2019-03-12 湖南金码智能设备制造有限公司 一种用于售货机的自适应检测接取商品状态的装置和方法
CN109859377A (zh) * 2018-09-12 2019-06-07 盈奇科技（深圳）有限公司 一种重力感应自动售货机重力感应部分连接结构
CN109741533A (zh) * 2018-09-12 2019-05-10 盈奇科技（深圳）有限公司 一种重力感应自动售货机固定结构
CN109658596A (zh) * 2018-09-12 2019-04-19 盈奇科技（深圳）有限公司 一种重力感应自动售货机
CN110895747B (zh) * 2018-09-13 2024-04-02 阿里巴巴集团控股有限公司 商品信息识别、显示、信息关联、结算方法及系统
CN109214484B (zh) * 2018-09-20 2021-07-13 深圳蓝胖子机器智能有限公司 无人便利店控制方法、装置和计算机可读存储介质
CN109330284B (zh) * 2018-09-21 2020-08-11 京东方科技集团股份有限公司 一种货架系统
CN109512217A (zh) * 2018-09-27 2019-03-26 安徽国药医疗科技有限公司 一种基于智能货架的物品管理系统
CN109341763B (zh) * 2018-10-10 2020-02-04 广东长盈科技股份有限公司 一种基于物联网的运输数据采集系统及方法
CN109409291B (zh) * 2018-10-26 2020-10-30 虫极科技（北京）有限公司 智能货柜的商品识别方法和系统及购物订单的生成方法
CN109685979A (zh) * 2018-11-01 2019-04-26 深圳市友族科技有限公司 一种无人售货柜及无人售货方法
CN109581532A (zh) * 2018-11-12 2019-04-05 百度在线网络技术（北京）有限公司 用于确定被取走或放置的物品的位置的设备、方法及货架
CN109363450B (zh) * 2018-12-03 2024-05-14 深圳市力合鑫源智能技术有限公司 智能货架及货架上被取走或被添加物品位置的识别方法
CN109741519B (zh) * 2018-12-10 2021-01-19 深圳市思拓通信系统有限公司 一种无人超市货架监控系统及其控制方法
CN111325049A (zh) * 2018-12-13 2020-06-23 北京京东尚科信息技术有限公司 商品识别方法、装置、电子设备及可读介质
CN109598874A (zh) * 2018-12-18 2019-04-09 北京缤歌网络科技有限公司 一种显示控制方法
US11017641B2 (en) * 2018-12-21 2021-05-25 Sbot Technologies Inc. Visual recognition and sensor fusion weight detection system and method
CN109726759B (zh) * 2018-12-28 2021-08-17 北京旷视科技有限公司 无人售货方法、装置、系统、电子设备及计算机可读介质
CN109711360B (zh) * 2018-12-29 2021-03-30 北京沃东天骏信息技术有限公司 售货机风险控制方法、装置和控制系统
CN111222389B (zh) * 2019-01-10 2023-08-29 图灵通诺（北京）科技有限公司 商超货架上商品的分析方法和系统
CN109872448A (zh) * 2019-01-19 2019-06-11 创新奇智(广州)科技有限公司 一种异物检测方法、计算机可读存储介质及检测系统
CN111222870B (zh) * 2019-01-24 2024-02-27 图灵通诺（北京）科技有限公司 结算方法、装置和系统
CN109858446A (zh) * 2019-01-31 2019-06-07 深兰科技（上海）有限公司 一种新零售场景下物品注册方法及装置
CN111523348B (zh) * 2019-02-01 2024-01-05 百度（美国）有限责任公司 信息生成方法和装置、用于人机交互的设备
CN109840504B (zh) * 2019-02-01 2022-11-25 腾讯科技（深圳）有限公司 物品取放行为识别方法、装置、存储介质及设备
CN109919040B (zh) * 2019-02-15 2022-04-19 北京清瞳时代科技有限公司 货物的品规信息识别方法及装置
CN109829348A (zh) * 2019-02-25 2019-05-31 浙江工业大学 基于stm32单片机的食品信息采集系统
CN111507702A (zh) * 2019-03-07 2020-08-07 河源市联腾实业有限公司 无人超市自助购物方法、计算机可读存储介质及系统
CN110007755A (zh) * 2019-03-15 2019-07-12 百度在线网络技术（北京）有限公司 基于动作识别的物体事件触发方法、装置及其相关设备
CN109977825B (zh) * 2019-03-15 2021-06-25 百度在线网络技术（北京）有限公司 物品识别方法及装置
CN109977826B (zh) * 2019-03-15 2021-11-02 百度在线网络技术（北京）有限公司 物体的类别识别方法和装置
CN110009836A (zh) * 2019-03-29 2019-07-12 江西理工大学 基于高光谱摄像技术的深度学习的系统及方法
CN110147723B (zh) * 2019-04-11 2022-08-19 苏宁云计算有限公司 一种无人店中顾客异常行为的处理方法及系统
CN110321797A (zh) * 2019-05-31 2019-10-11 苏宁云计算有限公司 商品识别方法和装置
CN110175590A (zh) * 2019-05-31 2019-08-27 北京华捷艾米科技有限公司 一种商品识别方法及装置
CN110260796A (zh) * 2019-06-04 2019-09-20 上海追月科技有限公司 货品感知系统、货品感知方法及电子设备
CN110150906A (zh) * 2019-06-04 2019-08-23 上海追月科技有限公司 托盘及货架
CN110197561A (zh) * 2019-06-10 2019-09-03 北京华捷艾米科技有限公司 一种商品识别方法、装置及系统
CN112115745A (zh) * 2019-06-21 2020-12-22 杭州海康威视数字技术股份有限公司 一种商品漏扫码行为识别方法、装置及系统
CN110298961A (zh) * 2019-06-28 2019-10-01 北京云迹科技有限公司 用于机器人的计费方法及装置
CN110403400B (zh) * 2019-07-16 2020-09-08 合肥美的智能科技有限公司 货柜
US20210027104A1 (en) * 2019-07-25 2021-01-28 Microsoft Technology Licensing, Llc Eyes-off annotated data collection framework for electronic messaging platforms
CN110837824B (zh) * 2019-08-09 2022-12-16 达闼科技（北京）有限公司 用于售货装置中的商品识别方法、售货装置及存储介质
CN111783509A (zh) * 2019-08-29 2020-10-16 北京京东尚科信息技术有限公司 自动结算方法、装置、系统和存储介质
JP7368982B2 (ja) * 2019-09-05 2023-10-25 東芝テック株式会社 販売管理システム及び販売管理方法
CN112466035B (zh) * 2019-09-06 2022-08-12 图灵通诺（北京）科技有限公司 基于视觉和重力感应的商品识别方法、装置和系统
CN110717593B (zh) * 2019-10-14 2022-04-19 上海商汤临港智能科技有限公司 神经网络训练、移动信息测量、关键帧检测的方法及装置
CN111753614A (zh) * 2019-11-01 2020-10-09 北京京东尚科信息技术有限公司 一种商品货架的监控方法和装置
CN111783513A (zh) * 2019-11-18 2020-10-16 北京沃东天骏信息技术有限公司 货物补充方法、装置和系统
CN111109947A (zh) * 2019-11-28 2020-05-08 上海追月科技有限公司 货架、货品识别方法及电子设备
CN111025417B (zh) * 2019-12-17 2022-04-29 万翼科技有限公司 建材存放区域异常检测方法及相关产品
CN111127174A (zh) * 2020-01-06 2020-05-08 鄂尔多斯市东驿科技有限公司 智能化无人超市控制系统
CN111145409A (zh) * 2020-01-06 2020-05-12 鄂尔多斯市东驿科技有限公司 人体识别定位跟踪系统
CN111207815A (zh) * 2020-01-16 2020-05-29 首钢京唐钢铁联合有限责任公司 一种汽车衡混装计量系统
CN111243166A (zh) * 2020-01-20 2020-06-05 广东优信无限网络股份有限公司 一种用于餐饮售卖机的提示方法
KR102177852B1 (ko) * 2020-01-31 2020-11-11 임시원 정신건강의학의 병원용 자산 관리를 위한 방법 및 장치
US11694501B2 (en) 2020-02-17 2023-07-04 True Manufacturing Co., Inc. Refrigerated vending system and method
CN111429655A (zh) * 2020-02-28 2020-07-17 上海追月科技有限公司 货品识别方法及货品识别系统、存储介质及电子设备
CN111428621A (zh) * 2020-03-20 2020-07-17 京东方科技集团股份有限公司 货架交互方法、装置和货架
CN111476609A (zh) * 2020-04-10 2020-07-31 广西中烟工业有限责任公司 零售数据获取方法、系统、设备及存储介质
CN111540106A (zh) * 2020-04-14 2020-08-14 合肥工业大学 一种无人超市系统
CN111553914B (zh) * 2020-05-08 2021-11-12 深圳前海微众银行股份有限公司 基于视觉的货物检测方法、装置、终端及可读存储介质
CN111667639A (zh) * 2020-05-28 2020-09-15 北京每日优鲜电子商务有限公司 图书归还服务的实现方法、装置及智能图书柜
CN111831673B (zh) * 2020-06-10 2024-04-12 上海追月科技有限公司 货品识别系统、货品识别方法及电子设备
CN111680657B (zh) * 2020-06-15 2023-05-05 杭州海康威视数字技术股份有限公司 一种物品取放事件的触发人员确定方法、装置及设备
CN111738184B (zh) * 2020-06-28 2023-09-26 杭州海康威视数字技术股份有限公司 一种商品取放识别方法、装置、系统及设备
CN111672774B (zh) * 2020-07-30 2021-07-06 梅卡曼德（北京）机器人科技有限公司 一种货品分拣系统及分拣方法
CN113808342B (zh) * 2020-08-19 2023-09-01 北京京东乾石科技有限公司 物品支付方法、装置、计算机可读介质及电子设备
CN111731669A (zh) * 2020-08-26 2020-10-02 江苏神彩科技股份有限公司 危险物品智能监管贮存箱及其监管系统
CN112084940A (zh) * 2020-09-08 2020-12-15 南京和瑞供应链管理有限公司 物资盘点管理系统及方法
KR102253385B1 (ko) * 2020-09-14 2021-05-18 주식회사농심 불량 멀티포장제품 검출시스템
CN112101867A (zh) * 2020-09-15 2020-12-18 四川精益达工程检测有限责任公司 一种用于货架上物品的信息生成方法及系统
CN112309031B (zh) * 2020-09-17 2022-06-07 北京京东乾石科技有限公司 无人货柜混放检测方法、装置、设备及可读存储介质
CN112215167B (zh) * 2020-10-14 2022-12-20 上海爱购智能科技有限公司 一种基于图像识别的智能商店控制方法和系统
CN112515428B (zh) * 2020-11-17 2023-11-03 上海追月科技有限公司 一种货架
JP2022097195A (ja) * 2020-12-18 2022-06-30 トヨタ自動車株式会社 情報処理装置、及び、情報処理方法
JP2022110700A (ja) * 2021-01-19 2022-07-29 東芝テック株式会社 報知装置及び報知方法
CN112419015B (zh) * 2021-01-25 2021-05-25 北京每日优鲜电子商务有限公司 物品信息推送方法、装置、电子设备和计算机可读介质
CN113067847B (zh) * 2021-02-02 2022-07-12 绍兴晨璞网络科技有限公司 一种匹配式超宽带定位系统架构设计方法
CN112985570B (zh) * 2021-02-04 2023-09-29 支付宝(杭州)信息技术有限公司 称重货架的传感器的标定方法和系统
CN112801055B (zh) * 2021-04-01 2021-06-22 湖南云星人工智能研究院有限公司 一种基于薄膜压力传感器阵列的无人超市定位跟踪方法
CN113554809A (zh) * 2021-04-22 2021-10-26 浙江星星冷链集成股份有限公司 一种商品信息更新装置及无人售货系统
CN113554808B (zh) * 2021-04-22 2023-09-29 浙江星星冷链集成股份有限公司 一种上货管理装置及无人售货系统
CN113524194B (zh) * 2021-04-28 2023-03-21 重庆理工大学 基于多模特征深度学习的机器人视觉抓取系统的目标抓取方法
CN113229686A (zh) * 2021-05-21 2021-08-10 上海追月科技有限公司 一种货架
CN113407571B (zh) * 2021-06-30 2023-04-07 重庆博尔德医疗科技股份有限公司 一种基于称重原理的计数方法
CN113558441A (zh) * 2021-07-07 2021-10-29 合肥美的智能科技有限公司 无人售货柜
CN113537773A (zh) * 2021-07-15 2021-10-22 深圳医智联科技有限公司 一种针对检验科培养瓶的管理方法及系统
CN113479534A (zh) * 2021-07-31 2021-10-08 深圳市坤同智能仓储科技有限公司 称重货架的模块化层及货架系统
CN113592339A (zh) * 2021-08-09 2021-11-02 四川长虹智能制造技术有限公司 一种货物信息校准系统、方法及电子设备
CN113421376B (zh) * 2021-08-23 2021-12-28 浙江星星冷链集成股份有限公司 一种无人售货系统
CN113869821A (zh) * 2021-09-23 2021-12-31 云南电力试验研究院(集团)有限公司 一种计量物资识别周转方法和系统
KR102417612B1 (ko) * 2021-10-28 2022-07-06 주식회사 케이엔케이 부품 검사 시스템 및 그 제어방법
CN113706227A (zh) * 2021-11-01 2021-11-26 微晟(武汉)技术有限公司 一种货架商品推荐方法及装置
CN113821674B (zh) * 2021-11-23 2022-02-25 北京中超伟业信息安全技术股份有限公司 一种基于孪生神经网络的智能货物监管方法及系统
CN114140696A (zh) * 2022-01-27 2022-03-04 深圳市慧为智能科技股份有限公司 商品识别系统优化方法、装置、设备及存储介质
CN114559431A (zh) * 2022-03-02 2022-05-31 上海擎朗智能科技有限公司 一种物品配送方法、装置、机器人及存储介质
ES2922762A1 (es) * 2022-05-27 2022-09-20 Ostirion S L U Procedimiento y equipo de localizacion e identificacion optica de instrumentos y aparatos
KR102597927B1 (ko) * 2022-08-16 2023-11-30 주식회사 포르망 화장품의 중량 선별 장치
KR102482821B1 (ko) * 2022-09-06 2022-12-29 주식회사 포이엔 무선 네트워크 기반의 실시간 계측 시스템
CN115510501B (zh) * 2022-11-22 2023-02-28 中汽信息科技(天津)有限公司 一种汽车数据防篡改方法和系统
CN116422602A (zh) * 2023-03-17 2023-07-14 广东铭钰科技股份有限公司 一种一体式产品称重标识检测系统及方法

Family Cites Families (111)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3384193A (en) * 1965-02-01 1968-05-21 Toledo Scale Corp Control circuitry
JPS6052366B2 (ja) * 1977-12-02 1985-11-19 オムロン株式会社 重量検知ユニツト
JPH1017121A (ja) * 1996-07-02 1998-01-20 Keiyo Syst Kk 自動棚卸し装置
JP4452786B2 (ja) * 1996-09-29 2010-04-21 雅信 鯨田 遠隔商品販売のための装置及び方法
JP3213696B2 (ja) * 1996-11-29 2001-10-02 東芝テック株式会社 買上商品登録装置及びこの登録装置を用いた買上商品決済システム
US6711293B1 (en) * 1999-03-08 2004-03-23 The University Of British Columbia Method and apparatus for identifying scale invariant features in an image and use of same for locating an object in an image
JP2002161942A (ja) * 2000-08-10 2002-06-07 Iau:Kk 免震装置、滑り支承また免震構造
KR20010067855A (ko) * 2001-04-03 2001-07-13 장성계 창고형 매장에서의 중량감지에 의한 상품 재고량 파악시스템
JP3837475B2 (ja) * 2001-07-19 2006-10-25 独立行政法人産業技術総合研究所 自動化ショッピングシステム
AU2003221887A1 (en) * 2002-04-12 2003-10-27 Tritek Technologies, Inc. Mail sorting processes and systems
US6747560B2 (en) * 2002-06-27 2004-06-08 Ncr Corporation System and method of detecting movement of an item
US7039222B2 (en) * 2003-02-28 2006-05-02 Eastman Kodak Company Method and system for enhancing portrait images that are processed in a batch mode
CN2634950Y (zh) * 2003-06-06 2004-08-25 何永强 货架柱
CN2788670Y (zh) * 2005-03-25 2006-06-21 李瑞铭 强化支撑架结构
US20080109375A1 (en) * 2006-11-08 2008-05-08 Ricci Christopher P Position-enhanced wireless transaction security
US7949568B2 (en) * 2007-08-31 2011-05-24 Accenture Global Services Limited Determination of product display parameters based on image processing
DE202007016804U1 (de) * 2007-12-01 2008-02-28 Golletz Gmbh Vorrichtung zur regalartigen Anordnung von Warenträgern
CN101604312A (zh) * 2007-12-07 2009-12-16 宗刚 信息的检索管理交流的方法和系统
US20090177089A1 (en) * 2008-01-04 2009-07-09 Assaf Govari Three-dimensional image reconstruction using doppler ultrasound
CN101281597B (zh) * 2008-04-25 2011-06-22 北京工业大学 一种产品包装图形标识信息在线实时采集识别装置及方法
KR100988754B1 (ko) * 2008-07-10 2010-10-25 박찬용 자동판매 기능을 갖는 무인 점포 시스템
KR101467509B1 (ko) 2008-07-25 2014-12-01 삼성전자주식회사 이미지 센서 및 이미지 센서 동작 방법
US8165929B2 (en) * 2008-08-04 2012-04-24 Chudy Group, LLC Adaptive pharmaceutical product management methods and system
KR101051355B1 (ko) * 2009-06-09 2011-07-22 (주)이지스 3차원 공간 데이터를 이용한 카메라 영상의 3차원 좌표 획득방법 및 이를 이용한 카메라 연동 제어방법
KR20100136089A (ko) 2009-06-18 2010-12-28 주식회사 비즈모델라인 인덱스 교환을 통한 다중 코드 생성 방식 오티피 출력 방법 및 시스템과 이를 위한 휴대폰 및 기록매체
CN101989333A (zh) * 2009-07-31 2011-03-23 上海杉达学院 仓储控制系统
KR20100003343A (ko) * 2009-12-15 2010-01-08 (주)리테일테크 압력 센서를 이용한 소매 매장의 결품 방지 시스템
US20110174753A1 (en) * 2010-01-18 2011-07-21 Matthew Pinto Stackable transport system
KR101179108B1 (ko) * 2010-04-27 2012-09-07 서울시립대학교 산학협력단 중첩 전방위 영상을 이용하여 객체의 3차원 좌표를 결정하기 위한 시스템 및 그 방법
EP2580687A4 (en) * 2010-06-14 2014-04-30 Trutag Technologies Inc SYSTEM FOR CHECKING AN ELEMENT IN A PACKAGE WITH A DATABASE
KR20120051212A (ko) * 2010-11-12 2012-05-22 엘지전자 주식회사 멀티미디어 장치의 사용자 제스쳐 인식 방법 및 그에 따른 멀티미디어 장치
KR101223206B1 (ko) * 2010-11-18 2013-01-17 광주과학기술원 3차원 영상 생성 방법 및 시스템
US20130046635A1 (en) * 2011-08-19 2013-02-21 Bank Of America Corporation Triggering offers based on detected location of a mobile point of sale device
CN102374860B (zh) * 2011-09-23 2014-10-01 奇瑞汽车股份有限公司 三维视觉定位方法及系统
JP6005848B2 (ja) * 2012-05-10 2016-10-12 エスセーアー・ハイジーン・プロダクツ・アーベー 保管場所にある所望のアイテムを見つける際に支援をする方法
CN202807610U (zh) * 2012-08-31 2013-03-20 宁波市新光货架有限公司 一种中型仓储货架
CN202820536U (zh) * 2012-10-22 2013-03-27 山东固美思耐金属制品有限公司 无线称重超市货架托板
KR101931819B1 (ko) * 2012-12-31 2018-12-26 (주)지오투정보기술 영상 정보와 항공사진 데이터를 비교하여 대상 객체를 결정하고 카메라 획득정보를 이용하여 대상 객체의 3차원 좌표를 획득하는 수치지도 제작 시스템
MX2015008992A (es) * 2013-01-11 2016-03-17 Tagnetics Inc Sensor que indica fuera de existencia.
US20160048798A1 (en) * 2013-01-11 2016-02-18 Tagnetics, Inc. Inventory sensor
CN103192958B (zh) * 2013-02-04 2015-07-08 中国科学院自动化研究所北仑科学艺术实验中心 船舶姿态显示装置的控制方法
CN104021538B (zh) * 2013-02-28 2017-05-17 株式会社理光 物体定位方法和装置
KR20140136089A (ko) * 2013-05-16 2014-11-28 (주)에스모바일텍 진열 상품 재고 관리 솔루션을 구비한 전자 선반 라벨 시스템
US10984372B2 (en) * 2013-05-24 2021-04-20 Amazon Technologies, Inc. Inventory transitions
US10268983B2 (en) * 2013-06-26 2019-04-23 Amazon Technologies, Inc. Detecting item interaction and movement
US10176456B2 (en) * 2013-06-26 2019-01-08 Amazon Technologies, Inc. Transitioning items from a materials handling facility
US10290031B2 (en) * 2013-07-24 2019-05-14 Gregorio Reid Method and system for automated retail checkout using context recognition
JP6168917B2 (ja) * 2013-08-26 2017-07-26 株式会社日立システムズ 健康管理システム及び健康管理方法
CN103557841B (zh) * 2013-08-28 2016-08-10 陈天恩 一种提高多相机合成影像摄影测量精度的方法
US20160203499A1 (en) * 2013-09-06 2016-07-14 Nec Corporation Customer behavior analysis system, customer behavior analysis method, non-transitory computer readable medium, and shelf system
AU2013101592A4 (en) * 2013-10-25 2014-01-16 Housl Pty Ltd Property Leasing
US9916561B2 (en) * 2013-11-05 2018-03-13 At&T Intellectual Property I, L.P. Methods, devices and computer readable storage devices for tracking inventory
CN103559758B (zh) * 2013-11-06 2015-12-30 上海煦荣信息技术有限公司 一种智能化的售货系统及售货方法
CN203693111U (zh) * 2014-02-28 2014-07-09 苏州工业园区航成工业零件有限公司 一种新型台面式称重货架
CN104240007A (zh) * 2014-07-09 2014-12-24 深圳市茁壮网络股份有限公司 一种基于射频识别的物品管理方法及系统
CN203975446U (zh) * 2014-07-29 2014-12-03 晟通科技集团有限公司 全铝托盘
CN104217231A (zh) * 2014-08-29 2014-12-17 南京大学 一种基于非精确锚节点的rfid定位系统及定位方法
CN104576845A (zh) * 2014-12-16 2015-04-29 深圳市德上光电有限公司 一种图形化的蓝宝石衬底的制造方法
CN105989389A (zh) * 2015-02-11 2016-10-05 北京鼎九信息工程研究院有限公司 一种二维码
WO2016133609A1 (en) * 2015-02-18 2016-08-25 Fedex Corporate Services, Inc. Managing logistics information related to a logistics container using a container interface display apparatus
WO2016152765A1 (ja) * 2015-03-23 2016-09-29 日本電気株式会社 商品情報管理装置、商品情報管理システム、商品情報管理方法およびプログラム
US10318917B1 (en) * 2015-03-31 2019-06-11 Amazon Technologies, Inc. Multiple sensor data fusion system
KR101807513B1 (ko) * 2015-05-13 2017-12-12 한국전자통신연구원 3차원 공간에서 영상정보를 이용한 사용자 의도 분석장치 및 분석방법
CN204883802U (zh) * 2015-06-24 2015-12-16 中山润涛智科技术有限公司 重量感应器及应用该感应器的物联网自动库存统计系统
US9911290B1 (en) * 2015-07-25 2018-03-06 Gary M. Zalewski Wireless coded communication (WCC) devices for tracking retail interactions with goods and association to user accounts
JP6562077B2 (ja) * 2015-08-20 2019-08-21 日本電気株式会社 展示装置、表示制御装置および展示システム
US20170061258A1 (en) * 2015-08-25 2017-03-02 Nokia Technologies Oy Method, apparatus, and computer program product for precluding image capture of an image presented on a display
CN105314315B (zh) * 2015-10-30 2017-12-15 无锡职业技术学院 智能货架测试系统
CN205251049U (zh) * 2015-11-06 2016-05-25 江苏正贸仓储设备制造有限公司 带计量器货架
CN106875203A (zh) * 2015-12-14 2017-06-20 阿里巴巴集团控股有限公司 一种确定商品图片的款式信息的方法及装置
US10592854B2 (en) * 2015-12-18 2020-03-17 Ricoh Co., Ltd. Planogram matching
JP6319669B2 (ja) * 2016-01-20 2018-05-09 パナソニックＩｐマネジメント株式会社 商品モニタリング装置、商品モニタリングシステムおよび商品モニタリング方法
CN105719188B (zh) 2016-01-22 2017-12-26 平安科技（深圳）有限公司 基于多张图片一致性实现保险理赔反欺诈的方法及服务器
SG11201807291PA (en) 2016-02-29 2018-09-27 Signpost Corp Information processing system
CN105787490A (zh) * 2016-03-24 2016-07-20 南京新与力文化传播有限公司 基于深度学习的商品潮流识别方法及装置
CN105678981A (zh) * 2016-03-26 2016-06-15 陈功平 一种组合式智能预警储物架
AU2017245132A1 (en) 2016-03-29 2018-10-11 Bossa Nova Robotics Ip, Inc. System and method for locating, identifying and counting ltems
JP6736334B2 (ja) * 2016-04-07 2020-08-05 東芝テック株式会社 画像処理装置
CN105910379B (zh) * 2016-05-09 2018-07-27 海信（山东）冰箱有限公司 一种用于冰箱的称重装置及称重抽屉
TWI578272B (zh) * 2016-05-18 2017-04-11 Chunghwa Telecom Co Ltd Shelf detection system and method
CN105901987A (zh) * 2016-06-30 2016-08-31 常州市南飞机械有限公司 可称重的货架
CN106326852A (zh) * 2016-08-18 2017-01-11 无锡天脉聚源传媒科技有限公司 一种基于深度学习的商品识别方法及装置
CN106355368A (zh) * 2016-08-29 2017-01-25 苏州倾爱娱乐传媒有限公司 一种卖场货品的智能管理系统
CN106408374A (zh) * 2016-08-30 2017-02-15 佛山市明扬软件科技有限公司 一种安全性高的超市自动收费方法
CN106441537A (zh) * 2016-09-08 2017-02-22 蝶和科技（中国）有限公司 一种称重货架的称重方法以及使用该方法的货架
CN106556341B (zh) * 2016-10-08 2019-12-03 浙江国自机器人技术有限公司 一种基于特征信息图形的货架位姿偏差检测方法和系统
CN106454733A (zh) * 2016-11-03 2017-02-22 安徽百慕文化科技有限公司 一种基于无线传感网络的在线监测系统
CN106781121A (zh) * 2016-12-14 2017-05-31 朱明� 基于视觉分析的超市自助结账智能系统
CN106845962A (zh) * 2016-12-29 2017-06-13 广州途威慧信息科技有限公司 一种自动结算货款系统及方法
CN106910086A (zh) * 2017-01-18 2017-06-30 甄启源 一种超市智能购物系统
CN106875465B (zh) * 2017-01-20 2021-06-11 奥比中光科技集团股份有限公司 基于rgbd图像的三维操控空间的建立方法及设备
CN106845965A (zh) * 2017-02-13 2017-06-13 上海云剑信息技术有限公司 线下超市自助购物的手机结账系统
CN106934692B (zh) * 2017-03-03 2020-12-22 陈维龙 物品信息处理系统、方法及装置
CN106920152A (zh) * 2017-03-07 2017-07-04 深圳市楼通宝实业有限公司 自助售卖方法及系统
US9805360B1 (en) * 2017-03-17 2017-10-31 Philz Coffee, Inc. Location based device flagging and interface
CN107134053B (zh) * 2017-04-19 2019-08-06 石道松 智能售货门店
CN107045641B (zh) * 2017-04-26 2020-07-28 广州图匠数据科技有限公司 一种基于图像识别技术的货架识别方法
CN107093269A (zh) * 2017-06-08 2017-08-25 上海励识电子科技有限公司 一种智能无人售货系统及方法
CN107463946B (zh) * 2017-07-12 2020-06-23 浙江大学 一种结合模板匹配与深度学习的商品种类检测方法
CN107123006A (zh) * 2017-07-12 2017-09-01 杨智勇 一种智能购物系统
CN107341523A (zh) * 2017-07-13 2017-11-10 浙江捷尚视觉科技股份有限公司 基于深度学习的快递单信息识别方法和系统
US10655945B2 (en) * 2017-07-19 2020-05-19 Symbol Technologies, Llc Methods and apparatus to coordinate movement of automated vehicles and freight dimensioning components
CN107480735B (zh) * 2017-07-24 2018-08-14 葛代荣 一种自助验货方法和系统
CN107451891A (zh) * 2017-07-27 2017-12-08 惠州市伊涅科技有限公司 便利店自动售卖方法
CN107481000A (zh) * 2017-07-27 2017-12-15 惠州市伊涅科技有限公司 无人超市售货方法
CN107451776A (zh) * 2017-07-27 2017-12-08 惠州市伊涅科技有限公司 无人超市补货方法
US10445694B2 (en) * 2017-08-07 2019-10-15 Standard Cognition, Corp. Realtime inventory tracking using deep learning
US11023850B2 (en) * 2017-08-07 2021-06-01 Standard Cognition, Corp. Realtime inventory location management using deep learning
CN107481414A (zh) * 2017-08-21 2017-12-15 文彬 一种开放式无人售卖装置的实时导购方法及系统
US20200394599A1 (en) 2017-11-29 2020-12-17 Ntt Docomo, Inc. Shelf-allocation information generating device and shelf-allocation information generating program
CN208044756U (zh) * 2017-12-18 2018-11-02 上海云拿智能科技有限公司 无人售货系统

Also Published As

Publication number Publication date
JP2022043070A (ja) 2022-03-15
JP7016187B2 (ja) 2022-02-04
CN108509847B (zh) 2021-07-20
WO2019144690A1 (zh) 2019-08-01
JP7229580B2 (ja) 2023-02-28
AU2018386790A1 (en) 2020-04-23
EP3745100A4 (en) 2021-03-24
US20200202163A1 (en) 2020-06-25
US11501523B2 (en) 2022-11-15
CN208044756U (zh) 2018-11-02
CN108551658A (zh) 2018-09-18
WO2019120039A1 (zh) 2019-06-27
EP3745100A1 (en) 2020-12-02
CN108551658B (zh) 2021-04-09
EP3745296A1 (en) 2020-12-02
CN108492482B (zh) 2020-11-03
CN108497839B (zh) 2021-04-09
SG11202003732YA (en) 2020-05-28
AU2018405072A1 (en) 2020-04-30
AU2018405073A1 (en) 2020-06-18
CN108332829A (zh) 2018-07-27
US20200202137A1 (en) 2020-06-25
JP7170355B2 (ja) 2022-11-14
CN208892110U (zh) 2019-05-24
AU2022203376A1 (en) 2022-06-09
US20200258069A1 (en) 2020-08-13
CN108492157B (zh) 2023-04-18
CN207783158U (zh) 2018-08-28
KR20200037833A (ko) 2020-04-09
JP7078275B2 (ja) 2022-05-31
EP3731546A4 (en) 2021-02-17
CN108497839A (zh) 2018-09-07
WO2019144691A1 (zh) 2019-08-01
KR20200023647A (ko) 2020-03-05
JP2022043067A (ja) 2022-03-15
EP3731546A1 (en) 2020-10-28
KR102510679B1 (ko) 2023-03-16
JP2021511554A (ja) 2021-05-06
SG11202004369RA (en) 2020-06-29
AU2018102235A4 (en) 2022-03-17
KR20200051705A (ko) 2020-05-13
JP2021507203A (ja) 2021-02-22
KR102378059B1 (ko) 2022-03-25
CN108520194A (zh) 2018-09-11
EP3745296A4 (en) 2021-03-24
SG11202003731PA (en) 2020-05-28
WO2019120040A1 (zh) 2019-06-27
KR102454854B1 (ko) 2022-10-14
CN108492157A (zh) 2018-09-04
CN108509847A (zh) 2018-09-07
CN108492482A (zh) 2018-09-04

Similar Documents

Publication Publication Date Title
WO2019120040A9 (zh) 目标物定位系统及定位方法
JP7264401B2 (ja) 会計方法、装置及びシステム
CN111428743B (zh) 商品识别方法、商品处理方法、装置及电子设备
WO2021147950A1 (zh) 货品识别方法、货品识别系统及电子设备
US20240029405A1 (en) System and method for selecting an item from a plurality of identified items by filtering out back images of the items
US20240029275A1 (en) System and method for identifying unmoved items on a platform during item identification
TW202147177A (zh) 適用於自助結帳服務之辨識設備及辨識方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18891454

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2019543888

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20207002600

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2018386790

Country of ref document: AU

Date of ref document: 20181123

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018891454

Country of ref document: EP

Effective date: 20200720

NENP Non-entry into the national phase

Ref country code: JP