WO2022254958A1 - 無人店舗の商品を管理する管理システム、管理方法及びプログラム - Google Patents

無人店舗の商品を管理する管理システム、管理方法及びプログラム Download PDF

Info

Publication number
WO2022254958A1
WO2022254958A1 PCT/JP2022/016902 JP2022016902W WO2022254958A1 WO 2022254958 A1 WO2022254958 A1 WO 2022254958A1 JP 2022016902 W JP2022016902 W JP 2022016902W WO 2022254958 A1 WO2022254958 A1 WO 2022254958A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
customer
information
weight
hand
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/016902
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
山本紘暉
宮島洋介
渡辺修司
水島達哉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Classmethod Inc
Original Assignee
Classmethod Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Classmethod Inc filed Critical Classmethod Inc
Publication of WO2022254958A1 publication Critical patent/WO2022254958A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
    • G16Y40/00—IoT characterised by the purpose of the information processing
    • G16Y40/10—Detection; Monitoring

Definitions

  • the present invention relates to a management server and management method for managing products in unmanned stores.
  • Patent Document 1 As a background art of this technical field, there is Japanese Patent Laying-Open No. 2020-173815 (Patent Document 1). This publication states, "In the autonomous store tracking system, products 111, 112, 113 and 114 are arranged in the product storage area 102 of the store 101. Person 103 is monitored while standing near merchandise storage area 102 at time 141.” (see abstract). Also, as a related background art, there is International Publication No. 2015/173869 (Patent Document 2). In this publication, ⁇ After detecting a person's hand within a first area set according to the position of the product, the detection unit (52) includes the same area as the first area or the first area. When a human hand is not detected in the second area that is set according to the position and the product is not detected in the product detection area set according to the position, it is detected that the product has been picked up.” (see summary).
  • the present application includes a plurality of means for solving the above problems.
  • a management system for managing products which is means for acquiring weight information transmitted from weight sensors installed on product shelves.
  • FIG. 1 is an example of a configuration diagram of an entire product management system 1.
  • FIG. 3 is an example of a hardware configuration of a management server 101; 1 is an example of a hardware configuration of a mobile terminal 102; 1 is an example of a hardware configuration of a signage terminal 103.
  • FIG. It is an example of the hardware configuration of the camera sensor units 104 and 104a.
  • 3 is an example of a hardware configuration of a weight sensor unit 105;
  • This is an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device by wireless connection.
  • This is an example of a connection configuration for communicating with a microcomputer 710 connected to a weight sensor device 608 via a wired connection.
  • FIG. 2 is a diagram illustrating a functional configuration example of the product management system 1; FIG. It is a figure explaining the functional structural example of a signage display. It is an example of product master information 1300 . It is an example of user master information 1400 . It is an example of segment information 1500 . It is an example of acquisition determination range information 1600 . 17 is an example of location tracking information 1700. FIG. It is an example of shelf event information 1800 . It is an example of estimate management information 1900 . It is an example of weight information 2000 transmitted from the microcomputer 710 to the data aggregation transmitter 630. FIG. It is a structural example of the frame for product shelf installation. It is a figure explaining the structure of a product shelf.
  • FIG. 2400 It is a schematic diagram explaining the acquisition determination range of goods. It is an example of the weight information accumulation processing flow 2400 . It is an example of the location information accumulation processing flow 2500 . It is an example of an overall processing flow 2600 for product purchase at a store. It is an example of the entering-store processing flow 2700. FIG. It is an example of the goods determination processing flow 2800. FIG. It is an example of the customer determination processing flow 2900. FIG. It is an example of the estimated display processing flow 3000 . 31 is an example of a payment processing flow 3100; It is an example of signage display processing flow 3200 . It is an example of a hardware reference model 3300 . It is an example of an order management screen. It is an example of a product purchase screen.
  • This embodiment assumes a product management system 1 that manages products in an unmanned store.
  • a customer enters a store, he or she holds the two-dimensional barcode or non-contact IC chip displayed on the customer's mobile terminal over the store entrance management terminal to record the store entry.
  • the store By picking up the products displayed on the product shelf in the store, the information and number of the products picked up are displayed on the customer's mobile terminal, and the payment for the product is completed by leaving the store.
  • the store may be manned.
  • cameras and product shelves are installed on several product shelf installation frames whose dimensions have been determined in advance, and then several pre-registered system initial settings are automatically executed, whereby unmanned operation can be performed. It is possible to reduce the labor, time, and cost due to the preparation of each store when setting up the store.
  • FIG. 1 is an example of a block diagram of an entire product management system 1.
  • the merchandise management system 1 includes a signage terminal 103 installed in or around a merchandise shelf installation frame 2100 in a store, a position detection camera sensor unit 104, a skeleton detection camera sensor unit 104a, a weight sensor unit 105, and entry management. It has a terminal 106 and is connected to the management server 101 and the settlement processing server 107 via a network.
  • the product management system 1 is accessed from the mobile terminal 102 via the network.
  • Each terminal can transmit and receive information via the network regardless of whether the network is wired or wireless.
  • the management server 101 accumulates and analyzes various events that occur in stores, and performs processes such as product determination, customer determination, and quotation display.
  • a mobile terminal 102 is a mobile terminal owned by a customer, such as a smart phone, tablet, or wearable terminal, and displays information on products picked up by the customer.
  • the camera sensor unit 104 has a camera management terminal 530 to which a plurality of camera devices 506 are connected, and acquires the movement trajectory (for example, walking trajectory) of the customer in the store.
  • the camera sensor unit 104a has a camera management terminal 530a to which a plurality of camera devices 506a are connected, and acquires the characteristic parts of the customer's skeleton in the store.
  • the camera management terminal 530 and the camera management terminal 530a are separate terminals, one terminal may have both functions.
  • the weight sensor unit 105 has a plurality of weight sensor devices 608 installed in product baskets on product shelves, a microcomputer 710, and a data aggregation transmitter 630, detects changes in weight, and transmits weight information to the management server 101. do.
  • the store entry management terminal 106 reads the two-dimensional bar code or contactless IC chip displayed on the mobile terminal owned by the customer to identify the customer's user information.
  • the payment processing server 107 receives a payment processing request for the product acquired by the customer from the management server 101, and executes the payment processing using the payment method associated with the customer's user information.
  • Each terminal of the product management system 1 and the management server 101 may be, for example, a portable terminal (mobile terminal) such as a smart phone, a tablet, a mobile phone, or a personal digital assistant (PDA).
  • a portable terminal such as a smart phone, a tablet, a mobile phone, or a personal digital assistant (PDA).
  • PDA personal digital assistant
  • the function may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal.
  • VR Virtual Reality
  • AR Augmented Reality
  • MR Multiple Reality
  • it may be a combination of these terminals.
  • a combination of one smart phone and one wearable terminal can logically function as one terminal.
  • Information processing terminals other than these may also be used.
  • Each terminal and management server 101 of the product management system 1 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as a RAM (Random Access Memory), an IC card, a hard disk drive, an SSD (Solid State Drive), flash memory and other auxiliary storage devices, communication control units such as network cards, wireless communication modules, mobile communication modules, etc., touch panels, keyboards, mice, voice input, and input from motion detection by imaging with the camera unit, etc.
  • An input device and an output device such as a monitor or display are provided.
  • the output device may be a device or terminal that transmits information for output to an external monitor, display, printer, device, or the like.
  • each module is stored in the main storage device, and each functional element of the entire system is realized by the processor executing these programs and applications.
  • each of these modules may be implemented by hardware by integration or the like.
  • Each module may be an independent program or application, or may be implemented in the form of a partial subprogram or function in one integrated program or application.
  • each module is described as an entity (subject) that performs processing, but actually a processor that processes various programs and applications (modules) executes processing.
  • DB databases
  • a “database” is a functional element (storage) that stores a set of data so as to accommodate any data manipulation (eg, extraction, addition, deletion, overwriting, etc.) from a processor or external computer.
  • the database implementation method is not limited, and may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.
  • RDBMS relational database
  • non-RDBMS non-relational database
  • FIG. 2 is an example of the hardware configuration of the management server 101.
  • the management server 101 is configured by a server arranged on a cloud, for example.
  • Various modules (programs and applications) 210 are stored in the main storage device 201, and each functional element of the management server 101 is realized by the processor 203 executing these programs and applications. Each module and its processing will be described later.
  • the auxiliary storage device 202 includes a master DB 220 and various information storage DBs 221 .
  • the master DB 220 stores user master information 1400, product master information 1300, and the like.
  • Various information storage DB 221 stores information transmitted from camera sensor unit 104, camera sensor unit 104a, weight sensor unit 105, and entry management terminal 106, or stores various information processed and processed in management server 101. Remember. The information stored in each will be described later.
  • FIG. 3 is an example of the hardware configuration of the mobile terminal 102.
  • the mobile terminal 102 is configured by a terminal such as a smart phone or a tablet, for example.
  • Programs and applications such as the product purchase management module 310 are stored in the main storage device 301, and each functional element of the mobile terminal 102 is realized by the processor 303 executing these programs and applications.
  • the product purchase management module 310 cooperates with the estimate display module 1106 and the mobile purchase management module 1108 of the management server 101 to display a product purchase screen and a purchase history.
  • Auxiliary storage device 302 stores user management information 320 stored in mobile terminal 102 .
  • FIG. 4 is an example of the hardware configuration of the signage terminal 103.
  • the signage terminal 103 is configured by a terminal such as a smart phone, tablet, notebook PC, desktop PC, microcomputer, single board computer, or the like.
  • a Raspberry Pi for example, can be used as a single board computer.
  • Programs and applications such as a signage information display module 410 are stored in the main storage device 401 , and each functional element of the signage terminal 103 is realized by the processor 403 executing these programs and applications.
  • Auxiliary storage device 402 stores signage information 420 such as advertisements of products to be displayed.
  • the signage information display module 410 receives a display instruction from the signage processing module 1202 of the management server 101, and displays signage information 420 corresponding to the display instruction on an output device 405 such as a display or electronic paper.
  • the configuration may be such that the signage information transmitted from the management server 101 is received and displayed.
  • FIG. 5A is an example of the hardware configuration of the position detection camera sensor unit 104.
  • FIG. A plurality of camera devices 506 are connected to a camera management terminal 530 via a network.
  • the camera management terminal 530 is composed of, for example, a desktop PC, a microcomputer, a single board computer, a cloud server, or the like.
  • Programs and applications such as the trajectory management module 510 are stored in the main storage device 501 , and each functional element of the camera management terminal 530 is realized by the processor 503 executing these programs and applications.
  • FIG. 5B is an example of the hardware configuration of the skeleton detection camera sensor unit 104a.
  • a plurality of camera devices 506a are connected to a camera management terminal 530a via a network.
  • the camera management terminal 530a is composed of, for example, a desktop PC, a microcomputer, a single board computer, a cloud server, or the like.
  • Programs and applications such as the skeleton detection module 510a are stored in the main storage device 501a, and each functional element of the camera management terminal 530a is realized by the processor 503a executing these programs and applications.
  • the camera device 506 includes a TOF (Time of Flight) sensor.
  • a TOF sensor is a sensor that measures the distance to an object by detecting the flight time (time difference) of light emitted from a light source, reflected by the object, and returning to the sensor.
  • a ToF image sensor can also detect distance information for each pixel and obtain a highly accurate distance image. Also, by using a plurality of ToF sensors and cameras, it is possible to determine the shape of the object and the three-dimensional overlap with higher accuracy. For example, when a plurality of customers are in the store at the same time, it is possible to determine the overlap of each person or detect the movement of hands.
  • the installation position and installation angle of each camera must be measured correctly, and detailed settings adjustment (calibration) must be performed. )Is required.
  • the camera devices 506 are installed at predetermined positions of the product shelf installation frame 2100 standardized in advance, and the settings of these cameras are collectively set from the management server 101 side. By doing so, the simplification and efficiency of the initial setting and the improvement of the acquisition accuracy of the trajectory information are realized (see FIG. 21).
  • a plurality of camera devices 506a can be similarly configured.
  • the trajectory management module 510 aggregates image information acquired from a plurality of camera devices 506, distance information to an object, etc., and executes analysis to determine the distance to the object, the shape of the object, and the shape of the object. Locate. It may also be called a position detection module 510 . In addition, by accumulating this distance information in time series, the customer's movement trajectory in the sensor acquisition range is acquired and accumulated in the trajectory information 520 . Specifically, when a plurality of camera devices 506 start detecting a trajectory, the trajectory management module 510 assigns a person ID to the detected person, and chronologically displays the position information of this person at each time. Stored in the trajectory information 520 . By connecting these pieces of time-series information, it is possible to acquire the movement trajectory of the person corresponding to the specific person ID.
  • the trajectory management module 510 transmits to the management server 101 the person ID for reproducing the movement trajectory, time information, and position information at that time. Alternatively, the trajectory information summarizing these is transmitted to the management server 101 .
  • the skeleton detection module 510a performs image analysis on the image information acquired from the plurality of camera devices 506a, extracts characteristic portions based on the skeleton model of the human body, and accumulates them in the characteristic portion information 520a. Further, the skeleton detection module 510a may calculate other feature parts based on the detected feature parts and store them in the feature part information 520a.
  • the skeleton detection module 510 a may directly transmit information on the customer's features to the management server 101 .
  • the skeleton detection module 510a transmits the information of the customer's characteristic part to the trajectory management module 510 of the camera management terminal 530, and after integrating the information detected by the trajectory management module 510 side and the information of the customer's characteristic part, The trajectory management module 510 of the camera management terminal 530 may transmit to the management module management server 101 .
  • FIG. 6 is an example of the hardware configuration of the weight sensor unit 105.
  • a plurality of weight sensor devices 608 are connected to a data aggregation transmitter 630 via a microcomputer (microcontroller) 710 .
  • the data aggregation transmitter 630 is composed of, for example, a desktop PC, a microcomputer, a single board computer, a cloud server, or the like.
  • a Raspberry Pi for example, can be used as the single board computer.
  • the main storage device 601 stores programs and applications such as various modules 610 for processing the weight information received from the microcomputer 710 and transmitting it to the management server 101, and the processor 603 executes these programs and applications. By doing so, each functional element of the data aggregation transmitter 630 is realized.
  • the auxiliary storage device 602 stores IoT device management information 620 such as setting information and management information of the data aggregation transmitter 630 , the microcomputer 710 and the weight sensor device 608 .
  • FIG. 7 is an example of a connection configuration for communicating with a microcomputer 710 connected to the weight sensor device 608 by wireless connection.
  • a plurality of weight sensor devices 608 are installed on the product shelf, and a product basket is installed on each weight sensor device 608 .
  • the weight sensor device 608 is a device that measures the weight of the products placed in the product basket and detects changes in the weight.
  • the acquired weight data is converted into digital information by the A/D converter 720 and transmitted to the microcomputer 710 .
  • the microcomputer 710 transmits the received weight information to the data aggregation transmitter 630 using wireless communication such as Bluetooth (registered trademark).
  • FIG. 20 is an example of weight information 2000 transmitted from microcomputer 710 to data aggregation transmitter 630 .
  • the weight information 2000 has information of each item shown in the item 2020, and has a value such as a sample value 2040 as a field name 2030.
  • FIG. The weight information 2000 has items such as product basket ID 2001, data type, weight 2002, number of times of transmission, state 2003, elapsed time after activation, previous weight 2004, delay time 2005, state change presence/absence 2006, and the like.
  • a product basket ID 2001 is an ID set for each weight sensor device 608 . It is associated with the product ID 1502 of the product put in the product basket, and from the change in the weight 2002 and the product ID 1502 associated with the product basket ID 2001, it is calculated how many of which products have been taken out of the product basket. be able to.
  • Weight 2002 displays the weight in the product basket in grams. The weight of the contents is displayed with the weight of the product basket removed from the measured value, and 0 is indicated when there is no contents. However, it may be configured to include the weight of the product basket.
  • the number of times of transmission indicates the number of times of data transmission since the start of the microcomputer 710 . The number of times of transmission is sequentially counted up, and the value is returned to 0 when the memory overflows.
  • State 2003 indicates the state of weight sensor device 608 . 0 is indicated when the measured weight data is in an UNSTABLE state, and 1 is indicated when the weight data is in a stable state.
  • the elapsed time after activation displays the elapsed time from the activation of the microcomputer 710 in units of milliseconds. It counts up every millisecond and resets the value to 0 if the memory overflows.
  • the previous weight 2004 displays the weight in grams at the time of the previous measurement. Delay time 2004 displays the time in milliseconds until state 2003 changes from UNSTABLE to STABLE.
  • a status change presence/absence 2006 is a flag indicating whether the status 2003 has changed since the previous measurement. It indicates 1 if the state 2003 has changed since the last measurement, and 0 if it has not.
  • the microcomputer 710 sets the state 2003 to 0 indicating "UNSTABLE state”, and sets the state change presence/absence 2006 to 1 indicating "yes”.
  • the microcomputer 710 constantly receives weight data from the weight sensor device 608, but if there is no change in weight (if the amount of change does not exceed the threshold value), the weight data is aggregated and sent periodically, such as every 10 seconds. device 630. In this case, the state 2003 is set to 1 indicating "STABLE state”, and the state change presence/absence 2006 is set to 0 indicating "no".
  • the microcomputer 710 transmits the weight information 2000 including the weight 2002 to the data aggregation transmitter 630 when the state 2003 changes from "UNSTABLE" to "STABLE".
  • the status change presence/absence 2006 indicates a change when the status 2003 between the previous measurement and the new measurement changes from "STABLE” to "UNSTABLE” or from "UNSTABLE” to “STABLE”.
  • the state may be changed simply when the difference between the previous weight and the latest weight exceeds a predetermined threshold value, for example, 5 g.
  • a predetermined threshold value for example, 5 g or less, it may be determined that the weight data is in a stable state.
  • the microcomputer 710 constantly receives weight data from the weight sensor device 608. As described above, if there is no change in weight, the weight information 2000 is sent to the data aggregation transmitter 630 every 10 seconds, for example. Send to Also, when the weight changes, such as when the product is picked up, the first weight information 2000 is transmitted at that timing, and the second weight information is transmitted after the weight stabilizes. I have to.
  • the weight information is sent twice to perform two-step processing. make it possible to
  • the signage display processing 3200 on the signage terminal 103 in response to the transmission of the weight information 2000 for the first time, when the product is picked up, a description of the product and a product that the customer wants to purchase together with the product can be displayed. can be immediately displayed on the signage terminal 103 without delay.
  • the product determination processing 2800 in response to the second transmission of the weight information 2000 based on the numerical value after the weight is stabilized, the number of products picked up can be correctly counted.
  • the number of each product can be calculated and used for inventory management. For example, in this embodiment, since we know where the product should be and the quantity, if the product is about to run out, we will send an out-of-stock forecast to prompt the staff in the backyard to replenish the product. In this case, it is possible to manage inventory by sending a layout alert and prompting the backyard staff to relocate.
  • a predetermined number of products such as 10 are placed in the product basket and sales are started, and the number of products falls below a predetermined number such as 3, the current number of products together with the product ID (or product basket ID) is displayed.
  • a system may be adopted in which the difference between 3 items and the initial 10 items is notified, and based on this notification, the 7 items are automatically replenished by a mechanism such as a robot or a conveyor.
  • the data aggregation transmitter 630 that has received the weight information 2000 converts the received information into a predetermined protocol such as the MQTT protocol, encrypts it, and transmits it to the management server 101 .
  • Time information can also be added to each piece of weight information 2000 .
  • the weight information 2000 may include time information at which the data aggregation transmitter 630 transmits data, or when the microcomputer 710 transmits the weight information 2000 to the data aggregation transmitter 630, the time information may be configured to include
  • weight time series data can be generated.
  • the data aggregation transmitter 630 can control and initialize the microcomputer 710 and weight sensor device 608 based on the control information received from the management server 101 . For example, it is possible to switch the power on/off, or to add or update setting information.
  • FIG. 8 shows an example of a connection configuration for communicating with the microcomputer 710 connected to the weight sensor device 608 via a wired connection.
  • a microcomputer 710 is connected to a plurality of weight sensor devices 608 via A/D converters 720 by wires.
  • the microcomputer is also preferentially connected to the data aggregation transmitter 630, and communicates with it by CAN (Controller Area Network).
  • the transceivers 730 and 731 transmit and receive data to and from each other by the CAN system, and a controller 740 connected by a daisy chain controls communication by the CAN system.
  • Microcomputer 710 can be powered via USB or CAN. 7, the microcomputer transmits weight information 2000 to data aggregation transmitter 630, and data aggregation transmitter 630 transmits control information to microcomputer 710.
  • FIG. 9 is an example of the hardware configuration of the entrance management terminal 106.
  • the entry management terminal 106 is configured by a terminal such as a smartphone or a tablet, or a dedicated terminal for reading a two-dimensional bar code or a non-contact IC chip.
  • Programs and applications such as an entry management module 910 are stored in the main storage device 901, and each functional element of the entry management terminal 106 is realized by the processor 903 executing these programs and applications.
  • the entry management module 910 reads a two-dimensional barcode displayed on the customer's mobile terminal 102 with the camera unit 906, or communicates with the non-contact IC chip of the customer's mobile terminal 102 with the input device 904, Receive user-specific information.
  • the store entry management module 910 transmits user identification information to the management server 101 as information that the user has entered the store. At this time, the date and time when the user identification information is received may be transmitted to the management server 101 as the store entry time.
  • the auxiliary storage device 902 stores entrance management terminal management information 920 . Here, user identification information and time information acquired when the entering process is performed may be stored.
  • Information input at the time of entering the store includes reading barcodes such as two-dimensional barcodes, communication using the contactless IC chip of the mobile terminal 102, electronic money, contactless IC cards, and magnetic information reading. may be performed by In this case, it is possible to change the operation at the exit gate when leaving the store depending on how the customer enters the store. For example, in the case of entering the store using a two-dimensional barcode or non-contact IC chip of an application on the mobile terminal 102, the exit gate automatically opens and credit card payment is performed on the application.
  • a two-dimensional bar code may be printed on the product picked up when leaving the store, and the printed matter may be read by an automatic payment machine so that the guest user can pay with a credit card, electronic money, or cash.
  • FIG. 10 is an example of the hardware configuration of the payment processing server 107.
  • the payment processing server 107 is configured by a server arranged on a cloud, for example.
  • a payment processing execution module 1010 is stored in the main storage device 1001, and each functional element of the payment processing server 107 is realized by the processor 1003 executing these programs and applications.
  • the payment processing execution module 1010 executes payment processing for product purchases by the mobile terminal 102 and product purchases by stores.
  • the executed settlement result is transmitted to the mobile terminal 102 and displayed on the screen which is the output device 305 in cooperation with the product purchase management module 310 .
  • the auxiliary storage device 1002 stores personal information, user information, credit card information, and payment processing information 1020 such as payment history necessary for payment processing.
  • FIG. 11 is a diagram for explaining a functional configuration example of the product management system 1.
  • the management server 101 includes a weight information processing module 1101, a position information processing module 1102, a skeleton detection processing module 1102a, an entrance/exit management module 1103, a product determination module 1104, a customer number determination module 1105, an estimate display module 1106, and a payment management module 1107. , a mobile purchase management module 1108, etc., and various modules 210, which are programs and applications, and execute various processes.
  • the management server 101 also stores segment information 1500 that stores information received from various sensors and information generated by the management server 101, acquisition determination range information 1600, position tracking information 1700, skeleton detection information 1700a, shelf event information 1800, It has estimate management information 1900, entry information 1111, and the like.
  • modules 210 are described as being arranged on one management server 101, they may be arranged on separate virtual servers on the cloud, or some of them may be arranged on separate virtual servers.
  • a configuration may be adopted in which each group is arranged on a virtual server.
  • Various types of information may also be stored in different storage devices on the cloud, or may be arranged on the cloud for each group of some of them.
  • the weight information processing module 1101 receives weight information including the product basket ID from the weight sensor unit 105 and accumulates the weight information together with time information corresponding to the weight information.
  • the weight information accumulation processing flow will be described later with reference to FIG.
  • the position information processing module 1102 receives from the camera sensor unit 104 the person ID, time information, and position information at that time for reproducing the movement locus.
  • the position information processing module 1102 specifies the range in which the person stayed based on the received position information, and accumulates information on the specified stay range together with time information.
  • the location information accumulation processing flow will be described later with reference to FIG.
  • the skeleton detection processing module 1102a receives a photographed image of a person in the store and time information from the camera sensor unit 104a.
  • the skeleton detection processing module 1102a can obtain predetermined feature points (estimated hand points) of the person's upper body by calculation. By using this information, the management server 101 can determine the correspondence relationship between the estimated points of the hand and the positions of the product shelves.
  • the skeleton detection processing flow will be described later with reference to FIG.
  • the entering/exiting management module 1103 receives customer user identification information from the entry management module 910 of the entry management terminal 106, identifies which user ID the received user identification information corresponds to, and receives the information. Stored in the entry information 1111 together with the time information. The time at which the bar code displayed on the customer's mobile terminal 102 was read or the time at which the information was obtained from the contactless IC chip through non-contact communication is stored instead of or together with the time information at which the information was received. good too.
  • the store entry processing flow will be described later with reference to FIG.
  • the merchandise determination module 1104 determines the merchandise picked up by the customer and the number thereof based on the information accumulated in the shelf event information 1800 .
  • the product determination processing flow will be described later with reference to FIG.
  • the customer number determination module 1105 Based on the information accumulated in the shelf event information 1800, the position tracking information 1700, and the skeleton detection information 1700a, the customer number determination module 1105 identifies the shelf where the weight change occurs (event occurs). Identify customers who were near At that time, it is determined which customer has picked up how many products. The customer number determination processing flow will be described later with reference to FIG.
  • the estimate display module 1106 compares the results of the store entry process, the product determination process, and the customer determination process, and displays on the mobile terminal 102 of the customer the details and the number of products picked up and the price.
  • the estimate display processing flow will be described later with reference to FIG.
  • the payment management module 1107 requests the payment processing server 107 to execute payment processing when the customer leaves the store after executing the estimation processing.
  • the payment processing flow will be described later with reference to FIG.
  • the mobile purchase management module 1108 accepts orders when online orders are placed directly from the product purchase management module 310 of the mobile terminal 102 .
  • the mobile purchase management module 1108 receives the selection of products and the selection of the quantity from the user, it calculates the total amount based on the product master information 1300 and displays it on the mobile terminal 102 .
  • the payment management module 1107 transmits user information such as the total amount and the corresponding user ID to the payment processing server 107, and the payment processing server 107 executes payment processing.
  • FIG. 12 is a diagram illustrating a functional configuration example of signage display.
  • the IoT information processing module 1201 of the management server 101 receives the weight information transmitted according to the MQTT format. Note that the IoT information processing module 1201 is arranged before the weight information processing module 1101 in FIG. 11 .
  • the data aggregation transmitter 630 transmits the weight information in two parts: a first time when the weight changes and a second time when the value is stable.
  • the IoT information processing module 1201 When the IoT information processing module 1201 receives the first weight information, it transfers the weight information to the signage processing module 1202 . Note that the product basket ID included in the weight information may be notified without transferring the weight information as it is.
  • the IoT information processing module 1201 transfers the weight information to the weight information processing module 1101 in FIG. 11 in order to specify the product and the number of products. Note that the IoT information processing module 1201 distributes each piece of weight information. may also be stored.
  • the IoT information processing module 1201 receives the weight information transmitted in MQTT format as an MQTT broker and distributes it to the signage processing module 1202 or the weight information processing module 1101 that subscribes to it.
  • the weight sensor unit 105 may distribute the first weight information and the second weight information to different destinations in advance.
  • the distribution of the IoT information processing module 1201 may be based on whether the number of times of reception after stabilization is the first or the second time, or based on the presence or absence of state change 2006 included in the weight information and the flag of the state 2003. It is also possible to adopt a configuration in which distribution is executed by In this case, for example, when the state change presence/absence 2006 is "changed” and the state 2003 is “unstable", it is determined that the weight information is the first time. When the state change presence/absence 2006 is "changed” and the state 2003 is “stable”, it is determined that the weight information is the second time. Alternatively, the determination may be made based only on the status change presence/absence 2006, which is a flag indicating that the weight has changed. For example, when the state change presence/absence 2006 becomes "changed” indicating that the weight has changed, the weight information may be sent to the signage processing module 1202 as the first information. .
  • FIG. 13 is an example of product master information 1300 .
  • the product master information 1300 has information on each item shown in item 1320 and has a value such as sample value 1340 as field name 1330 .
  • the product master information 1300 has items such as inventory availability, product image, product ID 1301, product name 1302, price 1303, priority, cost price, tax amount, sales type, weight 1304, signage information 1305, and the like.
  • the product ID 1301 is used as a key, and can be used to identify the product name 1302 and to display signage information 1305 corresponding to the product ID 1301 .
  • the weight 1304 is used when calculating how many pieces of the product are picked up from the product basket with respect to the amount of change in weight acquired by the weight sensor unit 105 .
  • FIG. 14 is an example of user master information 1400.
  • the user master information 1400 has information of each item shown in item 1420 and has a value such as sample value 1440 as field name 1430 .
  • the user master information 1400 has items such as a user ID 1401, user display ID, name, email address, date of birth, gender, user identification information 1402, and the like.
  • the user ID 1401 is information for specifying a user, and each user's mobile terminal 102 is managed in association with this user ID 1401. Based on the user ID 1401, estimate information for purchased products, etc. can be displayed.
  • User identification information 1402 is information used to identify a user, and is, for example, a value such as a token (hash value) that is periodically updated in consideration of security.
  • FIG. 15 is an example of segment information 1500 .
  • the segment information 1500 has information of each item shown in item 1520 and has a value such as sample value 1540 as field name 1530 .
  • the segment information 1500 is information that associates the product basket ID 1501 with the product ID 1502 . With this segment information 1500, it is possible to specify the product ID from the product basket ID included in the weight information received from the weight sensor unit 105, and to grasp which product has changed the weight.
  • the signage terminal 103 displays an instruction, for example, "Please put one product in the product basket", and asks the user to put one product in the product basket.
  • the weight sensor unit 105 acquires the weight of one product and transmits it to the management server 101 .
  • the product determination module 1104 of the management server 101 displays a list of candidates whose weight 1304 is close to the measured weight from the plurality of products registered in the product master information 1300 on the signage terminal 103 near the product shelf in order of closeness. Display an instruction such as "Please select an item in the shopping cart from the list.”
  • the manager selects the same information as the product actually placed in the product basket from the displayed list. According to this selection, the product determination module 1104 associates the product basket ID 1501 having the product basket with the product with the product ID 1502 and stores it in the segment information 1500 . Instead of selecting from a list, the configuration may be such that the weight is automatically associated with the closest product. Next, the signage terminal 103 displays an instruction such as "Please put three products in the product basket.” The average weight is calculated from the weight information of the three products, and the product determination module 1104 automatically sets the weight error when the product increases or decreases.
  • FIG. 16 is an example of acquisition determination range information 1600 .
  • the acquisition determination range information 1600 has information of each item shown in item 1620 and has a value such as sample value 1640 as field name 1630 .
  • Acquisition determination range information 1600 is information that defines, for each product basket, the range in which the product can be taken from the product basket.
  • the acquisition determination range information 1600 has a product basket ID 1601 , a determination area 1602 and a determination stay range 1603 .
  • the product basket ID 1601 is associated with the weight sensor device 608 of the weight sensor unit 105, and it is possible to specify from which product basket the weight of the product has changed as a result.
  • a determination area 1602 indicates the range in which the product associated with the product basket ID 1601 can be taken by coordinates in the product shelf setting frame 2100 (see FIG. 21).
  • the determination stay range 1603 indicates the range in which the product associated with the product basket ID 1601 can be obtained in units of partitions divided in advance.
  • FIG. FIG. 21 shows a configuration example of a product shelf installation frame.
  • cameras and product shelves are installed in a product shelf installation frame whose dimensions have been determined in advance, and by automatically executing several pre-registered system initial settings, an unmanned store can be set up. It is possible to reduce the labor, time, and cost due to the creation of each store.
  • the product shelf installation frame 2100 is, for example, a turret-shaped frame measuring 2.5 m in length and width and 3 m in height.
  • a camera device 506 is installed at a predetermined position on four pillars 2101 or their vicinity (for example, beams 2102). Alternatively, it may have a mounting portion for the camera device 506 so that the camera device 506 can be installed at this position.
  • One support 2101 of the frame has coordinates (0,0) and defines a coordinate plane in millimeters in the x and y directions.
  • the coordinates of the lower left column are (0, 0)
  • the coordinates of the lower right column are (2500, 0)
  • the coordinates of the right back column are (2500, 2500). Setting the camera sensor unit 104 and the weight sensor unit 105 is semi-automated by installing a product shelf at a predetermined position in this.
  • FIG. 22 is a diagram illustrating the configuration of a product shelf.
  • 3 ⁇ 3 product baskets 2202 are installed on one product shelf 2201 .
  • a weight sensor device 608 is installed under each product basket to measure the weight in each product basket.
  • the size of the product shelf is assumed to be width ⁇ depth ⁇ height of 450 ⁇ 500 ⁇ 1500 mm.
  • setting information from the management server 101 is sent to the camera sensor unit 104 and the weight sensor. It is sent to the unit 105 and the setting is semi-automatically completed.
  • a virtual acquisition area 2252 where products can be acquired is set in front of the product shelf 2251 .
  • the virtual acquisition area defines 760 mm from the top of the shelf, which is the same range that the customer can reach for the product.
  • FIG. 23 is a schematic diagram illustrating a product acquisition determination range.
  • the product shelf installation frame 2100 is viewed from above at 2300, and the same coordinates as in FIG. , 2500).
  • Four product shelves 2301 AS01 to AS04 are installed in the front part of FIG. 760 mm in the y-axis direction from the end of the product shelf and each 450 mm width of the product shelf 2301 in the x-axis direction is defined as one region.
  • the area up to is defined as the range in which the customer can acquire the product, that is, the acquisition determination range.
  • the acquisition determination range 2352 of the product shelf 2351 of AS02 is A0, A1, A2, A3, and A4.
  • the acquisition determination range of the product shelf of AS01 is A0, A1, A2, and A3, and the acquisition determination range of the product shelf of AS04 is A2, A3, A4, and A5.
  • the acquisition determination range for each product basket is stored in the acquisition determination range information 1600 of FIG.
  • AS02_B is stored as the product basket ID 1501 in the second product basket from the top of the product shelf AS02. , 1260) is the part surrounded by (2150, 500).
  • the interior of the store or the interior of the product shelf setting frame 2100 can also be represented by partitions that have been divided in advance. .
  • one section has a range of 450 mm ⁇ 760 mm.
  • the product acquisition range is within the plane range of the x and y axes.
  • a product basket can also be specified in the z-axis direction, such as a row, and accordingly, the determination area may also include values in the z-axis direction.
  • FIG. 17A is an example of location tracking information 1700.
  • the location tracking information 1700 has information of each item shown in item 1720 and has a value such as sample value 1740 as field name 1730 .
  • the location tracking information 1700 is information that stores in chronological order who was where in the store. It may also be called a position information storage unit or position information storage means.
  • the position tracking information 1700 has items such as a time limit 1705, a stay range 1701, a person ID 1702, a store ID 1703, a time stamp 1704, and the like.
  • the area of stay 1701 stores preliminarily partitioned sections within the store or the product shelf setting frame 2100, and is the location where the customer specified from the position information transmitted from the camera sensor unit 104 stayed. memorize
  • FIG. 17B is an example of skeleton detection information 1700a.
  • the skeleton detection information 1700a has information of each item shown in the item 1720a, and has a value such as a sample value 1740a as a field name 1730a.
  • the skeleton detection information 1700a is information about a characteristic portion detected by applying a skeleton detection module to a photographed image of a customer in the store. This information relates to the position coordinates of predetermined features of the upper body of the human body (eg, head, left and right shoulders, left and right elbows, left and right wrists) detected by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a. . Furthermore, using the positional coordinates of these characteristic parts, the positional coordinates of the characteristic parts of other parts of the human body (for example, estimated points of the left and right hands, etc.) obtained by calculation can be similarly included.
  • FIG. 17 illustrates skeleton detection information 1700a of a customer when one customer is detected on the photographed image.
  • skeleton detection information 1700a is created separately for each customer.
  • the table of (B) in FIG. 17 may be created separately for each customer.
  • the table of FIG. 17B may be collectively created for a plurality of customers on the same photographed image.
  • Predetermined features of the upper body and putative hand point information can be integrated with the detected information of the location tracking information 1700 .
  • the skeleton detection information 1700a can include deadlines, time stamps, and the like, as in the case of the location tracking information 1700.
  • the table in (B) of FIG. 17 can include tracking information within the image.
  • the table of (A) in FIG. 17 and the table of (B) in FIG. 17 may be created separately, or may be created collectively.
  • the data in FIG. 17B created based on the image captured by the camera sensor unit 104a is inserted into the table in FIG. 17A created based on the image captured by the camera sensor unit 104. good too. Or it may be vice versa.
  • a person ID 1702 is information specifying a person transmitted from the camera sensor unit 104, and is not associated with a user ID at this stage. Therefore, it is stored that a certain person was in the stay range 1701 at the time indicated by the time stamp 1704, although the user is not specified.
  • the shop ID 1703 stores the event at which shop.
  • data transmitted from the camera sensor unit 104 is stored, for example, as a UNIX (registered trademark) time stamp (in seconds). Also, these records are deleted or migrated to another inexpensive storage area when the time limit specified by the time limit 1705 is exceeded.
  • FIG. 18 is an example of shelf event information 1800 .
  • the shelf event information 1800 has information of each item shown in the item 1820 and has a value such as a sample value 1840 as a field name 1830 .
  • the shelf event information 1800 stores events such as weight changes that occur on product shelves based on weight information transmitted from the weight sensor unit 105 . It may also be called a weight information storage section or weight information storage means.
  • the shelf event information 1800 has items such as event ID, product basket ID 1801, previous weight 1802, store ID, time stamp 1803, type, weight 1804, weight change amount 1805, and the like.
  • a product basket ID 1801 stores a value corresponding to the product basket ID 2001 whose weight has changed and which is transmitted from the weight sensor unit 105 .
  • the values of the previous weight 2004 and the weight 2002 transmitted from the weight sensor unit 105 are stored in the previous weight 1802 and the weight 1804, respectively.
  • the previous weight 1802 may store the previous weight stored in the management server 101 instead of the transmitted value.
  • the weight change amount 1805 the value of the difference between the current weight 1804 and the previous weight 1802 is stored.
  • the time stamp 1803 stores the time when the event occurred. Although the time when the weight information processing module 1101 receives the weight information is stored, the time when the data aggregation transmitter 630 transmits the weight information may be used, the time when the microcomputer 710 transmits the weight information, or the time when the microcomputer 710 transmits the weight information. The time 710 obtained the weight value and the time the weight sensor device 608 obtained the weight value may be used.
  • FIG. 19 is an example of the estimate management information 1900.
  • the estimate management information 1900 has information of each item shown in item 1920, and has a value such as sample value 1940 as field name 1930.
  • FIG. The estimate management information 1900 stores a user ID 1901, a person ID 1902, a product ID 1903, the number of products 1904, and a time stamp 1905 specified by the estimate display module 1106.
  • FIG. In other words, it is the information that records the result of judgment as to who picked up what and how many from the product shelf.
  • FIG. 24 to 32 are examples of processing executed by each terminal of the merchandise management system 1 and each module of the management server 101.
  • FIG. FIG. 24 is an example of weight information accumulation processing flow 2400 .
  • the weight information processing module 1101 receives weight information transmitted from the weight sensor unit 105 (step 2410).
  • the weight information processing module 1101 acquires the time information of the received time (step 2420). It also acquires or calculates other information stored in the shelf event information 1800 .
  • the weight information processing module 1101 associates the weight information and other information with the acquired time information and stores them in the shelf event information 1800 (step 2430).
  • the time information is the time when the weight information processing module 1101 receives the weight information, the time when the data aggregation transmitter 630 transmits the weight information may be used, or the time when the microcomputer 710 transmits the weight information.
  • the time, the time when the microcomputer 710 acquires the weight value, or the time when the weight sensor device 608 acquires the weight value may be used.
  • FIG. 25 is an example of a location information accumulation processing flow 2500 .
  • the position information processing module 1102 receives the person ID, the time information, and the position information at that time transmitted from the camera sensor unit 104 (step 2510).
  • the position information processing module 1102 identifies the stay range in which the person indicated by the person ID stayed based on the position information (step 2520).
  • the range of stay is associated with the information of the divisions that have been divided in advance in the store or in the product shelf installation frame 2100, and specifies to which division it belongs. It also acquires or calculates other information in item 1720 shown in location tracking information 1700 .
  • the location information processing module 1102 associates the specified stay range information, person ID, time information, and other information and stores them in the location tracking information 1700 (step 2530).
  • the information of the predetermined section such as A1 and A2 of the sample value 1740 of the stay range 1701 is stored, but the coordinate information received as the position information may be stored.
  • a configuration may be employed in which trajectory information in which a plurality of pieces of position information are collected is stored.
  • FIG. 42 is an example of a skeleton detection processing flow 4200 .
  • the skeleton detection processing module 1102a detects feature points (for example, the head, right shoulder, left shoulder, right elbow, left elbow, right wrist, left wrist) of the upper body of the human body detected by applying the skeleton detection module on the captured image. ) is received (step 4210).
  • This feature point is obtained by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a.
  • estimated points of the hand are calculated (step 4220).
  • This type of estimated point is determined by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a.
  • the position coordinates of other parts of the human body such as the neck may be obtained together.
  • the information on the estimated points of the hand is accumulated together with the time information (step 4230).
  • the position information acquired by the position information processing module 1102 and the position information acquired by the skeleton detection processing module 1102a can be associated with each other in terms of position coordinates.
  • the skeleton detection camera sensor unit 104a may execute the skeleton detection processing flow 4200 and transmit the feature point information, the estimated hand position coordinates calculated from the feature point information, and the like to the management server 101. .
  • the flow of steps 4210, 4220, and 4230 may be executed on the management server 101 side based on the captured image.
  • FIG. 26 is an example of an overall processing flow 2600 for product purchase at a store.
  • the store entrance/exit management module 1103 executes the store entry processing 2700 (step 2610).
  • Information on the trajectory of the customer moving in the store is stored in the sequential position tracking information 1700 , and information on acquisition of the product from the product shelf is stored in the shelf event information 1800 .
  • the product determination module 1104 executes a product determination process 2800 that identifies the content and number of products picked up by the customer (step 2620).
  • the customer number determination module 1105 performs customer determination processing 2900 for identifying customers who were within reach of the product basket in which weight change occurred, based on the stored position tracking information 1700, skeleton detection information 1700a, and shelf event information 1800. (step 2630). At this time, based on the result of determining the correlation between the estimated hand point obtained by calculation and the position of the product shelf and the result of determining the change in the output of the sensor that detected the weight change, which customer preferably determines how many items have been picked up from the shelf.
  • the estimate display module 1106 compares the processing results of the store entry process 2700, the product determination process 2800, and the customer determination process 2900, calculates which user picked up which product and how many, and sends the estimate to the mobile terminal 102 of the user.
  • the estimate display process 3000 for displaying information is executed (step 2640).
  • the payment management module 1107 executes payment processing 3100 for requesting the payment processing server 107 to perform payment processing for the estimated product based on the receipt of the information indicating that the customer has left the store (step 2650).
  • FIG. 27 is an example of an entry processing flow 2700 .
  • the store entrance/exit management module 1103 receives the customer's user identification information from the store entrance management module 910 of the store entrance management terminal 106 (step 2710), and acquires the received time information as the store entrance time (step 2720). ).
  • the entering/exiting management module 1103 compares the received user identification information with the user identification information 1402 of the stored user master information 1400, and obtains the corresponding user ID 1401 to determine which user has entered the store. is identified (step 2730).
  • the entry/exit management module 1103 stores the specified user ID 1401 and entry time in the entry information 1111 (step 2740).
  • the store entry time is defined as the time when the entry/exit management module 1103 of the management server 101 receives the customer identification information. At the timing of reading the bar code displayed in the , or at the timing of acquiring information from the contactless IC chip by contactless communication, the store entry time is generated and transmitted to the management server 101 together with the customer identification information. .
  • FIG. 28 is an example of a product determination processing flow 2800.
  • the product determination module 1104 acquires information on the time when the weight change occurred (event occurrence time) from the shelf event information 1800 (step 2810).
  • the product determination module 1104 acquires the product basket ID in which the weight change occurred and the weight change amount from the shelf event information 1800 (step 2820).
  • the product determination module 1104 acquires information on the product ID 1502 associated with the product basket ID 1501 based on the segment information 1500 and product information corresponding to the product basket ID in which the weight change occurs from the product master information 1300. (Step 2830). The product determination module 1104 calculates the number of products picked up by the customer based on the amount of change in weight and the product weight 1304 stored in the product master information 1300 (step 2840). The product determination module 1104 outputs the acquired product information and the calculated product quantity information (step 2850). It is preferable to prevent deterioration of the determination accuracy by matching the determination result of the correlation between the estimated point of the hand obtained by the calculation and the position of the product shelf.
  • FIG. 29 is an example of customer determination processing flow 2900 .
  • the customer quantity determination module 1105 acquires the time when the event occurred from the shelf event information 1800 (step 2910).
  • the customer number determination module 1105 identifies the person ID of the person who stayed near the place where the event occurred from the shelf event information 1800, the store entry information 1111, and the location tracking information 1700 (step 2920).
  • the customer number determination module 1105 outputs the specified person ID (step 2930).
  • the customer quantity determination module 1105 acquires from the shelf event information 1800 the time stamp 1803 corresponding to the time when the weight changed.
  • the determination stay range 1603 corresponding to the product basket ID 1601 of the acquisition determination range information 1600 with the stay range 1701 corresponding to the person ID 1702 of the location tracking information 1700, the product basket whose weight has changed The person ID of the customer who stayed at the event occurrence time within the range where the product can be obtained is specified.
  • the customer number determination module 1105 acquires a time stamp corresponding to the store entry time from the store entry information 1111 . Next, by comparing the person ID of the store entry information 1111 with the location information (stay range 1701) and person ID 1702 of the location tracking information 1700, the number of customers who stayed near the entrance management terminal 106 at the time of entering the store is identified. Identify the person ID.
  • the weight sensor unit 105 transmits the weight information in two parts, the time when the weight acquired by the weight sensor device 608 changes and the time when the weight change stabilizes. It can be configured to store both information, or it can be configured to store only the weight information after the second weight is stabilized.
  • the shelf event information 1800 can also store the difference time between the first time when the weight change corresponding to the first weight information occurs and the second time when the weight change is stable. can specify the person ID at that time based on the time obtained by subtracting the difference time from the second time corresponding to the weight information of the second time, even if the configuration does not store the weight information of the first time. . That is, based on the second time when the change in weight is stable and the difference time, the number of products can be associated with the customer, and the number of products can be displayed on the mobile terminal 102 of the customer.
  • the person ID is specified in a broader sense.
  • the corresponding person ID is specified from the time stamp 1803 of the time when the weight change occurred and the location tracking information 1700 for a total of 3 seconds, 1 second before and 1 second after that. If the corresponding person ID cannot be specified here, the range is expanded to a total of 5 seconds including the time stamp 1803 and 2 seconds before and after, and it is confirmed whether the person ID can be specified. In this way, the determination time is gradually extended, and the process is repeated until the person can be identified.
  • the person with the highest probability is identified as the target. For example, the person closest to the product basket or the person who reached out to the product basket is specified as the person with the highest probability.
  • These pieces of information can be acquired from the customer's position information acquired by the camera sensor unit 104 and information on hand movements. Furthermore, these pieces of information are compared with the determination results of the interrelationship between the estimated hand points and the positions of the product shelves, which are obtained by calculation using the skeletal features extracted on the photographed screen from the camera device 506a. can also
  • a configuration may be adopted in which the determination time range used for identification is changed.
  • the judgment time range For example, the person ID of the customer staying in the determination stay range 1603 of the product basket for a total of 11 seconds, 5 seconds before and after the time when the weight change occurs, is specified.
  • the determination time range is shortened to perform processing with higher accuracy.
  • the person ID of the customer who stays in the determination stay range 1603 of the product basket within 1 second of the time when the weight change occurs is specified. Repeat the judgment with In this way, the determination time range is expanded step by step. In this way, the determination time range for identifying a person can be dynamically changed according to the product shelf setting frame 2100 and the number of customers staying in the store at the event occurrence time such as the time when the weight changes. It is possible to improve the speed and accuracy of judgment.
  • FIG. 30 is an example of an estimate display processing flow 3000.
  • the quotation display module 1106 acquires the processing results of the store entry processing 2700, product determination processing 2800, and customer determination processing 2900, and matches them (step 3010).
  • the estimate display module 1106 acquires and stores the user ID, product information, and product quantity information corresponding to the event occurrence time (step 3020). For example, the product information and product number information at the time when the weight change specified by the product determination process 2800 is associated with the person ID at the time when the weight change specified by the customer determination process 2900 occurs. Also, the user ID of the person who entered the store around the time when the person with the person ID identified by the customer determination process 2900 stayed near the entrance management terminal 106 is associated.
  • the estimate display module 1106 stores the associated user ID, person ID, product ID, quantity, and time stamp in the estimate management information 1900 .
  • the weight information transmitted from the weight sensor unit 105 installed on the product shelf and the first time information corresponding to this weight information are accumulated in the shelf event information 1800 in chronological order.
  • the customer's location information transmitted from the camera sensor unit 104 and the second time information corresponding to this location information are accumulated in the location tracking information 1700 in chronological order
  • the position information of the customer's skeleton transmitted from the camera sensor unit 104a and the third time information corresponding to this position information are accumulated in the skeleton detection information 1700a in chronological order
  • the fourth time information when the customer enters the store is accumulated in the store entry information 1111 in chronological order, By comparing the stored information based on the first time information, the second time information, the third time information, and the fourth time information, the number of products, the customer, and the user information are associated, and the user information is displayed.
  • Product quantity information can be displayed on the mobile terminal 102 of the customer identified by the information.
  • the shelf event information 1800, the position tracking information 1700, the skeleton detection information 1700a, and the store entry information 1111 may each or part of them be stored in different storage means, or all may be stored in the same storage means.
  • the estimate display module 1106 transmits and displays product information and product quantity information to the mobile terminal 102 of the customer corresponding to the user ID 1901 (step 3030).
  • the product purchase management module 310 of the mobile terminal 102 displays the received product information and product quantity information on the output device 305 such as a display.
  • the estimate display module 1106 calculates the total price of the products taken out of the product basket by the time the customer leaves the store, and displays it on the customer's mobile terminal 102 (step 3040).
  • the information is stored in the necessary check list. For example, when the detection of the amount of change in weight is dubious, when the detection of the position information from the camera device 506 and the calculation of the estimated point of the hand based on the captured image from the camera device 506a do not go well, or when the image is hidden. When an irregular event occurs such as a case where the determination cannot be made, the information is stored in a necessary check list for final human check. As a result, when the management server 101 cannot make the determination, a person can assist the user later, thereby realizing a highly accurate shopping experience. Since the payment process itself is executed after leaving the store, the user's customer experience is not impaired.
  • information (trajectory of movement) on the customer's position information acquired from the camera sensor unit 104 information on detection of the customer's skeleton (estimated hand points, etc.) acquired from the camera sensor unit 104a, and weight sensor unit 105
  • information on changes in weight obtained from the store entry management terminal 106 and the information on customers entering the store obtained from the store entrance management terminal 106 all the information about who, where, and when happened in the store. They are saved in the management server 101 on the cloud in chronological order, and can be reproduced in chronological order.
  • all sensors operate independently, and by accumulating each information in chronological order, it is possible to improve various analyzes and customer judgment systems. It is also easy to add sensors to estimate.
  • FIG. 31 is an example payment processing flow 3100 .
  • the payment management module 1107 receives information indicating that the customer has left the exit area for a certain period of time, for example, 10 seconds or more (step 3110). Whether or not the customer has left the exit area is determined by detecting whether or not the customer has left the predetermined coordinate range by the camera sensor unit 104 . In the example using the product shelf installation frame 2100 of FIGS. 21 and 23, the range surrounded by the coordinates (0, 0), (0, 2500), (2500, 2500), (2500, 0) is area. In a configuration that does not use the product shelf installation frame 2100, the coordinates of the exit area may be arbitrarily set inside or near the store.
  • the payment management module 1107 calculates the total price of all the products stored in the estimate display module 1106 (step 3120).
  • the payment management module 1107 transmits the user ID who picked up the product from the product basket and the total amount to the payment processing server 107, and the payment processing server 107 performs payment processing based on the payment processing information 1020 corresponding to the user ID. Execute (step 3130).
  • a refund can be executed from the product purchase management module 310 (for example, an application on a smartphone) of the mobile terminal 102 .
  • the payment process after leaving the store, payment is reserved by the payment processing server 107, and payment confirmation processing is performed after a certain period of time. I am losing it.
  • FIG. 32 is an example of signage display processing flow 3200 .
  • the signage processing module 1202 receives, from the IoT information processing module 1201, the first weight information at the timing when the weight value measured by the weight sensor unit 105 changes from a stable state to an unstable state (step 3210).
  • the weight information may be received from the data aggregation transmitter 630 instead of being distributed and received by the IoT information processing module 1201 .
  • the signage processing module 1202 acquires the product basket ID in which the weight change occurs from the weight information (step 3220).
  • the signage processing module 1202 acquires the product information such as the product name 1302 and the price 1303 of the product ID corresponding to the product basket ID and the signage information 1305 from the segment information 1500 and the product master information 1300 (step 3230).
  • the signage processing module 1202 transmits the acquired signage information to the signage terminal 103 and displays it (step 3240).
  • the signage information for example, information such as the product name, price, advertisement, etc. of the product ID corresponding to the product basket ID, information such as the product name, price, advertisement, etc. of the product associated with the corresponding product ID (for example, onion When you pick it up, information about curry roux, which is a dish using this onion) is remembered.
  • the signage information is transmitted by the signage processing module 1202
  • the signage information 420 stored in the signage terminal 103 is transmitted in response to the transmission of the information specifying the signage information such as the product ID from the signage processing module 1202. It may be configured to read out.
  • the signage terminal 103 is, for example, a tablet terminal, and may be configured to display signage information on its own display.
  • a single board computer such as a Raspberry Pi or a computer displays signage information on an external display. It may be configured to be displayed.
  • the signage processing module 1202 may be configured to transmit the signage information and information specifying the signage information such as product IDs in Push format. It may be configured to acquire information from the processing module 1202 in a pull format.
  • FIG. 33 is an example of a reference model 3300 of hardware.
  • a reference model 3300 of hardware usually, when trying to realize an unmanned store, it is necessary to individually set the positions and number of multiple camera sensors for each store according to the store structure and product layout, which requires labor, time, and cost. will be required.
  • a product shelf installation frame 2100 having a predetermined size is prepared, and a camera device 506 is installed at a predetermined position as shown in FIG.
  • a camera device 506 is installed at a predetermined position as shown in FIG.
  • the configuration of the walk-through (for indoor use) 3310 using the product shelf installation frame 2100 has been described, but it is also possible to adopt the configuration of the walk-through (for outdoor use) 3320 using a larger container. Also in this case, it is possible to install the camera device 506 at a predetermined position in the container in the same manner as the product shelf installation frame 2100, thereby simplifying various initial settings.
  • a predetermined model such as the configuration of such an unmanned shop and the number of camera devices 506 and 506a can be used as if selecting a cloud service.
  • An instance family 3301 represents the configuration of unmanned stores and mobile orders.
  • the instance type 3302 indicates a type indicating each configuration, such as IaaS (Infrastructure as a Service) in cloud services.
  • An image 3303 shows the assumed store format when installing each instance type.
  • a camera 3304 indicates the presence or absence of a camera.
  • the cameras may include camera device 506 and camera device 506a.
  • a multi-person correspondence 3305 indicates whether or not a plurality of customers can be discriminated. Indoors 3306 and outdoors 3307 indicate indoor stores and outdoor stores. The approximate size indicates the size of this merchandise management system 1 . For example, in a simple product management system 1 in which the multi-person correspondence 3305 is "None" and there is only one customer, only the weight sensor unit 105 is used, and the camera sensor unit 104 and/or the camera sensor unit 104a is not installed. , a space-saving and inexpensive system can be constructed.
  • the management server 101 accepts the selection of the instance type 3302 from the user, and after delivering necessary hardware such as the product shelf installation frame 2100, the camera device 506, and the camera device 506a from the warehouse or the like to the store in some cases, After installing these, the management server 101 transmits the initial setting information and the initial setting procedure to the camera management terminal 530, the camera management terminal 530a, and the data aggregation transmitter 630, and these terminals and devices execute them. , the initial settings for the camera sensor unit 104, the camera sensor unit 104a, and the weight sensor unit 105 are completed.
  • the control server 101 on the cloud is automatically sent to the management server 101 via WiFi (or LTE) modem.
  • WiFi or LTE
  • an ID and device information assigned to each device or device are transmitted to the management server 101 .
  • the management server 101 sends setup data and a setup program corresponding to each device and device based on the configuration specified by the selected instance type 3302 to each device and device specified by the transmitted ID and device information. etc. and run the setup procedure.
  • the management server 101 on the cloud can grasp the setting status and the operating status.
  • remote operation is possible in which the management server 101 performs operations such as rebooting and resetting of the camera sensor unit 104, the camera sensor unit 104a, and the weight sensor unit 105.
  • the management server sends information (each product basket ID) specifying a plurality of microcomputers 710 of weight sensors (or weight sensor devices 608) and setting information to a data aggregation transmitter 630.
  • the data aggregation transmitter 630 executes the setting of the plurality of microcomputers 710 in the management server.
  • the camera device 506a may be installed at a position different from that of the camera device 506. FIG.
  • the camera sensor unit 104 stores the product basket ID that identifies the microcomputer 710 of the weight sensor unit 105 in association with the coordinate information in the product shelf installation frame 2100 as shown in FIGS.
  • the product basket IDs on the product shelf AS01 are all in the range enclosed by the coordinates (350, 0) (350, 500) (800, 500) (800, 0) in AS01_A, AS01_B, and AS01_C. are associated.
  • FIG. 34 is an example of an order management screen 3400.
  • the estimate display module 1106 of the management server 101 manages the user's order history.
  • a history 3401 of picking up the product and a history 3402 of returning the product to the product basket are displayed. I understand. Also, there is a history 3403 of picking up caramel waffles, and it is displayed that the total amount 3404 is 162 yen.
  • FIG. 35 is an example of a product purchase screen. It is an example of a screen displayed on the mobile terminal 102 identified by the user ID associated with the customer who picks up the product from the product basket.
  • a screen 3500 is an example of a screen displayed when it is detected that two candies have been taken from the product basket. One piece of white chocolate and two pieces of candy are displayed for the product 3501 .
  • a screen 3550 is an example of a screen displayed after it is detected that one candy has been returned to the product basket.
  • a product 3502 displays one white chocolate and one candy.
  • the order in which the purchased items were placed in the basket is unknown.
  • in-store information such as which merchandise shelf was visited in front of which merchandise in what order, what items were picked up and returned, and as a result, what items were purchased in what order, etc. I keep track of everything in chronological order. As a result, it is possible to obtain more detailed behavior and purchase data, and for products that you picked up but did not buy or products that have been on the shelf for a long time, you can issue coupons that encourage you to purchase them again. , the history of interest can be displayed on the application.
  • the attribute information and behavior tracking in the store enable the customer to reach the front of the product shelf. When they arrive, it is possible to display signage information that matches their past purchase history, or to display a coupon that encourages upselling on the signage.
  • dynamic pricing and dynamic offers are performed, such as displaying the discount amount to which the coupon price is applied on the electronic price tag for each customer, and using the amount displayed by the quotation display module 1106 as the discount price. is also possible.
  • the camera sensor unit 104, the camera sensor unit 104a, and the weight sensor unit 105 can be used to realize a mechanism for easily managing sales of various merchandise.
  • setting information and setting procedures are sent from the management server 101 in a push format to the data aggregation transmitter 630 and the camera management terminal 530. And by transmitting to the camera management terminal 530a and performing initial setting, it is possible to easily introduce the merchandise management system 1 such as an unmanned terminal.
  • Example 2 of the merchandise management system 1 according to the present invention will be described. In order to avoid duplication of description, mainly differences from the first embodiment will be described.
  • Example 2 may be implemented in combination with Example 1, or may be implemented separately.
  • FIG. 36 the situation inside the store to which the second embodiment can be applied is shown divided into two (A) and (B).
  • the situation inside the same store is photographed from the same viewpoint at different times.
  • a plurality of product shelves are installed in the store. On each shelf, multiple items are displayed for sale. Between each product shelf is a passage where customers can freely walk around.
  • a store refers to a facility that provides goods or services for economic activities.
  • a store only needs to have at least a space for installing product shelves, and does not necessarily require a wall or ceiling surrounding it.
  • a store may be provided as a single section of a building, such as a shopping center, department store, outlet mall, or the like.
  • the store may be provided as an entire building, such as a small supermarket or a sole proprietorship store.
  • the store may be provided as an outdoor section without buildings.
  • Articles refer to goods that are traded as objects of economic activity within stores.
  • Articles are tangible objects such as food, beverages, stationery, and clothing, for example.
  • This tangible object also includes any kind of storage medium that temporarily stores data, programs, or the like.
  • This tangible object can also include any medium for service provision.
  • the product shelf refers to any structure for displaying products.
  • the product shelf may include bars for displaying products in one or more stages in the vertical direction.
  • the bar may have one end fixed to the main body of the product shelf and the other end serving as a free end to hook the product.
  • the product shelf may include plates for displaying products in one or more stages in the vertical direction.
  • the board has a substantially horizontal top surface and is fixed at one end to the body of the shelf and may carry one or more items on its top surface.
  • the product shelf may include product baskets for displaying products in one or more stages in the vertical direction (see FIG. 22, etc.).
  • a sensor for detecting or calculating the weight of the displayed product can be installed on each shelf of the product shelf.
  • a sensor that detects the weight of a product hooked on a bar on a product shelf is installed in association with the bar.
  • a sensor that detects the weight of the product placed on the plate of the product shelf or the product basket is installed in association with the plate or product basket. The sensor can quantify and output the weight of the detected bar or board or product basket, and transmit the value to the management server 101 .
  • the senor is a weight sensor.
  • the sensor is a load cell, an electromagnetic balance sensor, or the like.
  • the sensors can include sensors that are available today as well as sensors that are available in the future.
  • this sensor is not limited to directly detecting weight.
  • this sensor may detect a different amount of displacement than weight, and calculate the change in weight based on the amount of displacement.
  • each stage of the product shelf is provided with a weight sensor for directly detecting the weight of the product placed on the bar or plate of the product shelf or the product basket.
  • This weight sensor may be the weight sensor device 608 (see FIGS. 7 and 8) of the weight sensor unit 105 used in the first embodiment.
  • the weight sensor device 608 detects in advance the weight of the bar or plate on the product shelf or the weight of the product basket with nothing placed on it as an initial value, and transmits the initial value to the management server 101 .
  • the initial value is not limited to 0g.
  • the weight sensor device 608 periodically detects the weight of one or more items placed on each shelf (e.g., bar or plate or basket of items) at predetermined intervals to 101. Therefore, based on the detection value of the weight sensor device 608, the amount of change in the weight of each shelf on the product shelf is periodically transmitted to the management server 101 at predetermined intervals.
  • This information can be stored in any storage device associated with management server 101 .
  • two types of cameras are provided in the store to photograph the inside of the store.
  • One is a camera (first camera sensor) that functions as a camera that detects the customer's position in the store.
  • This camera may be the camera device 506 of the position detection camera sensor unit 104 of the first embodiment.
  • the camera device 506 periodically acquires the customer's movement trajectory (for example, walking trajectory) in the store at predetermined intervals and transmits it to the management server 101 .
  • One is a camera (second camera sensor) that captures the appearance of a customer in a store, applies a human skeleton model to the captured image, and extracts characteristic portions.
  • This camera may be the camera device 506a of the skeleton detection camera sensor unit 104a of the first embodiment.
  • the camera device 506 a periodically extracts feature points of customers in the store at predetermined intervals and transmits them to the management server 101 .
  • the skeleton detection module 510a of the camera sensor unit 104a may calculate estimated hand points and the like based on the feature points, and transmit the values to the management server 101.
  • the skeleton detection processing module 1102a of the management server 101 may calculate estimated hand points and the like based on the transmitted photographed image or feature points.
  • the position coordinates of camera device 506 and camera device 506a are known to management server 101 that receives these outputs. Therefore, the management server 101 can mutually register or coordinate-transform the position coordinates of the captured images.
  • positional coordinates can include both the position (x, y, z) in the three-dimensional space and rotation information ( ⁇ , ⁇ , ⁇ ) as parameters of the camera. . Therefore, based on the images captured by these cameras, the movement trajectory of each customer and the feature points based on the skeletal model of each customer are periodically transmitted to the management server 101 so that they are associated with each other (Fig. 11). These information can be stored in any storage device associated with management server 101 .
  • the management server 101 can collectively associate the person ID, chronological position information, and chronological skeleton information with each customer in the store.
  • the skeleton information includes the position information of the feature parts (head, neck, shoulder, elbow, wrist, etc.) obtained by applying the skeleton detection module on the captured image, and the position information obtained by calculation using this feature part. and the position information of the feature (such as a hand).
  • the output from the camera sensor unit 104 for position detection and the output from the camera sensor unit 104a for skeleton detection are separately transmitted to the management server 101, but in another embodiment, After integrating the output of the camera sensor unit 104 for position detection and the output of the camera sensor unit 104a for skeleton detection, the outputs may be combined and transmitted to the management server 101 .
  • the skeleton detection camera sensor unit 104a transmits information on the estimated position coordinates of the characteristic portion of the skeleton and the hand to the position detection camera sensor unit 104a. After that, the position detection camera sensor unit 104 may coordinate-transform these pieces of information, associate them with the person's coordinate information, and transmit these pieces of information to the management server 101 .
  • the position detection camera 104 (camera device 506) and the skeleton detection camera 104a (camera device 506a) are installed in the store as separate cameras (see FIG. 36A).
  • Each camera device 506, 506a may capture a subject as a still image (photograph) and acquire, record, or output the still image data.
  • each camera may capture a subject in motion picture format and acquire, record, or output still images from the motion picture format file. Since a moving image can be understood as a series of still images arranged in chronological order, the still images are electronically captured by each camera.
  • Each of the camera devices 506 and 506a has a difference in camera size (change in small size, large size, etc.), a difference in color information of still images captured by the camera (changes in bit number of RGB, full color, true color, etc.), Differences in camera shooting methods (regular shooting, infrared shooting, changes in depth cameras that acquire depth information, etc.), differences in camera installation positions (changes in the upper, side, or lower part of the store), etc. It can vary depending on the embodiment.
  • Each camera device 506, 506a has a POV (Point-Of-View) or "viewpoint”, and electronically photographs an object within the FOV (Field-Of-View) or "field of view” through the respective POV. go to Preferably, each camera device 506, 506a has a pre-fixed POV/FOV for each of the plurality of product shelves in order to optimally photograph the movement of people around each product shelf. However, each camera device 506, 506a can be configured with variable POV/FOV.
  • Each camera device 506, 506a may be installed so as to photograph a person from the front side at approximately the same height.
  • the skeleton detection model can be applied in a two-dimensional manner to the upper half of the body and the lower half of the human body on the photographing screen.
  • the places where cameras can be installed are limited.
  • the POV/FOV of each camera tends to be obstructed.
  • the detection accuracy is greatly affected.
  • Each camera device 506, 506a may be installed so as to photograph a person looking down from above.
  • a three-dimensional skeleton detection model is applied. Therefore, when detecting feature points in the upper and lower bodies of a person, the image analysis may become more difficult.
  • the POV/FOV of each camera above the person is less likely to be disturbed.
  • a wider range of camera installation locations can be secured. In particular, since it is possible to avoid people from overlapping each other on the captured image, detection accuracy can be improved.
  • camera devices 506 and 506a installed on the ceiling or above the store are used to photograph the inside of the store as if looking down (see (A) in FIG. 36).
  • a skeleton detection model is applied based on a photographed image, an image photographed from above and the skeleton coordinates (correct data) appearing in the image are given for learning.
  • transfer learning/fine tuning of deep learning can be used.
  • a device is added to apply the skeleton detection model as simply as possible. Thereby, the skeleton can be detected with relatively high accuracy based on the images captured by the camera devices 506 and 506a looking down from above.
  • each camera device 506, 506a is capable of photographing the customer from the side, below, or from other positions within the store.
  • a three-dimensional sensor or TOF (Time of Flight) sensor that measures the distance to the subject may be used as the position detection camera device 506, a three-dimensional sensor or TOF (Time of Flight) sensor that measures the distance to the subject may be used.
  • the three-dimensional sensor can determine the distance to the object based on the time it takes for the pulsed light to be projected and the pulsed light to be reflected by the object and returned.
  • distance information of a subject can be recorded for each pixel (picture element) or for each group of pixels on a photographing screen.
  • a depth camera or a three-dimensional camera which is a camera with a built-in depth sensor that acquires depth information
  • the 3D camera may be composed of an RGB camera and an infrared camera to obtain depth information in addition to color information, thereby enabling three-dimensional recognition.
  • part of the legs could be hidden in the image due to changes in posture such as crossing the front and back legs.
  • Invisible parts tend to cause discrepancies between the detection result and the correct answer, and there is a risk that learning will be difficult to progress.
  • the lower body tends to be hidden by the upper body, so it may be difficult to match the output quality of the upper and lower bodies. was there.
  • Patent Literature 2 describes that "the management server 10120 detects a person's hand by skeletal analysis performed on the image of the person included in the captured image (see paragraph 0079)".
  • the hands and fingers are relatively small and perform complex movements, prior learning processing is difficult, actual work takes a long time, and errors are likely to occur.
  • the wrist is the part that connects the arm of the human body and the palm of the hand, and is associated with a joint (radial carpal joint).
  • the hand at the tip of the wrist typically has five fingers (first to fifth fingers), and each finger is associated with multiple joints (first to fourth joints). It is Therefore, the wrist, hand and fingers perform complex movements to vary their positions. As a result, applying a skeleton detection module to captured images to track fine movements of the hands and fingers presents unique difficulties.
  • a control device for example, the skeleton detection module 510a of the skeleton detection camera sensor unit 104a or the skeleton detection processing module 1102a of the management server 101 displays a human skeleton model on the screen captured by the camera device 506a. Apply the skeleton detection module based on At this time, feature points are detected only for the customer's upper body. The customer's lower body is excluded from application of the skeleton detection module.
  • This embodiment utilizes the fact that the movement of the human body that contributes to product pick-up is limited to the upper body, and that the movement of the lower body can be substantially ignored.
  • the preliminary learning process is made easier, the actual work time is shortened, the occurrence of errors is suppressed, and the reliability of the output is improved. can enhance sexuality.
  • the control device displays the skeleton detection module on the screen captured by the camera device 506a.
  • the parts beyond the wrist (hands and fingers) of the human body are excluded from detection targets.
  • the position of the part beyond the wrist is calculated.
  • This embodiment utilizes the fact that the position of the hand can be estimated based on the skeletal structure of the human body if the characteristic points of a part of the upper body (eg, shoulder, elbow, wrist) can be acquired.
  • a depth camera or a three-dimensional camera that photographs the inside of the store from the ceiling photographs the user, and it is determined which product on which shelf the user picked up.
  • the position information of the hand and fingers that overlap the shelf is not used, and the position of the hand is calculated and obtained from the position coordinate information of any part of the wrist, elbow, or arm that does not overlap the shelf.
  • two positions of the elbow and wrist of one arm used for picking up the product in the customer's upper body can be detected. is detected, and other feature points such as hands and fingers are calculated based on their positions.
  • the skeleton detection module on the screen captured by the camera device 506a of the camera sensor unit 104a, feature points are detected at four positions of the left and right elbows and the left and right wrists of the customer's upper body.
  • other feature points such as hands and fingers are calculated based on these positions.
  • six positions of left and right shoulders, left and right elbows, and left and right wrists of the customer's upper body are detected by applying the skeleton detection module on the screen captured by the camera device 506a of the camera sensor unit 104a. is detected, and other feature points such as hands and fingers are calculated based on their positions. More preferably, by applying the skeleton detection module on the screen captured by the camera device 506a of the camera sensor unit 104a, the head, left and right shoulders, left and right elbows, and left and right wrists of the customer's upper body. Feature points are detected by limiting to a total of seven positions, and other feature points such as hands and fingers are calculated based on these positions.
  • the skeleton detection module on the screen captured by the camera device 506a of the camera sensor unit 104a, the head, neck, left and right shoulders, left and right elbows, left and right sides of the customer's upper body.
  • Feature points are detected by limiting to a total of eight positions on the wrist, and other feature points such as hands and fingers are calculated based on these positions.
  • characteristic points of the human body when applying the skeleton detection module, which has been applied in the prior art are illustrated.
  • 18 feature points are detected from the upper body and lower body of the human body.
  • the human body preferably includes at most the head (1), neck (2), left and right shoulders (3, 6), left and right elbows (4, 7), left and right wrists (5, By detecting the eight positions (thick line parts) in 8), the positions of other parts of the human body (hands, neck, etc.) can be calculated.
  • each feature point is based on pixel coordinates in the captured image.
  • the neck (2 ) is obtained by calculation.
  • the positions of the extracted features (head (1), left and right shoulders (3, 6), left and right elbows (4, 7), left and right wrists (5, 8)), left and right wrists are calculated.
  • the positions of the body such as the chest and back are excluded from the detection targets, but the front-back direction of the human body can be detected or calculated.
  • the positions of the arm, forearm, and upper arm can be calculated.
  • the "arm” refers to the portion of the human body from the shoulder (3, 6) to the wrist (5, 8).
  • the “forearm” refers to the part of the human body that is closer to the wrist (5, 8) when the arm is divided into two sections with the elbow (4, 7) as a boundary.
  • the "upper arm” refers to the part of the human body that is closer to the shoulders (3, 6) when the arm is divided into two sections with the elbows (4, 7) as a boundary.
  • “Forearm” and “upper arm” can be determined linearly based on the position coordinates in the three-dimensional space of the shoulder (3, 6), elbow (4, 7), and wrist (5, 8).
  • the "forearm” and “upper arm” are obtained as simple straight lines based on the skeletal structure of the human body.
  • a "forearm center point” can be obtained by calculation as the position of the center that bisects the length of the forearm from the elbows (4, 7) to the wrists (5, 8).
  • an "upper arm center point” as the position of the center that bisects the length of the upper arm from the elbow (4, 7) to the shoulder (3, 6).
  • the "forearm” and the "forearm center point” are calculated.
  • the length of the forearm from the elbow (4, 7) to the wrist (5, 8) is divided into three equal parts, and the one closer to the wrist (5, 8) is calculated you may ask.
  • the length of the forearm from the elbow (4, 7) to the wrist (5, 8) is divided into three equal parts, and the one closer to the shoulder (3, 6) is calculated as you may ask.
  • the length of the forearm from the elbow (4, 7) to the wrist (5, 8) may be divided into a plurality of sections, and any section may be calculated. .
  • neck (2) can be found as the part of the human body that connects the head and torso.
  • the position of the neck (2) may be calculated based on the positions of the left and right shoulders (3, 6). Since the length (shoulder width) of the left and right shoulders (3, 6) usually does not change greatly due to the movement of the human body, by obtaining the intermediate position coordinates in the three-dimensional space, the horizontal neck (2 ) can be determined.
  • the position of the neck (2) in the vertical direction can be determined based on the positions of the left and right shoulders (3, 6). For example, it may be the horizontal central position of the positions of the left and right shoulders (3, 6) and a position lifted upward by a predetermined distance. At this time, the position of the neck (2) may be determined using the position of the head (1) and the positions of the left and right shoulders (3, 6).
  • the head (1) may be obtained as the highest part of the human body and having a predetermined size.
  • the position of the head (1) may be determined in a unified manner regardless of changes in hair style, presence or absence of hats, and the like.
  • the position coordinates of the head (1) can be obtained relatively easily.
  • the estimated hand points can be calculated based on the positions of the elbows (4, 7), wrists (5, 8), etc., which are directly obtained by image analysis performed on the captured image. .
  • the estimated point of the hand changes depending on which position of the hand is used as a reference.
  • the estimated hand points are obtained based on the actual product purchasing process. For example, an estimated point of the hand may be obtained as the central position of the palm, focusing on the action of grasping the product. Alternatively, the estimated point of the hand may be obtained as the tip of the finger when the hand is opened, focusing on the maximum reachable range of the product. Alternatively, an estimated point of the hand may be determined at another position, focusing on the action of picking up the item.
  • the "estimated point of the hand” is the distance from the elbow (4, 7) to the wrist (5, 8) when a straight line is applied along the direction of the forearm, based on the position of the elbow (4, 7). It may be obtained as a position where the length of the forearm is extended by a predetermined multiple.
  • the customer By setting an "estimated point of the hand” at a position extending from the wrist by a predetermined multiple along the forearm, the customer actually reaches out toward the product shelf and tries to grab the product. Get the approximate center point of the hand (or the tip of the hand) at . Therefore, regardless of various movements of the wrist, fingers, etc., the reference position of the hand that is going to grasp the product is acquired in a unified manner.
  • the above-mentioned "predetermined multiple” is a value derived by trial and error through various tests by the applicant.
  • the estimated point of the hand is "4/3 ” is the double extended position.
  • the estimated hand point is "1.33 ” is the double extended position.
  • the estimated point of the hand is "1.3 ” is the double extended position.
  • the predetermined multiple can change depending on how the reference point of the hand is set. For example, the predetermined multiple can change depending on whether the center position of the palm is used as the reference or the tips of the fingers when the hand is opened. Also, the proportion of hands hidden when purchasing a product may vary depending on the size, shape, etc. of the product shelf in the store of the embodiment. Therefore, the "predetermined multiple" can be modified according to the embodiment.
  • the estimated point of the hand is “4/3" times, "1.33" times, or “1.3” times the length of the forearm from the elbow to the wrist along the direction of the forearm with reference to the elbow.
  • the double-extended position may be corrected within a range of about ⁇ 5%, about ⁇ 10%, about ⁇ 15%, about ⁇ 20%, or about ⁇ 25%.
  • the estimated hand point is obtained by extending the length from the elbow to the "forearm center point" by a predetermined multiple (e.g., 8/3 times, 2.67 times, 2.7 times, etc.). may be asked.
  • the estimated point of the hand is obtained by extending the length from the "forearm center point" to the wrist by a predetermined multiple (e.g., 8/3 times, 2.67 times, 2.7 times, etc.). may be asked. That is, for example, when the distance from the elbow to the center of the forearm is 1, the distance from the elbow to the wrist is 2, and the distance from the wrist to the hand is 0.67 (elbow to wrist/3), the distance from the elbow to the hand is 2.67.
  • the predetermined multiple and the percentage of its correction can be determined in the same manner as for extending the length of the forearm.
  • the estimated point of the hand may be obtained by extending the length from "arbitrary position of the arm" to the wrist by a predetermined multiple. Furthermore, the estimated point of the hand may be obtained as one point in the three-dimensional space, or as an arbitrary spread (width/size) in the three-dimensional space. For example, the estimated point of the hand may be determined as the spread from the fingertips of the hand to the center of the palm.
  • the present embodiment can determine a "hand estimate" regardless of the gender of the customer.
  • hand size tends to be proportional to height.
  • taller people tend to have longer forearms and larger hands.
  • shorter people tend to have shorter arms and smaller hands. This applies equally to the relationship between adults and children. Therefore, this embodiment can obtain the "estimated point of the hand" regardless of the height and age of the customer.
  • a device is added so that the position coordinates in the height direction related to the "estimated point of the hand" can be obtained with relatively high accuracy. Therefore, in this embodiment, since the "estimated hand points" are obtained based on the feature points detected only on the upper half of the human body, even for customers who come to the store in wheelchairs, other customers who normally walk Similarly, we can find the "estimated point of the hand”. Therefore, this embodiment can determine the customer's "estimated hand point" without being affected by the difference between a healthy person who walks normally and a disabled person who moves in a wheelchair.
  • the "estimated hand point” can be obtained regardless of the state of the lower half of the human body. Therefore, this embodiment can determine the customer's "estimated hand point” without being affected by the difference between an able-bodied person who walks normally and a disabled person who uses crutches. Therefore, this embodiment can obtain the customer's "estimated hand point” uniformly under substantially most circumstances.
  • FIGS. 38A and 38B an application example of the skeleton detection model for the captured image is shown.
  • an application example of a skeleton detection model for each person is shown based on an image of a person below captured by a camera (for example, the camera device 506a of the camera sensor unit 104a) installed near the ceiling of the store. ing.
  • a camera for example, the camera device 506a of the camera sensor unit 104a
  • a plurality of feature points for example, 18 points
  • the detection accuracy is greatly disturbed at the feet. Also, it is found that detection is difficult for small parts of the hand and fingers beyond the wrist.
  • a plurality of feature points (e.g., eight) are detected in a more limited number limited to the upper body. (See (B) of FIG. 37).
  • a decrease in the accuracy of application of the skeleton detection model is avoided.
  • positions that cannot be obtained simply by applying the skeleton detection model can also be obtained.
  • FIG. 39 an example of a skeleton detection model applied to a captured image is shown.
  • a normal skeleton detection model when applied, there are cases where the wrist and beyond are not output. In the direction of the forearm, if only the position of the wrist is detected, the distance to the product is too large.
  • detection errors and setting errors are likely to occur when determining whether to pick up a product based on the position of the wrist. As a result, there is a high risk of making an incorrect pick-up determination with other nearby customers. In this embodiment, since the estimated points of the hand are obtained by computation, such risk is minimized.
  • a device is added so that the position coordinates in the height direction related to the "estimated point of the hand" can be obtained with relatively high accuracy.
  • FIG. 40A a case is illustrated in which the part beyond the wrist is hidden by the top plate of the product shelf when the skeleton detection model is applied. If an attempt is made to detect the position of the wrist in this state, the wrist may be erroneously detected on a part of the product shelf (top plate, etc.) as illustrated in FIG. 40(B). As a result, when an attempt is made to obtain position coordinates in the height direction (Z-axis direction) at pixel positions on the image in order to obtain the position coordinates of the wrist, a deviation may occur in the detection target.
  • the Z-axis coordinate is not the positional coordinate of the wrist, but the part of the product shelf (top board, etc.) Location information may be obtained incorrectly. In that case, there is a possibility that the position coordinates of the wrist cannot be obtained accurately.
  • the position between the elbow and the wrist can be used instead of the position of the wrist when acquiring the position information of the hand or wrist.
  • a "forearm center point" may be obtained as a center position that bisects the length of the forearm from the elbow to the wrist, and the depth may be acquired at that position.
  • the wrist and part of the forearm may be hidden when trying to pick up the product on the shelf.
  • the "forearm center point" is rarely hidden in the product shelf. Therefore, in this embodiment, for example, when a customer approaches a product shelf, the position of the "forearm center point" may be obtained as wrist position information (especially depth).
  • the position at which the depth is acquired may change based on the size of the plate on each shelf of the product shelf. Therefore, in this embodiment, the position of the wrist in the Z-axis direction is not limited to the position of the "forearm center point".
  • the length of the forearm from the elbow to the wrist may be divided into thirds, the position closer to the wrist or the side closer to the shoulder may be obtained, and the position in the Z-axis direction may be acquired at that position.
  • an arbitrary position may be obtained within the length of the forearm from the elbow to the wrist, and the position in the Z-axis direction may be acquired at that position.
  • the position information of the "forearm center point” is acquired, and the position information of other parts of the upper body, such as the shoulder, the upper arm center point, and the elbow, is acquired, and these values are obtained. They may be used in combination. Therefore, in this embodiment, when a hand is put into an arbitrary stage of the product shelf, even if the tip of the wrist that is trying to grasp the product is hidden by the top board or the like above, the Z axis Directional position can be determined relatively accurately.
  • the process of determining the estimated hand points in this example is illustrated.
  • This step can be performed by the skeleton detection module 510a of the skeleton detection camera sensor unit 104a (see FIG. 5B).
  • the skeleton detection module 510a receives an image captured by the camera device 506a as an input (step S1).
  • This captured image is, for example, an RGB image.
  • the skeleton detection module 510a obtains the coordinates (X, Y coordinates) of the skeleton for a predetermined part based on the photographed image (step S2). For example, the (X, Y coordinates) of the elbow and wrist are obtained. Next, the skeleton detection module 510a calculates (X, Y coordinates) of the "forearm center point” based on the (X, Y coordinates) of the elbow and wrist (step S3). Next, the skeleton detection module 510a acquires the "forearm center point" and/or the depth (Z coordinate, etc.) at the elbow position (step S4).
  • the skeleton detection module 510a extends the length of the forearm by a predetermined multiple with the elbow as a reference (step S5). For example, the skeleton detection module 510a determines the length from the elbow to the "forearm midpoint" (or from the "forearm midpoint” to the wrist) and extends that length by 2.67. As a result, the skeleton detection module 510a calculates the coordinates (X, Y coordinates) of the "estimated hand point”. Next, the skeleton detection module 510a acquires the Z coordinate (depth) of the elbow and the "forearm center point” (step S6), and similarly multiplies this by 2.67 to obtain the Z coordinate of the "estimated hand point". Calculate the coordinates. In this way, the skeleton detection module 510a obtains the positional coordinates (X, Y, Z coordinates) in the three-dimensional space of the "estimated hand point" from the coordinates of the elbow and the "forearm center point”.
  • a skeleton detection model is actually applied to the upper half of the human body based on the image captured by the camera device 506a of the camera sensor unit 104a for skeleton detection installed in the store.
  • the skeleton detection module 510a can detect eight feature points of the head, neck, both shoulders, both elbows, and both wrists.
  • Various learning operations are performed in advance for the skeleton detection model to be applied on the captured image.
  • easy-to-detect feature points are based on the movement of people in the store (walking, etc.), the actual product purchase process, etc.
  • the motion of the human body is considered.
  • the human body can perform various actions such as bending the waist, bending the body, moving the arms up and down, moving the arms back and forth, and crossing the arms.
  • learning work is performed by concentrating on the expected movements of people in the store, thereby facilitating the operation.
  • learning is mainly based on normal walking postures in a store, postures with hunched backs, etc., and special motions such as somersaults are excluded from the learning content in advance.
  • the skeleton detection module 510a of the camera management terminal 530a extracts still images based on characteristic shapes, figures, contours, lines, vertices, colors, etc., particularly for the head, shoulders, elbows, and wrists of the human body. image analysis may be performed. At this time, the skeleton detection module 510a may classify the received still image by pixel or pixel set. Characteristic targets relating to the head, shoulders, elbows, and wrists may be identified by identifying shapes, figures, contours, lines, vertices, colors, etc. for each pixel or each pixel set.
  • the skeleton detection module 510a preferably undergoes image analysis training for various images in advance by machine learning using AI.
  • image analysis we actually perform image analysis, and perform training to identify and extract targets (shapes, figures, contours, lines, vertices, colors, etc.) from images. Training is performed on, for example, hundreds, thousands, or more images of different targets, accumulating various results. Statistical data may then be calculated and generated for the results. After a high identification rate, for example, about 99% or about 99.9% or more, is obtained through training, image processing is actually performed to identify the target from the image. good too.
  • Various image information may be used as training data in the training performed by machine learning using AI.
  • machine learning may be used to generate a judgment model whose input is a still image and whose output is information about the shape, figure, contour, line, vertex, color, and the like of a characteristic object.
  • the training may take into account what the store looks like at different times of the day. For example, it is possible to learn the difference in how shadows appear in the daytime when light shines into the store and in the middle of the night when only the lights in the store are used. In addition, it may be possible to learn about various in-store situations, such as when there are no people in the store, when there are only a few people in the store, and when there are many people in the store.
  • a characteristic target may be obtained as an output.
  • the skeleton detection module 510a may create a confidence level for each feature point based on data accumulated during prior learning work. For example, when the detected position matches the pre-learned position at almost the same rate (approximately 100% or approximately 99%), highly reliable information may be given. Also, information with lower reliability may be given as the deviation of the detected position becomes larger compared to the position learned in advance.
  • the position pixel coordinates in the image
  • the reliability (0 A value from 1 to 1, a value from 1 to 10, etc., where the higher the value, the higher the accuracy of the position is expected to be displayed).
  • the latter may be displayed by changing the color of the position of the joint in the image. For example, green may indicate high confidence, yellow may indicate medium confidence, and orange may indicate low confidence.
  • the camera device 506a of the skeleton detection camera sensor unit 104a installed on or above the ceiling of the store is used to photograph the customer looking down, thereby minimizing the obstruction between the subject and the camera. (See (A) of FIG. 36).
  • a part of the customer's body such as arms and shoulders may not be captured completely depending on the customer's standing position, how the customer overlaps with other customers, and the like.
  • a plurality of camera devices 506a are installed so that each product shelf installed in the store can be photographed optimally (see FIG. 36A). Therefore, it is possible that at least two camera devices 506a capture images of customers near the same product shelf almost simultaneously from different POV/FOV. Therefore, if the image taken from one camera device 506a alone cannot sufficiently detect the predetermined skeleton of the upper body of a certain customer, the skeleton detection module 510a uses the image taken from another camera device 506a to , the missing part may be compensated for.
  • FIGS. 43A and 43B captured images of the same customer in the store captured from POV/FOV of two different camera devices 506a are illustrated.
  • (A) of FIG. 43 the appearance of the customer facing the product shelf is photographed from the front side, and the appearance of one arm being lowered is shown.
  • (B) of FIG. 43 the customer facing the product shelf is photographed from the side, and one arm is lowered and the other arm is bent.
  • (A) in FIG. 42 in which the customer facing the product shelf is photographed from the front, is preferable. It can happen. In that case, an image captured by another camera device 506a that captures the same customer almost simultaneously may be used as an auxiliary.
  • each camera device 506a was able to photograph the same person at approximately the same time.
  • the position coordinates of each camera device 506a are known to the camera management terminal 530a that receives each captured image, and the position coordinates on each captured image can be mutually registered or coordinate-transformed.
  • the skeleton detection module 510a uses the features of the skeleton of the person photographed in FIG. 43(B). At that time, the positional coordinates of the characteristic portion of FIG. 43B are converted into the positional coordinates of FIG. positions may be superimposed.
  • FIGS. 44A and 44B an example is shown in which the positions of the skeleton detected based on the images captured by the two camera devices 506a are summarized on the same coordinates. For example, when each joint detected from each image can be superimposed on each other's position coordinates, each is linked to the same joint.
  • the skeleton detection module 510a groups the position coordinates of the joints detected from each camera device 506a, especially based on the coordinates of the head, of the other joints. do.
  • the skeleton detection module 510a then superimposes the coordinates of the head, neck, both shoulders, both elbows, and both wrists detected from each camera device 506a, and associates the position coordinates of the joints detected from each camera with each other. .
  • the head position coordinates obtained from the two camera devices 506a overlap each other within a predetermined threshold, for example, within the range of 100% to 99%, or within the range of 100% to 95%, Alternatively, when an overlap within the range of 100% to 90% is obtained, both are tied to each other. The same is true for the neck, shoulders, elbows, wrists, and so on.
  • FIG. 44B illustrates a case where the skeleton detection module 510a aggregates the position coordinates of each feature acquired from the two camera devices 506a into one position coordinate.
  • the skeleton detection module 510a predetermines the order of priority when combining the two position coordinates in the three-dimensional space into one. For example, a feature based on a still image taken from the front of a product shelf is prioritized over a feature based on a still image taken at an angle (e.g., from the side) to the product shelf.
  • the skeleton detection module 510a can supplement the missing part with the photographed part.
  • the skeleton detection module 510a applies the skeleton detection module based on a captured image obtained from a camera device 506a to determine the customer's upper body. , and the "estimated point of the hand" may be calculated.
  • skeleton detection module 510a applies skeleton detection module based on a combination of two or more captured images obtained from two or more camera devices 506a. By doing so, the features of the customer's upper body may be detected and the "estimated point of the hand" may be calculated. Therefore, in this embodiment, it is possible to acquire the "estimated point of the hand" that serves as a reference when the customer picks up the product under various circumstances.
  • pick-up means that one or a plurality of products displayed on a product shelf in a store are picked up by a customer.
  • the shape of the hand, the position of the fingers, and the like are arbitrary, and the direction, height, and the like in which the object is picked up are arbitrary.
  • the position coordinates of the “estimated hand point” acquired by the skeleton detection module 510 a of the camera management terminal 530 a are transmitted to the management server 101 .
  • the information may be transmitted to the management server 101 after being integrated with the information acquired by the position detection module 510 of the camera management terminal 530 .
  • the management server 101 compares the already known position of the product shelf (position of the product) with the position of the "estimated point of the hand” in the three-dimensional space. can be determined.
  • the determination is made based on the degree of proximity of both three-dimensional coordinates within the same three-dimensional space. For example, the range of 100% to 90%, the range of 100% to 95%, or the range of 100% to 99% with respect to the position of the product shelf (product position) If it is determined that the positions of the "estimated points" overlap, it can be determined that the hand is near the item on the shelf. Or, for example, for a pre-determined threshold associated with each item on a shelf, if an "estimated hand point" is detected above that threshold, determine that a hand is near that item. be able to. In these cases, the customer quantity determination module 1105 of the management server 101 can estimate/assume the occurrence of an event in which a product is picked up.
  • the output of the weight sensor device 608 associated with the product shelf is sent to the customer quantity determination module 1105 of the management server 101 . Therefore, the management server 101 can further combine and use the output values from the weight sensor device 608 to determine with high accuracy whether or not an event in which the product is actually picked up has occurred.
  • each stage of the product shelf incorporates a weight sensor device 608 that detects the weight of the product placed on the respective bar or plate or product basket (FIGS. 7 and 8). 8 etc.).
  • the customer number determination module 1105 of the management server 101 periodically receives the output of the weight sensor device 608 at predetermined intervals. Generally, the weight of the item itself does not change over time. Therefore, as long as the weight sensor device 608 operates according to specifications, the customer quantity determination module 1105 of the management server 101 can constantly track whether or not there is any product on the product shelf.
  • the customer number determination module 1105 of the management server 101 converts the weight data periodically (through loop processing) obtained by the weight sensor device 608 on each stage of the product shelf into "weight time-series data (part of weight information). )”.
  • the weight sensor may acquire weight data for each stage of the product shelf at a cycle of 100 ms.
  • the camera devices 506 and 506a are provided near the product shelf, and the surroundings of the product shelf can be photographed periodically (by loop processing).
  • the period in which the weight sensor device 608 detects the weight and the period in which the camera devices 506 and 506a take images need not match, but should be short enough to sequentially track the movement of people in the store. and
  • the camera device 506a may always take an image of a person near the product shelf periodically at predetermined intervals. Alternatively, when the camera device 506a detects that a person approaches the product shelf, the output of the camera device 506a may be used as a trigger to capture an image of the person. Alternatively, when the camera device 506a detects that a person has approached the product shelf and has extended his or her arm toward the product shelf, the output of the camera device 506a can be used as a trigger to capture an image of the person. good.
  • the camera management terminal 530a or the management server 101 receives the input of the captured image transmitted from the camera device 506a as a trigger, and the position coordinates (X, Y , Z coordinate) may be calculated.
  • the processing (flow) of customer number determination performed by the customer number determination module 1105 of the management server 101 has a flow consisting of the following steps.
  • the customer number determination module 1105 determines in advance the distance between the average position of the customer's hand and the product segment or product on the product shelf (or a threshold for determining that the hand is close to the product). Measure or calculate and store the value (preprocessing).
  • the customer number determination module 1105 receives captured images tracking the positions of each customer in the store by the camera device 506 of the camera sensor unit 104 for position detection.
  • the customer number determination module 1105 receives captured images of each customer in the store by the camera device 506a of the skeleton detection camera sensor unit 104a.
  • the customer number determination module 1105 applies the skeleton detection processing module to the captured image from the camera device 506a, and the position of the estimated point of the hand calculated based on the predetermined characteristic part of the upper body of the detected customer. Get coordinates.
  • the customer number determination module 1105 may combine the output from the position detection camera-sensor unit 104 and the output from the skeleton detection camera-sensor unit 104a in one three-dimensional space. These pieces of information are aggregated into one person ID for each customer and accumulated in chronological order.
  • the customer population determination module 1105 may utilize the calibration information to transform each customer's hand position (in the skeleton detection information) from pixel coordinates in the image to coordinates in the position tracking space. .
  • the customer count determination module 1105 may match the location of each customer in the location tracking information with the location of each customer in the skeleton detection information to determine the person ID of the customer in the skeleton detection information.
  • the customer quantity determination module 1105 knows the position coordinates of each product shelf in the store. Therefore, it is possible to determine which of the customers in the store is reaching out to which product shelf. For example, the management server 101 calculates the distance between the estimated hand point and the product segment or product position (hereinafter, hand segment distance). Furthermore, the customer quantity determination module 1105 receives time-series weight data for each product shelf on which products are displayed, and continuously analyzes time-series changes in product weight.
  • the output of the skeleton detection camera sensor unit 104a, the output of the position detection camera sensor unit 104, and the output of the weight sensor unit 105 are transmitted to the customer number determination module 1105 of the management server 101. Examples are given.
  • the skeleton detection camera sensor unit 104a acquires an RGB image and a depth image (depth image) by photographing the inside of the store. These images can be processed by the skeleton detection module 510a at the edge device.
  • step X1 by applying the skeleton detection model to the RGB image, predetermined characteristic portions (head, neck, both shoulders, both elbows, and both wrists) are detected based on the human skeleton model. X, Y coordinates of all or any part thereof, and so on) can be obtained.
  • the depth image can be associated with the RGB image.
  • the X, Y coordinates and the depth information (Z coordinate) of the predetermined characteristic portion can be integrated.
  • step X4 by applying coordinate transformation distortion correction to this result, the distortion of the X and Y coordinates on the RGB image can be corrected.
  • X, Y coordinates and depth information (Z coordinate) of a predetermined characteristic portion can be acquired as camera coordinates.
  • step X5 the coordinates of the estimated position of the hand can be obtained by performing a predetermined calculation based on this position information. Therefore, finally, the X, Y, Z camera coordinates of the predetermined feature of the skeleton and the estimated point of the hand can be obtained.
  • the position detection camera sensor unit 104 can acquire an ID and X, Y, and Z coordinates as ToF coordinates for a person by photographing the interior of the store, as shown in step Y1. This image is processed by the position detection module 510 on the edge device side such as a PC. On the other hand, in step Y2, the position detection camera sensor unit 104 receives the output from the skeleton detection camera sensor unit 104a, and calculates the X, Y, and Z camera coordinates of the predetermined feature of the skeleton and the estimated point of the hand. can be obtained.
  • step Y3 by applying coordinate transformation correction to this result, the camera coordinates can be associated with the ToF coordinates. Furthermore, in step Y4, by combining the outputs of steps Y1 and Y3 in the same space, the skeleton and the person can be associated. Therefore, finally, the ID of the person (customer), the X, Y, and Z coordinates as the ToF coordinates of the estimated points of the head and hands can be obtained.
  • the weight sensor device 608 of the weight sensor unit 105 acquires weight data in chronological order. As shown in step Z1, this data is processed by module 610 on the edge device side, such as the Raspberry Pi®.
  • the customer number determination module 1105 of the management server 101 estimates the positions of products on product shelves in the store, which are obtained in advance, and the ID, head, and hand of a person (customer) obtained from the camera sensor unit 104 for position detection. It receives the X, Y, Z coordinates as the ToF coordinates of the point and the time series weight data obtained from the weight sensor device 608 . Therefore, by using these data in combination, it is possible to determine whether the customer's merchandise is picked up.
  • the customer number determination module 1105 can determine how many items were picked up at what time and by whom when a change occurs in which items are continuously picked up within a short period of time. can be done. Thus, in this embodiment, when multiple customers pick up one or more items that are close to each other at or near the same time, it is possible to determine which customer picked up which item and how many. Determination of product pick-up will be described in more detail below using the output of the weight sensor device 608 and the estimated points on the hand.
  • FIG. 45 changes in the output of the weight sensor device 608 and changes in the distance between the estimated point of the customer's hand and the shelf, which are sent to the customer quantity determination module 1105, are illustrated superimposed. .
  • an example of weight time-series data of each product shelf is shown in a graph.
  • the horizontal axis of this graph is time (ms) and the vertical axis is the value (g) recorded by the weight sensor device 608 .
  • the customer quantity determination module 1105 can infer that the products on the product shelf have not changed. A drop in the output of weight sensor device 608 is recorded from t2 to t4 (ms). Ultimately, a decrease in the output of the 50 g weight sensor device 608 is recorded during this period. Therefore, the customer quantity determination module 1105 can infer that one or a plurality of commodities corresponding to the weight of this difference (50 g) have been picked up.
  • the customer quantity determination module 1105 can infer that the products on the product shelf have not changed.
  • the output of the weight sensor device 608 is simplified and shown in a straight line. In practice, when a product hooked on a bar on a product shelf is picked up, a load is applied to the bar, so the output of the weight sensor does not become linear. The same is true when products are placed on the boards of product shelves or product baskets.
  • the customer number determination module 1105 performs determination by combining each hand segment distance of user 1 and user 2 and the time of weight time series data.
  • the estimated point of the hand of user 1 was away from the product shelf until around time t1, but the estimated point of the hand was closer to the product shelf from around time t1 to around time t5.
  • the time t1 is when the distance between the hand and the shelf becomes below (or above) a predetermined threshold after the user 1 begins to extend his/her hand toward the product shelf, and the hand approaches the product segment. is the time when it is determined that
  • time t5 after the hand has started to move away from the product shelf from the state in which the hand was brought close to the product segment, the distance between the hand and the shelf exceeds a predetermined threshold, and the hand moves away from the product segment. is the time when it is determined that the In the graph, the outputs of the estimated points of the hand are simplified and shown in a straight line. It should be understood that in practice the trajectory will not be straight.
  • the estimated point of the hand was away from the product shelf until around time t3, but the estimated point of the hand was closer to the product shelf from around time t3 to around time t6. Recognize. Note that at time t3, after User 1 begins to extend his/her hand toward the product shelf, the distance between the hand and the shelf falls below (or exceeds) a predetermined threshold, and the hand approaches the product segment. is the time when it is determined that
  • the distance between the hand and the shelf exceeds a predetermined threshold, and the hand moves away from the product segment. is the time when it is determined that the The difference in the size of the vertical axis on the graphs of User 1 and User 2 is based on the difference in the distance between each customer and the product shelf.
  • the threshold can be set as an arbitrary value in consideration of the configuration of the product shelf, the size of the product, and the like.
  • the customer number determination module 1105 finds that the output of the weight sensor device 608 changes between times t2 and t4. From this, the customer number determination module 1105 can infer that a product pick-up event has occurred especially during this period. However, the customer quantity determination module 1105 can understand that there were an estimated point on the hand of User 1 and an estimated point on the hand of User 2 near the item during this period. Therefore, the customer number determination module 1105 performs processing for determining whether the item was picked up by either user 1 or user 2 .
  • the customer quantity determination module 1105 performs calculations for determining who has picked up which product and how many.
  • the customer number determination module 1105 divides the period of weight change (from time t2 to time t4) for each user, calculates the weight difference between both ends of each time interval, and calculates the weight taken by the user. tied as That is, the customer number determination module 1105 estimates the stretched section for each user and acquires the weight change at both ends thereof. For example, the customer number determination module 1105 acquires time t2 and time t4 as the period during which the weight changed.
  • time t1 and time t5 are acquired as a period during which the hand movement that enables picking up of the product occurred (hand stretching section).
  • time t3 and time t6 are acquired as a period during which the hand has moved to enable the product to be picked up (stretching period).
  • the management server 101 divides the period (from time t2 to time t4) so that these pieces of information are collected in chronological order, compared, and the movement of each user is made clearer.
  • the customer number determination module 1105 determines that during the period from time t1 to time t3, the movement of the hand of the user 1 is the main problem, and the movement of the hand of the user 2 is not the problem. Further, the customer number determination module 1105 determines that the movement of the hands of the users 1 and 2 is the main problem during the period from the time t3 to the time t4. Therefore, the customer number determination module 1105 divides the period (from t2 to t4) in which the weight changes into the first section (from t2 to t3) and the second section based on each user's hand movement. (from t3 to t4).
  • the customer number determination module 1105 presumes that during the first period (from t2 to t3), the movement of the hand of the user 1 mainly contributes to the weight change. In addition, the customer number determination module 1105 presumes that during the second period (from t3 to t4), the movements of the hands of the users 1 and 2 mainly contribute to the weight change.
  • the customer number determination module 1105 also divides the weight corresponding to the division of the period (from t2 to t4). That is, the customer quantity determination module 1105 assumes that the change in the weight of the product (50 g in total) has occurred linearly, divides this weight change into a first section and a second section, Calculate proportionally. For example, the customer number determination module 1105 calculates that the weight of the first section has changed by 35 g when the total weight is 50 g. Furthermore, this change in weight (35 g) is associated only with User 1. Further, the customer number determination module 1105 calculates that the weight of the second section has changed by 15 g when the total weight is 50 g. Furthermore, this change in weight (15 g) is linked to both User 1 and User 2 .
  • the customer quantity determination module 1105 totals the weights that can be assumed for each user, compares them with the weights of products (registered in advance), and converts them into the number of products. For example, the customer number determination module 1105 associates only the user 1 with the weight sensor change (35 g) during the first period (from t2 to t3). For this period, the management server 101 does not need to distinguish between user 1 and user 2 . However, in the second period (from t3 to t4), the weight sensor change (15 g) is associated with User 1 and User 2. For this period, the customer quantity determination module 1105 needs to consider whether the weight is attributed to User 1 or User 2 on a case-by-case basis.
  • the customer quantity determination module 1105 predetermines an allowable error per product. For example, for a 50g product, an error of ⁇ 0.2 pieces (10g) is permissible.
  • the customer quantity determination module 1105 determines whether each hypothesis 1 and 2 is within the permissible error range for each product. Assumption 1 can be determined to be within tolerance for both users 1 and 2 . For hypothesis 2, it can be determined to be outside the tolerance for both users 1 and 2. Therefore, the customer quantity determination module 1105 can finally determine that User 1 has picked up 50 g of product per item, and User 2 has not picked up any product.
  • the allowable error used in the determination by the customer number determination module 1105 can be set variously. For example, if the weight change is 100g and it is within the tolerance of 2 items (50 x 2 ⁇ 10 x 2) (2.0 items), the number of items picked up is determined as 2 items. good too. Also, if the weight change is 105g and it is within the tolerance of 2 items (50 x 2 ⁇ 10 x 2) (2.1 items), the number of items picked up is determined as 2 items. good too. In addition, if the weight change is 125g and it is not within the tolerance of 2 items (50 x 2 ⁇ 10 x 2) (2.5 items), the number of items picked up should not be determined as 2 items. , may be indeterminate. Also, if the weight change is 160g and it is within the tolerance of 3 items (50 x 3 ⁇ 10 x 3) (3.2 items), the number of items picked up is determined as 3 items. good too.
  • more subdivided control can be applied to the conversion processing of the number of weights.
  • the larger the width of the weight change the smaller the error. Therefore, in this embodiment, when the weight change from the weight sensor device 608 is large, the allowable error may be relatively small.
  • the weight per product is 100 g
  • the change in each value from 1 to 9 is illustrated when the weight change is negative (the product is taken). ing. This value is used in the determination processing of the customer number determination module 1105 described above.
  • the customer number determination module 1105 can finely set numerical values that serve as criteria for determination at the stage of determining how many items have been picked up according to the weight (number) of the items to be picked up. This makes it possible to improve the accuracy of determination as the number of objects picked up increases based on the characteristics of the weight sensor.
  • the number of products to be picked up is limited to within the range of one to nine. If this range is exceeded, a decision may be made by a judge or the like. Alternatively, the ranges shown in FIG. 46 may be extended to accommodate a greater number of ranges.
  • the customer number determination module 1105 avoids overlapping (mixing) of results by preferentially applying the determination from the smaller number. For example, in this case, the number of seven is preferentially applied.
  • the customer number determination module 1105 acquires the stretching section for each user. For example, for user 1, the section between time t1 and time t5 is acquired as the section in which the hand is extended toward the product. As for User 2, the section between time t3 and time t6 is acquired as the section in which the hand is extended toward the product.
  • the hand stretching section linked to each user is classified. For example, section 1 between time t1 and time t3 can be associated only with user 1 .
  • section 2 between time t3 and time t5 can be redundantly linked to user 1 and user 2 .
  • section 3 between time t5 and time t6 can be associated only with user 2 .
  • step 6330 the weight change at both ends of each segment divided into cases is obtained. For example, for section 1, a weight change of 35 g is obtained. For interval 2, a weight change of 15 g is obtained. For interval 3, a weight change of 0 g is obtained.
  • step 6340 it is assumed that the change in weight is distributed to each user for the section linked to a plurality of users. For example, for section 1, a weight change of 35 g is obtained only for user 1. For interval 2, a weight change of 15 g is obtained for users 1 and 2, so it is assumed that a weight change of 15 g is distributed for each of users 1 and 2 (see assumptions 1 and 2 above).
  • step 6350 a determination of the validity of the above assumptions is made regarding the weight per item. For example, assumption 1 is determined to be valid in comparison with the weight of 50 g for one product, because the product weighing 50 g is distributed only to user 1 . Assumption 2 is determined to be invalid because 35 g of product is distributed to user 1 and 15 g of product is distributed to user 2, compared with the weight of one product of 50 g.
  • the determination performed by the customer number determination module 1105 regarding which user has picked up which product and how many is not limited to the flow of FIG. 63 .
  • FIG. 64 another example of the determination flow 6400 of the customer quantity determination module 1105 is simplified.
  • the customer number determination module 1105 acquires the stretching section for each user.
  • step 6420 the customer number determination module 1105 acquires the weight change at both ends of the stretching section for each user.
  • step 6430 the customer number determination module 1105 acquires the ratio of weight change per product for each user.
  • step 6440 the customer count determination module 1105 compares the percent weight change for all users.
  • FIG. 65 an output example of the flow illustrated in FIG. 64 is shown.
  • the period during which the customer reached out for the product is obtained, and the ratio of the weight change amount corresponding to that period is obtained.
  • a weight change amount of 20 g is obtained for customer #1
  • a weight change amount of 120 g is obtained for customer #2
  • a weight change amount of 10 g is obtained for customer #3.
  • a weight change is required.
  • the customer number determination module 1105 can estimate that the product was picked up by customer #2 by comparing the ratios of these three weight change amounts. As shown in FIG. 65, the change in weight in the stretching section may be displayed graphically for each user to facilitate visual comparison. This output may be displayed on the screen of the user terminal or the like. The display mode is not limited to FIG. 65, and may be displayed in a pie chart or the like. In this embodiment, the determination performed by the customer number determination module 1105 regarding which user has picked up which product and how many is not limited to the flow of FIGS. 63 and 64 . Various modifications and changes are possible for the flows illustrated in FIGS. 63 and 64 .
  • the customer quantity determination module 1105 cannot determine the products purchased by the customer and the quantity based on the information from the weight sensor unit 105 and the images from the camera sensor units 104 and 104a, the information is stored in the required check list. preferably.
  • the customer number determination module 1105 may detect the amount of change in weight when it is doubtful, or when detection of position information from the camera device 506 and calculation of estimated hand points based on the captured image from the camera device 506a are not successful.
  • the information is stored in a necessary check list for final human check.
  • human assistance can be provided later, thereby realizing a highly accurate shopping experience. Since the settlement process itself is executed after leaving the store, the customer's experience is minimized.
  • the management server 101 preferably displays an appropriate instruction on each user's terminal as determination impossible. For example, for products for which the customer quantity determination module 1105 could not make a sufficient determination, a store clerk may display an instruction to inquire of the user in question and inquire about which products have been picked up. good.
  • each user may be asked to select which product they picked up by self-declaration.
  • This post-processing is preferably performed before the user leaves the store, but it can also be performed after the user leaves the store.
  • the display of instructions for asking for user cooperation can be done in a variety of ways. For example, for products that could not be determined, a message such as "under confirmation" may be displayed on the terminal for the user to confirm.
  • FIG. 47 another example of product pick-up determination performed by the management server 101 is shown.
  • one product is picked up.
  • this embodiment makes it possible to calculate the number of items taken. Note that the example of FIG. 47 can have all of the determination contents described above with reference to FIGS.
  • FIG. 47 changes in the output of the weight sensor and changes in estimated hand points of two users are shown superimposed.
  • an example of weight time-series data of each product shelf is shown in a graph.
  • the horizontal axis of this graph is time (ms), and the left vertical axis is the value (g) recorded by the weight sensor.
  • the figure graphically shows an example of changes in the estimated point of each hand for two users near a shelf equipped with the weight sensor device 608 .
  • the horizontal axis of this graph is time (ms), and the vertical axis on the right side is the distance (mm) between each customer's hand and the product.
  • a person ID (identification number) of A and B is assigned to each user.
  • the output of weight sensor device 608 registers a large drop with a momentary bounce, while between 5000 ms and 6000 ms, the output of weight sensor device 608 stabilizes. Therefore, the customer number determination module 1105 can estimate that the product on the product shelf has been picked up. During this period, the output of the weight sensor device 608 finally decreases from approximately 300 g to 80 g. Therefore, the customer quantity determination module 1105 can estimate that one or a plurality of commodities with the weight of this difference (approximately 220 g) have been picked up.
  • the customer quantity determination module 1105 can understand that there were hand movements of two users near the item in question. Therefore, the customer number determination module 1105 aligns the hand segment distances of the users A and B with the times of the weight time series data. Then, from the hand segment distance information, the period during which each customer reaches out to the corresponding segment (hand reaching section) is determined. At that time, for each of users A and B, it is determined whether or not the hand movement exceeds a predetermined threshold.
  • the customer number determination module 1105 estimates a stretched section and acquires weight changes at both ends thereof. For example, the customer quantity determination module 1105 finds that for user A, the estimated point of the hand is positioned near the product on the shelf within a period of approximately 2200 ms to 3100 ms. Further, the customer number determination module 1105 finds that the estimated point of the hand of customer B is positioned near the product on the product shelf within a period of approximately 3600 ms to 4500 ms. The customer count determination module 1105 then calculates the difference in weights at both ends for each period and associates it as the weight taken by the associated user. At this time, the customer number determination module 1105 sums up the weight for each user, compares it with the weight of the product (registered in advance), and converts it into the number of products.
  • the customer number determination module 1105 performs processing including the following steps.
  • the customer quantity determination module 1105 previously sets the weight per product and the ratio of allowable error, and stores the corresponding data in the storage device. For example, if the weight of one product is 50 g, the allowable margin of error is set to 0.2 (10 g).
  • the customer quantity determination module 1105 receives a numerical value (g) indicating a change in the weight of the product when the product is picked up by the weight sensor device 608 combined with the product shelf.
  • the customer quantity determination module 1105 divides the changed weight numerical value (g) by the pre-stored weight (g) per product.
  • the customer quantity determination module 1105 compares the above calculation result with the pre-stored error rate of the product. As a result of the above comparison, if it is determined that the number is within the allowable error ratio, the customer number determination module 1105 determines the number.
  • the customer number determination module 1105 mainly considers user A's hand movements during the period from time t11 to time t12. During this period, user B's hand movement is recognized, but the hand movement can be ignored due to the long distance from the product segment. Further, the customer quantity determination module 1105 learns that there was a weight change of about 100g (300g-200g) during this period. Further, the customer number determination module 1105 mainly considers user B's hand movements during the period from time t13 to time t14. Further, we know that there was a weight change of about 120g (200g-80g) during this period.
  • the customer number determination module 1105 can assume that User A picked up a product weighing 100 g, and then User B picked up a product weighing 120 g. Therefore, the customer quantity determination module 1105 verifies the above assumption based on the stored weight of the product in the product segment. As a result of the above verification, if it is determined that the error is out of the allowable error, the customer number determination module 1105 determines that the number is indeterminate, and vice versa.
  • the customer number determination module 1105 of the management server 101 is configured to enable determination of product pick-up even if the weight change based on the output of the weight sensor device 608 deviates from the ideal case. I am devising.
  • FIGS. 48 and 49 further examples of weight sensor outputs are shown.
  • changes in the output of the weight sensor and changes in the estimated points of the hands of the two users are superimposed.
  • the examples of FIGS. 48 and 49 can include all of the judgment contents described above with reference to FIGS. 45, 46, 47, 63, and 64. The explanation about is omitted.
  • the output of weight sensor device 608 does not necessarily have ideal weight changes.
  • a sensor weight sensor device 608, etc.
  • a load is applied to the bar, plate, product basket, or the like when the product is picked up, and the weight increases. may vary.
  • the weight change in the output of the weight sensor device 608 may take time.
  • vertical fluctuations may occur in changes in weight. In these cases, in the prior art, it was sometimes difficult to determine how many commodities were picked up at what time based only on the weight information of the weight sensor device 608 .
  • the customer count determination module 1105 of the management server 101 tracks the estimated point of each user's hand near the shelf in question.
  • the estimated hand point is compared to a predetermined threshold to determine whether the hand is extended close to the item. Therefore, the management server 101 estimates the stretched section of each user.
  • the customer number determination module 1105 can refer to the output from the weight sensor device 608 or the like for the times at both ends of each section. Therefore, the customer number determination module 1105 can acquire the weight change at both ends of each user's stretching section.
  • the customer number determination module 1105 can classify each user into cases of conversion from weight change to number. For example, if a change in the estimated hand point is detected for only one user, the weight change is converted to count for that user only. Also, when changes in the estimated hand points are detected only for two or more users, cases are classified for each user, and each weight change is converted into the number. The customer number determination module 1105 determines whether or not the finally obtained change in weight is appropriate for each case.
  • the customer quantity determination module 1105 can determine the number of items picked up even if the output of the weight sensor device 608 is not ideal.
  • the output of the weight sensor device 608 and the position of the estimated point of the hand obtained by calculation are not always obtained accurately. Therefore, in this embodiment, further improvements can be made so that the customer number determination module 1105 of the management server 101 can determine more satisfactorily whether or not products are being picked up. For example, there is a slight difference between the time when the user actually reaches out toward the product shelf and tries to grab the product and the time when the weight sensor device 608 actually outputs the variation in weight of the product. Differences can occur. This is because a change in the output of the weight sensor device 608 may be obtained before and after the moment when the product is actually picked up.
  • the upper part shows the change in the output of the weight sensor device 608, and the lower part shows the change in the distance of the estimated point of the user's hand with respect to the product.
  • the user extends his/her hand toward the product, and the distance between the hand and the product exceeds the threshold and approaches. Further, at time T2, the user moves his/her hand away from the item, and the distance between the hand and the item exceeds the threshold. Somewhat later than this movement of the user's hand, a drop in the output of weight sensor device 608 is detected between times Ta and Tb.
  • the management server 101 calculates the movement of the estimated point of the user's hand and the weight sensor device. There is a risk that an accurate correspondence relationship cannot be obtained with the change in the output of 608 .
  • the customer number determination module 1105 of the management server 101 determines that there is a discrepancy between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608. is detected, by shifting the time of one of them back and forth, correction is made so that the correspondence between the two becomes clearer.
  • the customer quantity determination module 1105 slightly shifts the times Ta and Tb based on the actual output of the weight sensor device 608 forward (see times Ta' and Tb').
  • This correction time can be predetermined as a fixed value. In this case, if it is detected that there is a gap between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608, either one of the times may be shifted forward or backward by a predetermined amount. shift to Alternatively, this correction time (Ta-Ta' or Tb-Tb') can be predetermined as a variable value. In this case, when it is detected that there is a difference between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608, one of the shift the time forward or backward.
  • the customer number determination module 1105 can modify either the time based on the user's hand movement or the time based on the output of the weight sensor device 608. By setting the correction time in this way, the customer number determination module 1105 can obtain a more accurate correspondence relationship between the movement of the estimated point of the customer's hand and the change in the output of the weight sensor device 608.
  • the weight sensor device 608 may detect the event with a delay. Even in such a case, the present embodiment introduces a correction time so that it is possible to actually determine whether the product is picked up more accurately. As a precondition for introducing the correction time, it is determined whether or not the difference between the time based on the movement of the user's hand and the time based on the output of the weight sensor device 608 is smaller than a predetermined amount. is preferred.
  • further refinements may be made to allow the customer count determination module 1105 to better determine whether an item is being picked up. For example, when a customer actually reaches out toward a product shelf and tries to pick up a product, when the skeleton is detected based on the photographed image, the position of the actual characteristic portion and the position of the characteristic portion obtained by calculation are not the same. A slight deviation may occur between them. If this deviation is large, it may be detected that the hand is not extended even though the hand is actually extended to pick up the product.
  • the upper part shows the change in the output of the weight sensor device 608, and the lower part shows the change in the distance of the estimated point of the user's hand with respect to the product.
  • the user extends his/her hand toward the product, and the distance between the hand and the product exceeds the threshold and approaches. Further, at time T2, the user moves his/her hand away from the item, and the distance between the hand and the item exceeds the threshold. Further, during this period, it is detected that the user puts his hand back and extends his hand for a short period of time between times Ta and Tb. Therefore, the customer number determination module 1105 cannot associate the movement of the user's hand with the change in the output of the weight sensor device 608 between times Ta and Tb (during that period, the user It is determined that the product has not been picked up).
  • the length and direction of the forearm are obtained from the positions of the elbow and wrist, and the position of the estimated point of the hand can be calculated by extending the length. If the positions of the elbows and wrists, which are used as references for this calculation, are displaced, the estimated points of the hands may be displaced. For example, it may be determined that the user's hand does not reach a predetermined position even though the user is actually reaching into the product shelf. For example, in (A) of FIG. 51, although the user actually stretches out his hand toward the product continuously between times T1 and T2, between times Ta and Tb, It is detected that there is a break in the movement of the hand. If this movement of the cut is caused by an inaccurate calculation of the estimated point of the hand, the customer quantity determination module 1105 cannot make a correct product pick-up determination.
  • the customer number determination module 1105 constantly monitors the position of the estimated hand point associated with the person ID. When the estimated point of a hand associated with a certain person ID exceeds a predetermined threshold, it is estimated that the hand has reached a position where the product can be picked up from the product shelf (for example, time T1).
  • the estimated point of the hand associated with the same person ID falls below a predetermined threshold, it is estimated that the hand has moved away from the position where the product can be picked up from the product shelf (for example, the time Ta).
  • the estimated point of the hand linked to the same person ID exceeds the predetermined threshold again, it is estimated that the hand has reached a position where the product can be picked up from the product shelf (for example, , time Tb).
  • the estimated point of the hand associated with the same person ID falls below a predetermined threshold, it is estimated that the hand has moved away from the position where the product can be picked up from the product shelf (for example, the time T2).
  • the customer number determination module 1105 performs the following determination when a discontinuity is detected during the period in which the customer's hand is extended (for example, between times Ta and Tb). Before and after the detected break, it is determined whether or not the person ID of the hand at the position where the product can be picked up is the same. In the above determination, if the person IDs before and after the discontinuity are the same, it is further determined whether or not the time interval (between times Ta and Tb) at which the discontinuity occurred is below a predetermined threshold. .
  • the customer quantity determination module 1105 determines that the break can be ignored, and makes corrections so that the break does not actually exist. Add As a result, the determination of item pickup may be avoided or minimized from being adversely affected by the cut. For example, the customer quantity determination module 1105 minimizes or zeroes the interval between times Ta and Tb, as exemplified by time Tc. As a result, even if there is a break in the movement of the user's hand, it is possible to avoid an error in the product pick-up determination.
  • the quality of the photographed screen of the camera device 506a of the camera sensor unit 104a is not uniform, and there is a possibility that the positions of the elbows and wrists on the photographed screen may shift or blur due to sudden movements of the customer to be photographed. .
  • the customer number determination module 1105 determines whether or not positional deviation is caused for each person ID, and if the interval is short (for example, between times Ta and Tb), It is corrected to be ignored (for example, time Tc).
  • the interval is short (for example, between times Ta and Tb), It is corrected to be ignored (for example, time Tc).
  • the customer count determination module 1105 may be made to better determine whether an item is being picked up.
  • the upper part shows the change in the output of the weight sensor device 608, and the lower part shows the change in the distance of the estimated points on the hands of the users 1 and 2 regarding the commodity.
  • (A) of FIG. 52 at time T1, user 1 extends his hand toward the product, and the distance between the hand and the product exceeds the threshold and approaches. Further, at time T2, User 1 puts his/her hand away from the product, and the distance between the hand and the product exceeds the threshold.
  • the customer number determination module 1105 cannot correctly determine whether to pick up the product.
  • the upper portion shows changes in the output of the weight sensor device 608, and the lower portion shows changes in the distance between the item and the estimated point on the user's hand.
  • the customer number determination module 1105 determines that there are hand movements of two or more users 1 and 2 before and after the change in the output of the weight sensor device 608 occurs, and that between the movements of each of the users 1 and 2 Suppose that a break is detected (for example, between time T2 and T3 or a predetermined period after time T4).
  • the customer number determination module 1105 compares the size of the gap (for example, between time T2 and T3, or a predetermined period after time T4) with a predetermined threshold to determine whether the size is negligible. conduct. If it is determined that it cannot be ignored, the customer number determination module 1105 expands the movement of each of the users 1 and 2 back and forth for a predetermined period of time, and adjusts the size of the gap (for example, between times T2 and T3, (predetermined period after time T4) is reduced. Thereby, the movement of each user 1, 2 is better matched with the change in the output of the weight sensor.
  • the customer count determination module 1105 does any one or more or all of the following.
  • the time T1 during which the user 1 extends his or her hand toward the product shelf is shifted forward by a predetermined amount of time (for example, time Ta).
  • the time T2 at which the user 1 puts his/her hand back from the product shelf is delayed by a predetermined amount of time (for example, time Tb).
  • the time T3 during which the user 2 extends his or her hand toward the product shelf is shifted forward by a predetermined amount of time (for example, time Tc).
  • the time T4 at which the user 2 puts his/her hand back from the product shelf is delayed by a predetermined amount of time (for example, time Td).
  • the magnitudes of time Ta, Tb, Tc, and Td introduced by the correction may be the same. Also, the magnitudes of these times Ta, Tb, Tc, and Td may change depending on the implementation status. Therefore, the customer number determination module 1105 shortens or minimizes the interval in which no one reaches out by extending the hand movement of each user 1, 2 back and forth, and changes the weight of the user 1, 2. make it easy to associate with
  • the customer number determination module 1105 may perform the controls illustrated in FIGS. 50B, 51B, and 52B independently or in combination with each other. . Furthermore, in these judgment controls, the controls illustrated in FIGS. 45, 46, 47, 48, 49, 63 and 64 can be appropriately selected and combined. For example, when a change in the weight of a product is detected in a section where multiple users are simultaneously reaching out around a product, there is a risk that it will not be possible to sufficiently determine which user has picked up how many products. It can happen. In the present embodiment, when such a user/number tends to be uncertain, the determination logic when the hands overlap may be introduced to cope with the situation. Therefore, in this embodiment, regarding the determination of whether or not an event to actually pick up a product using the "estimated point of the hand" has occurred, the actual operation is made easier and the determination accuracy is improved. I do.
  • the customer number determination module 1105 can perform the following control when there are multiple customers near a certain product in the store. For each customer, determine the correlation between the calculated estimated hand point and the shelf (item) location. In particular, the time interval during which each customer reaches out near the product is obtained. A change in the output of the weight sensor is determined. In particular, among the time-series weight data acquired by the weight sensor, the amount of change in the weight of the product corresponding to the time interval is obtained for each customer. To determine whether or not a product has been picked up from a product shelf for each customer. In particular, the ratio of the weight change amount of the product corresponding to the above time interval for each customer to the weight of one product is obtained.
  • the present embodiment enables determination of the user's movement to pick up items under various circumstances. For example, in this embodiment, it can be determined which user has taken which product and how many. In particular, this embodiment makes it possible to correctly determine the user who picked up the item and the number of items. In particular, the pick-up of the product is determined for each person ID of the user. Furthermore, this embodiment determines the pick-up of products including the number/type for each person ID of the user. In particular, it is possible to deal with even if the product is continuously acquired at intervals that the weight sensor cannot separate.
  • the conventional technology when the movement of a customer picking up a product is determined using the image captured by the camera and the output of the weight sensor, a plurality of customers are present at the same time within a distance from which the product can be picked up. (When multiple hands exist near the product at the same time), it was sometimes difficult to judge. Further, in the conventional technology, when a plurality of commodities are continuously picked up at the same position (when the times of the commodities to be picked up are close), the determination tends to be difficult. Further, in the conventional technology, when a plurality of adjacent products are picked up at the same time (when the positions of the products to be picked up are close), the determination tends to be difficult. Moreover, in the conventional technology, when a plurality of products are picked up at the same time, it tends to be difficult to judge.
  • the "weight information" of the product includes only the weight change difference before and after the state stabilizes. Further, in the prior art, when determining the movement of the customer picking up the product, it was assumed that only one customer picked up the product at the time when the weight information changed. Further, in the conventional technology, it is determined that the user near the weight sensor has picked up the product, and the result is output. For this reason, when a plurality of users existed nearby, there was a case where it was output as a case in which a sufficient determination could not be made.
  • DNN Deep Neural Network
  • a DNN is a deep learning neural network with four or more layers. DNN can be applied to image recognition. In particular, when two-dimensional image data is converted into an appropriate one-dimensional numerical value sequence and input to DNN, it recognizes what is shown in the image and at what position. can output the results.
  • DNN When DNN is used for image recognition, there are the following problems.
  • DNN the types of products to be judged are limited.
  • the development of new products for products sold in general stores continues almost continuously. For example, supermarkets, convenience stores, etc. are constantly introducing new products.
  • DNN when trying to increase the types of products to be judged, it is necessary to change the configuration of the model and learn the model again. If the intervals between new product introductions are short, a sufficient learning period cannot be ensured, and operational issues remain.
  • DNN when DNN is applied on the captured image, only a display method in which the presence or absence of the product is captured by the camera can be adopted.
  • a product shelf having a plurality of levels flat stacking method, etc.
  • DNN can be applied.
  • the whole product cannot be photographed on the second and subsequent stages of the product shelf, it may be difficult to apply the DNN. Therefore, in a store with multiple product shelves, DNN-based image recognition still has operational issues.
  • the customer number determination module 1105 of the management server 101 uses the customer's movement trajectory obtained based on the photographed image from the camera device 506 of the camera sensor unit 104 and the camera sensor unit 104a.
  • the customer's movement trajectory obtained based on the photographed image from the camera device 506 of the camera sensor unit 104 and the camera sensor unit 104a.
  • this embodiment it is not necessary for the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a to photograph the entire product. Therefore, this embodiment can be applied regardless of how the products are displayed. For example, this embodiment can be applied to any shape of product shelf (flat stacking method, etc.).
  • a device is devised to make the pick-up determination of the product more favorable.
  • a store has a plurality of product shelves (see FIGS. 36A and 36B). Therefore, it is possible that the same customer's behavior is captured by a plurality of cameras (eg, camera device 506, camera device 506a, etc.) at approximately the same time.
  • a plurality of cameras eg, camera device 506, camera device 506a, etc.
  • the front side of the customer facing the product shelf it is preferable to photograph the front side of the customer facing the product shelf from above. In that case, it is preferable to photograph the entire movement of the customer's arm trying to grab the product. More preferably, the left and right arms of the customer are photographed relatively evenly. On the other hand, when the customer is photographed from the side, the customer's body may hide the movement of the customer's arm to grab the product. In addition, it becomes difficult to photograph the left and right arms of the customer relatively evenly.
  • the content of the images captured by each camera is not uniform.
  • the former should be given higher priority than the latter. preferable. This ensures that the image analysis is based on images of preferred quality.
  • FIG. 53 a plan view of the inside of an actual store that can use this embodiment is shown.
  • the store has an entrance/exit (entering area/exiting area), followed by a floor on which a plurality of product shelves are installed.
  • the floor is shown to have a substantially rectangular shape in the figure, the actual shape of the floor can be configured in various ways.
  • each product shelf installed on the floor is usually divided into multiple tiers above and below, and each tier presents multiple products. Between these product shelves is a free space where customers can freely walk around.
  • one floor has a plurality of areas.
  • Each area is exemplified by a substantially rectangular frame, and one or more product shelves can be installed in the frame.
  • the shape and size of this area can be variously configured. Preferably, it is associated with a range that can be satisfactorily photographed by a camera installed nearby. Adjacent areas may have a gap between them. Adjacent areas may also partially overlap between the two.
  • FIG. 55 there is shown an example of an actual camera-captured image of the product shelf and/or area in the store illustrated in FIG.
  • This captured image is applicable to both the camera device 506 of the camera sensor unit 104 and the camera device 506a of the camera sensor unit 104a.
  • the area setting may be set so as to surround the front of the shelf (the position where the product is placed and the slightly closer side).
  • the POV/FOV of the camera shoots the product shelf from almost directly above, and the customer's arm trying to pick up the products displayed on the product shelf is viewed from the front. Allows you to take pictures.
  • a customer faces the product shelf in order to check the products displayed in the product shelf. In this case, the appearance of the customer's upper body can be photographed relatively well. It should be noted that even when the user stands obliquely to the product shelf and picks up the product from the side, at least the state of the outstretched arm can be photographed.
  • the position of each product shelf and/or area and the POV/FOV of each camera that captures its surroundings are known in advance. Therefore, it is already known which product shelf can be photographed optimally by which camera.
  • the management server 101 determines the preferred camera priority order for each product shelf and/or area, and can perform image analysis of captured images according to the order. For example, if it can be assumed in advance that images taken by multiple cameras are available, images taken from the side will result in lower positional accuracy, so if images that can be taken from the front are available, the latter is given priority. use for purpose.
  • the priority can be set variously based on the situation in the store. For example, priority may be based on the distance between the position of each camera 506, 506a and the position of the shelf and/or area. For example, the priority may be based on the correspondence between the POV/FOV of each camera 506, 506a and the people who can be present around the shelf and/or area. For example, the priority may be based on the correspondence between the captured images of the cameras 506 and 506a and the direction of the subject. For example, the priority may be based on the presence or absence of an obstacle between the images captured by the cameras 506 and 506a and the subject. For example, the priority may be based on the captured images of the cameras 506 and 506a and the accuracy of bone structure detection based thereon.
  • the management server 101 may set the priority of which camera's captured image is to be used in advance so that the image analysis quality satisfies a predetermined standard.
  • FIG. 56 eight areas are demarcated in a plan view (floor) of a certain store, and the shooting conditions of each area are exemplified. These captured images can be applied to both or either of the cameras 506, 506a. As can be understood from FIG. 56, the inside of the store can be acquired as a plurality of (for example, eight) captured images. Tracking of a customer moving in the store may change depending on which of these captured images is used.
  • the management server 101 determines in advance which photographed image is preferable to be used among a plurality of possible photographed images (e.g., eight) in accordance with changes in the position of the customer, changes in the movement of the customer, and the like. You can decide the degree.
  • FIG. 57 in the case illustrated in FIG. 56, there is shown an example in which the appearance of a customer moving in the store is tracked from two different cameras.
  • eight cameras indicated by reference numerals 0236, 0108, 0227, 0357, 0570, 0390, 0391, and 0138 are illustrated from upper right to lower left. These cameras may be camera device 506 of camera sensor unit 104 and/or camera device 506a of camera sensor unit 104a.
  • a user with a person ID of 837 is detected almost simultaneously by two cameras (0227 and 0391).
  • the person ID “837” displayed in a smaller size corresponds to the person ID output based on the captured image (ToF) of the camera device 506 of the camera sensor unit 104 .
  • the person whose person ID "837" is displayed in a large size is the person ID of the user estimated to exist on the floor as a result of linking the skeleton and position based on the captured image of the camera device 506a of the camera sensor unit 104a. is equivalent to
  • Customer position information detected by each camera may change depending on the relationship between the POV/FOV of each camera and the position of the customer in the store.
  • the management server 101 can continuously track the position of the customer and determine in advance the priority of images taken from which camera to use according to changes in the position. As a result, the management server 101 can always track the position of a customer who moves within the store with high accuracy. This can be applied to both the camera device 506 of the customer position detection camera sensor unit 104 and the camera device 506a of the customer skeleton detection camera sensor unit 104a.
  • the POV/FOV of each camera 506, 506a are different from each other so as to adequately cover the vicinity of the corresponding shelf.
  • the POV/FOV of each camera 506, 506a is pre-fixed so as to capture a good picture of nearby product shelves.
  • the management server 101 may change the POV/FOV of the cameras 506 and 506a.
  • the management server 101 can preset the priority of each camera 506, 506a and modify the POV/FOV of each camera 506, 506a in accordance with the actual operational situation. In this way, this embodiment is devised so that the movements of customers can be tracked continuously.
  • This control may be performed by the position information processing module 1102, for example.
  • the skeleton detection processing module 1102a, the customer number determination module 1105, or the like may be used.
  • the customer may not be completely tracked by camera device 506 of camera sensor unit 104 or camera device 506a of camera sensor unit 104a. If the customer's location information cannot be tracked accurately, it becomes a problem because it is not possible to make a sufficient determination of the subsequent pick-up of the product.
  • FIG. 58 there is a simplified illustration of mistracking/swapping of the camera device (ToF) 506 of the camera sensor unit 104 in the prior art.
  • ToF camera device
  • FIG. 58 there are two users in the store.
  • 0 and 1 are assigned as person IDs, respectively.
  • Each user 0, 1 is tracked by the same or different cameras.
  • (1) and (2) of the figure when two customers are close to each other, a situation may arise in which they cannot be clearly identified.
  • the person IDs 0 and 1 of the two customers who suddenly approach each other are exchanged as shown in (2) to (3) of the figure. In this way, if the customer's person ID is replaced in the tracking information (personal information), there is a risk that subsequent tracking of the customer, skeleton detection, and the like will all be erroneous in a chain reaction.
  • means are provided to prevent the occurrence of the above situation.
  • information on the clothing of the customer is extracted at the same time. do.
  • Each customer can be tracked by combining location information and clothing information (color information). Therefore, even if two customers cannot be clearly identified only by the position information as a result of the positions of the two customers being close to each other, they can be distinguished from each other based on the information of the clothes of both customers.
  • the color or shape of the customer's clothing is extracted based on the image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. For example, extract the color of the customer's shoulders.
  • the customer's head may lose consistency in color and shape information due to putting on and taking off a hat or the like.
  • Elbows, wrists, arms, etc. can move relatively freely and can be hidden depending on the viewing angle.
  • the elbow when the detection position is shifted or when the arm is rolled up, it can be either bare skin or clothes. Also, in the case of the wrist, when entering and exiting a pocket or the like, it can be either clothes or a hand. In contrast, shoulder position is relatively continuously visible in many situations. Therefore, by using the shoulder color information, the customer's dress information can be continuously tracked.
  • the customer's other color information may be used.
  • shoulder color information and chest or back color information may be used in combination.
  • customer shoulder, chest and/or back clothing shape information may be used.
  • shape, pattern, color, and/or feature amount of the entire outfit for example, the feature amount can be extracted by a separately created model/feature amount calculation method), etc. may be used. .
  • the color of the clothing is determined by acquiring the color of the customer's shoulder coordinates from the RGB image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a. good too.
  • the position of the head and the position of the shoulders may partially overlap due to the left and right movement of the head. Therefore, it is preferable to adopt the coordinates of the farther shoulder from the head, out of the coordinates of both shoulders. As a result, depending on the viewing angle of the user, the head may overlap the shoulder, but such a situation can be avoided.
  • the position of the head and the positions of the left and right shoulders are compared, and the right shoulder, which is farther from the head, may be adopted, and the left shoulder may not be adopted.
  • the location information and the color of the shoulder clothing are recorded for a certain period of time immediately after entering the store.
  • clothing information may be obtained not only immediately after entering the store, but at a position/timing at which the user's clothing can be stably determined.
  • the customer number determination module 1105 periodically records the color of the shoulders for several tens of seconds, one minute, several minutes, or any other arbitrary period after starting acquisition of clothing information at an arbitrary timing. The average value is calculated and registered as the color of the customer's clothing.
  • the customer number determination module 1105 continues to acquire the shoulder color of each customer, and when a plurality of customers approach each other (see (2) in FIG. 58), each customer's location information and clothes. are used in combination with color information to distinguish from each other. If location information and/or clothing color information are temporarily exchanged among multiple customers, the identification number is exchanged to prevent or prevent the identification number (person ID) from being erroneously assigned. correct.
  • the customer quantity determination module 1105 functions as follows. For example, when a customer comes to a preset appearance area (near the entrance) in a store, each customer is given a unique person ID and tracking begins.
  • a person ID may be configured using numbers and/or letters and the like. For example, the person ID may be given as a serial number to each customer who visits the store. Also, for example, the person ID may be given as a combination of date/time. For example, an identification number of 20210530071030 may be given to a customer who came to the store at 7:10:30 am on May 30, 2021.
  • suffixes such as a, b, and c.
  • the person ID assigned to each customer remains the same as long as the customer exists in the store. Then, when the customer comes to a preset disappearing area (near the exit) in the store, the person ID assigned to each customer is erased and the tracking ends. When the same person visits the same store again, a different person ID is given each time. In the conventional technology, the person ID may disappear even though the customer has not come to the predetermined lost area in the store (lost person ID). In that case, there is a problem that the user's tracking information is no longer output. In addition, when the customer comes to a predetermined appearance area (near the entrance) in the store, another person ID may be given to the customer (an unintended reappearance of the person ID). In that case, there is a problem that the customer will be judged as if he/she is another customer (another person) thereafter.
  • the skeleton detected from the image captured by the camera device 506 of the camera sensor unit 104 or the camera device 506a of the camera sensor unit 104a The customer can be tracked based on the information and the color of the clothing.
  • their person ID can be maintained continuously.
  • even if the customer comes to a predetermined appearance area (near the entrance) in the store it is possible to avoid/correct giving a new person ID to the customer.
  • the customer quantity determination module 1105 maintains person IDs of customers being tracked by the camera device 506 of the camera sensor unit 104. . Therefore, as illustrated in (1) to (2) of FIG. 60, customers close to each other can be identified. Furthermore, as illustrated in (3) of FIG. 60, when the tracking temporarily ends without the customer's position entering the lost area (see the dotted line with symbol A), for example, the customer number determination module 1105 , the person ID can be retained, assuming the customer is staying in the store.
  • the customer count determination module 1105 keeps the data available for each customer currently being tracked. For example, each customer that is currently being tracked is associated with their skeleton information and/or clothing color information. For example, if the customer number determination module 1105 finds excess skeletal information and/or clothing color information among all the customers in the store, the customer who is assumed to be staying in the store (temporarily cannot be tracked) The missing customer) may be assumed to be at the location where the skeletal and/or clothing color information was found, and tracking may resume from that location.
  • the customer number determination module 1105 may further detect color information near the shoulders of the human body by applying a skeleton detection model to the image captured by the camera. For example, the customer number determination module 1105 may assign a person ID to the owner of the shoulder based on the detected shoulder color information.
  • a camera capable of acquiring depth information (depth) can be used as the camera sensor unit 104a.
  • the camera sensor unit 104a can be a regular camera (2D camera).
  • the camera sensor unit 104 is also capable of position tracking with a ToF sensor module.
  • the camera sensor unit 104 position tracking module itself may be omitted.
  • user identification and matching may be performed between captured images from a plurality of camera sensor units 104a. In this way, it is possible to both track the customer's position and acquire the skeleton information only with the plurality of camera sensor units 104a.
  • the reference (threshold value) for determining the degree of approach when the customer reaches out toward the product can be set by manually acquiring the position of the product.
  • the position of each product may be automatically corrected by obtaining the coordinates of the customer's hand when picking up the product and overwriting the position of each product.
  • a human determination can be made by a judge.
  • the judging staff can reproduce the moving images captured by the camera sensor units 104 and 104a as they are.
  • means can be provided to assist the judge's decision.
  • the position/orientation (camera parameters) of each camera can be calculated by calibrating the camera that shoots the moving image for determination. Furthermore, by using this parameter, it is possible to calculate where the position of the product and the position of the user are on the screen, and annotate it in the video to make it easier for the judges to understand. good.
  • the present invention is not limited to the above-described embodiments, and includes various modifications.
  • the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations.
  • it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
  • each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them with an integrated circuit.
  • each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
  • Information such as programs, tables, and files that implement each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.
  • control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected. It should be noted that the above embodiments disclose at least the structures described in the claims.

Landscapes

  • Business, Economics & Management (AREA)
  • Finance (AREA)
  • Engineering & Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Economics (AREA)
  • Development Economics (AREA)
  • Computing Systems (AREA)
  • Marketing (AREA)
  • Strategic Management (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
PCT/JP2022/016902 2021-06-03 2022-03-31 無人店舗の商品を管理する管理システム、管理方法及びプログラム Ceased WO2022254958A1 (ja)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021093705A JP7663178B2 (ja) 2021-06-03 2021-06-03 無人店舗の商品を管理する管理サーバ及び管理方法
JP2021-093705 2021-06-03

Publications (1)

Publication Number Publication Date
WO2022254958A1 true WO2022254958A1 (ja) 2022-12-08

Family

ID=84323120

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/016902 Ceased WO2022254958A1 (ja) 2021-06-03 2022-03-31 無人店舗の商品を管理する管理システム、管理方法及びプログラム

Country Status (2)

Country Link
JP (1) JP7663178B2 (https=)
WO (1) WO2022254958A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7318680B2 (ja) * 2021-07-30 2023-08-01 富士通株式会社 情報処理プログラム、情報処理方法、および情報処理装置
JP7708735B2 (ja) * 2022-12-26 2025-07-15 株式会社日立製作所 無人店舗運営支援装置及び無人店舗運営支援方法
JP7667365B1 (ja) 2024-09-27 2025-04-22 株式会社サイバーエージェント スマートグリップ

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019181499A1 (ja) * 2018-03-20 2019-09-26 日本電気株式会社 店舗管理装置および店舗管理方法
US20200193347A1 (en) * 2018-12-17 2020-06-18 Beijing Baidu Netcom Science Technology Co., Ltd. Vehicle passenger flow statistical method, apparatus, device, and storage medium
CN111831673A (zh) * 2020-06-10 2020-10-27 上海追月科技有限公司 货品识别系统、货品识别方法及电子设备
JP2021012518A (ja) * 2019-07-05 2021-02-04 クラスメソッド株式会社 無人店舗の商品を管理する管理サーバ及び管理方法
CN112434566A (zh) * 2020-11-04 2021-03-02 深圳云天励飞技术股份有限公司 客流统计方法、装置、电子设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019181499A1 (ja) * 2018-03-20 2019-09-26 日本電気株式会社 店舗管理装置および店舗管理方法
US20200193347A1 (en) * 2018-12-17 2020-06-18 Beijing Baidu Netcom Science Technology Co., Ltd. Vehicle passenger flow statistical method, apparatus, device, and storage medium
JP2021012518A (ja) * 2019-07-05 2021-02-04 クラスメソッド株式会社 無人店舗の商品を管理する管理サーバ及び管理方法
CN111831673A (zh) * 2020-06-10 2020-10-27 上海追月科技有限公司 货品识别系统、货品识别方法及电子设备
CN112434566A (zh) * 2020-11-04 2021-03-02 深圳云天励飞技术股份有限公司 客流统计方法、装置、电子设备及存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
MIYAJIMA YOSUKE: "Using Cubemos' Skeleton Detection SDK and RealSense to Detect Human Reaching a Shelf", DEVELOPSIO, 18 May 2020 (2020-05-18), pages 1 - 10, XP093010961, Retrieved from the Internet <URL:https://dev.classmethod.jp/articles/detecting-human-hands-entering-a-shelf-using-cubemos-skeleton-tracking-sdk-and-realsense-d435i/> [retrieved on 20230102] *
MIZUKI MARUYAMA, SHUVOZIT GHOSE, KATSUFUMI INOUE, PARTHA PRATIM ROY, MASAKAZU IWAMURA, MICHIFUMI YOSHIOKA: "Word-level sign language recognition with Multi-stream Neural Networks Focusing on Local Region", IEICE TECHNICAL REPORT, PRMU, vol. 120, no. 409 (PRMU2020-78), 25 February 2021 (2021-02-25), JP, pages 53 - 58, XP009541632 *

Also Published As

Publication number Publication date
JP2022185837A (ja) 2022-12-15
JP7663178B2 (ja) 2025-04-16

Similar Documents

Publication Publication Date Title
US12039508B2 (en) Information processing system
RU2727084C1 (ru) Устройство и способ определения информации о заказе
US20200202163A1 (en) Target positioning system and target positioning method
US10290031B2 (en) Method and system for automated retail checkout using context recognition
US20190244161A1 (en) Inventory control
WO2022254958A1 (ja) 無人店舗の商品を管理する管理システム、管理方法及びプログラム
RU2739542C1 (ru) Система автоматической регистрации для торговой точки
US20210304291A1 (en) Identifying objects of interest for handicapped individuals based on eye movement patterns
KR102254639B1 (ko) 매장 관리 시스템 및 방법
JP7565023B2 (ja) 無人店舗の商品を管理する管理サーバ及び管理方法
EP4125059B1 (en) Information processing program, information processing method, and information processing apparatus
US20120130867A1 (en) Commodity information providing system and commodity information providing method
EP4125056B1 (en) Information processing program, information processing method, and information processing device
US12462565B2 (en) Computer-readable recording medium, estimation method, and estimation device
US12412148B2 (en) Cart-based availability determination for an online concierge system
US20240320701A1 (en) Wearable device, information processing method, non-transitory computer readable recording medium storing information processing program, and information providing system
JP7608581B2 (ja) 情報提供装置及びその制御プログラム
JP6598321B1 (ja) 情報処理装置、制御方法、及びプログラム
GB2530770A (en) System and method for monitoring display unit compliance
JP2021189691A (ja) 管理装置および商品棚
US20170278112A1 (en) Information processing apparatus, information processing method, and non-transitory computer readable medium
JP2025082224A (ja) 商品購入決済システムおよび商品購入決済方法
HK1257385A1 (zh) 图像识别方法及装置和电子设备
HK1242023A1 (en) Method and apparatus for determining order information
HK1242023A (en) Method and apparatus for determining order information

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22815715

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.03.2024)

122 Ep: pct application non-entry in european phase

Ref document number: 22815715

Country of ref document: EP

Kind code of ref document: A1