WO2021125357A1 - 情報処理システム - Google Patents

情報処理システム Download PDF

Info

Publication number
WO2021125357A1
WO2021125357A1 PCT/JP2020/047765 JP2020047765W WO2021125357A1 WO 2021125357 A1 WO2021125357 A1 WO 2021125357A1 JP 2020047765 W JP2020047765 W JP 2020047765W WO 2021125357 A1 WO2021125357 A1 WO 2021125357A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
moving object
information
unit
settlement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/047765
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智紀 阿久津
敏也 波川
英揮 川端
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Touch to Go Co Ltd
Original Assignee
Touch to Go Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Touch to Go Co Ltd filed Critical Touch to Go Co Ltd
Priority to JP2021565702A priority Critical patent/JP7225434B2/ja
Priority to EP20900801.0A priority patent/EP4075399A4/en
Priority to US17/786,817 priority patent/US20230027382A1/en
Publication of WO2021125357A1 publication Critical patent/WO2021125357A1/ja
Anticipated expiration legal-status Critical
Priority to JP2022190789A priority patent/JP2023014207A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G3/00—Alarm indicators, e.g. bells
    • G07G3/003—Anti-theft control
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • G06Q20/203—Inventory monitoring
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • G06Q20/206—Point-of-sale [POS] network systems comprising security or operator identification provisions, e.g. password entry
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q20/00—Payment architectures, schemes or protocols
    • G06Q20/08—Payment architectures
    • G06Q20/20—Point-of-sale [POS] network systems
    • G06Q20/208—Input by product or record sensing, e.g. weighing or scanner processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0639—Locating goods or services, e.g. based on physical position of the goods or services within a shopping facility
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/0054—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles
    • G07G1/0063—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/0036—Checkout procedures
    • G07G1/0045—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader
    • G07G1/0054—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles
    • G07G1/0072—Checkout procedures with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader with control of supplementary check-parameters, e.g. weight or number of articles with means for detecting the weight of the article of which the code is read, for the verification of the registration
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G1/00—Cash registers
    • G07G1/12—Cash registers electronically operated
    • G07G1/14—Systems including one or more distant stations co-operating with a central processing unit
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07G—REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS
    • G07G3/00—Alarm indicators, e.g. bells
    • G07G3/006—False operation

Definitions

  • the present invention relates to an information processing system.
  • the present invention has been made in view of such a situation.
  • the purpose is to automate the settlement of the product price when the purchaser purchases the product displayed in the store, shorten the time required to settle the product price, and prevent fraud by the purchaser or the cashier. To do.
  • the information processing system of one aspect of the present invention is In an information processing system used when people move and purchase products
  • a moving object tracking means for tracking a moving object, using the person or an object moving with the person as a moving object.
  • a product identification means for identifying a product by a predetermined method using at least one of a weight sensor and a camera.
  • An associating means for associating the specified product with the moving object having a predetermined positional relationship. To be equipped.
  • the payment of the product is automated, the time required for the payment of the product is shortened, and fraud prevention by the purchaser or the cashier is prevented. Can be planned.
  • FIG. 5 It is a block diagram which shows the hardware structure of the settlement gate among the structure which concerns on 1st Embodiment of the information processing system of this invention.
  • a functional block showing an example of a functional configuration of the server of FIG. 5, the sales floor device of FIG. 6, the settlement gate of FIG. 7, and the inspection terminal. It is a figure.
  • it is a functional block diagram showing a detailed functional configuration example of a moving object tracking unit provided in the sales floor device of FIG.
  • FIG. 8 It is a flowchart explaining the automatic settlement process executed by the server of FIG. 8, the sales floor apparatus, the settlement gate, and the inspection terminal in the configuration which concerns on 1st Embodiment of the information processing system of this invention. It is a flowchart explaining the automatic settlement process executed by the server of FIG.
  • FIG. 8 shows the sales floor apparatus, the settlement gate, and the inspection terminal in the configuration which concerns on 1st Embodiment of the information processing system of this invention.
  • FIG. 1 shows the layout example of the supermarket which adopts the information processing system of FIG. 1, and is the figure which shows the example different from FIG.
  • FIG. 1 is a block diagram which shows the hardware structure of the sales floor apparatus in the structure which concerns on 2nd Embodiment of the information processing system of this invention.
  • a functional block showing an example of a functional configuration of the server of FIG. 5, the sales floor device of FIG. 6, the settlement gate of FIG. 7, and the inspection terminal. It is a figure.
  • FIG. 5 is a flowchart illustrating an automatic settlement process executed by the server, the sales floor device, the settlement gate, and the inspection terminal in FIG. 15 in the configuration according to the second embodiment of the information processing system of the present invention.
  • FIG. 5 is a flowchart illustrating an automatic settlement process executed by the server, the sales floor device, the settlement gate, and the inspection terminal in FIG. 15 in the configuration according to the second embodiment of the information processing system of the present invention.
  • the information processing system according to the first embodiment of the present invention is applied as a system for automatically paying for goods.
  • FIG. 1 is a diagram showing the main points of the information processing system according to the first embodiment of the present invention.
  • the implementation store of one embodiment of the present invention is described.
  • this implementation store is merely an example, and the application destination of one embodiment of the present invention is not particularly limited.
  • the "payment place” column of FIG. 1 the place where the shopper makes the payment in one embodiment of the present invention is described.
  • the checkout location in one embodiment of the present invention exemplifies a checkout gate.
  • the checkout gate has a function of paying for the goods without placing the goods already identified as goods from the object on the cash register.
  • the "Summary” column of FIG. 1 an outline of one embodiment of the present invention is described.
  • the “Details” column of FIG. 1 details of one embodiment of the present invention are described.
  • an image captured by an image sensor such as a camera is hereinafter referred to as an “captured image”.
  • a captured image including an object as a subject is hereinafter referred to as an "object captured image”.
  • a captured image including a product as a subject is hereinafter referred to as a "product captured image”.
  • the object captured image and the product captured image for example, an image obtained by individually cutting out the product, an image of the product logo, an image of the product bar code, an image of the product label, an image captured of the entire shelf of the product, and the like.
  • An image of the inside of the store which is an image of the inside of the store from the ceiling or the like, can be used.
  • the product when the product is settled, the product cannot be specified as a product or the product whose sales are restricted can be specified or sold by the eye examiner.
  • the determination may be made. Specifically, for example, information about an object or a product is transmitted to a visual inspection terminal. Then, the eye examiner identifies the product and determines the trading restriction (determination of whether or not to release the trading restriction, etc.), and the determination result is transmitted from the visual inspection terminal. At this time, the clerk may respond on the spot in response to the result of the determination to lift the restriction on the sale of the restricted product, or the result of the inspection may be adopted as the system.
  • the product may be specified and the settlement may be performed by appropriately referring to the result of the eye inspection by the eye examiner.
  • FIG. 2 is a schematic diagram showing an outline of the flow of the hybrid check system (visual inspection) of the information processing system of FIG.
  • the product is recognized and the product is specified in the store (inside the sales floor), and (2) it is determined that the product cannot be specified as a product or the product is a restricted product.
  • Related information about these objects or products is transmitted to the inspection terminal.
  • the related information about the object or the product the information obtained by various sensing devices corresponds. Specific examples include, for example, an object captured image, a product captured image, a log of recognition results, a product candidate list, position information on which the object captured image is captured, weight information detected by a weight sensor, position information of fixtures such as shelves, and shelves.
  • the object captured image is adopted as the related information here, the object captured image of a plurality of frames may be transmitted to the visual inspection terminal. Then, (4) the request for the eye examination (reception of related information regarding the object of the eye examination) is notified on the terminal for the eye examination, and (5) the eye examiner executes the eye examination.
  • the result of the inspection by the eye examiner (the result of specifying the product or the result of the judgment of the trading restriction, etc.) is transmitted to the sales floor device installed in the sales floor, and (7) the inspection result is referred to in the sales floor device. Will be done. At this time, it is also possible to notify the sales floor device that the product is a restricted product. If the result of unspecified product is returned even after the inspection, or if the information that contributes to the decision to lift the sales restriction of the restricted product is insufficient, the purchaser is notified by the screen display etc.
  • a pre-recorded voice may be played.
  • the purchaser and the eye examiner or the like may talk with each other via the microphone and the speaker provided in the sales floor device or the like.
  • an embodiment of an information processing system in which such a flow is executed can be realized as the following information processing system.
  • the information processing system according to the embodiment of the present invention is applied to retail stores such as supermarkets. Specifically, not only shoppers and shopping carts and carts placed in supermarkets, but also baskets including shoppers' my bags and shopping bags and shoppers are recognized and tracked as moving objects Mo. .. Then, when the object is taken from the shelf, the product is recognized and specified, and the specified product is linked to the moving object so that the product can be automatically settled at the cash register even if the product is not placed on the cash register terminal. To do. At this time, according to the set conditions, the object taken from the shelf is transferred to a sensing device such as a camera installed in the sales floor and an eye inspection terminal connected via a network. It may be confirmed by an eye examiner. Then, the information processing system of the embodiment of the present invention may further refer to this confirmation result (visual inspection result) to specify the product.
  • a sensing device such as a camera installed in the sales floor and an eye inspection terminal connected via a network. It may be confirmed by an eye examiner.
  • the moving object Mo will be described below as a shopper.
  • the passage 44 including the settlement gates 5-1 to 5-3 is defined as the settlement area 45.
  • FIG. 3 is a diagram showing a layout example of a supermarket that employs the information processing system of FIG.
  • the information processing system of one embodiment of the present invention is applied to a store 40 such as a supermarket as shown in FIG.
  • the information processing system of the embodiment of the present invention is adapted so that the product can be automatically settled only by passing through the checkout gates 5-1 to 5-3 without being placed on the cash register.
  • a shopping cart, a shopping cart, etc. (without numbering) are placed at the entrance 41.
  • a checkout gate 5-1 to 5-3 and a checkout register 6 operated by a clerk Mt are installed in front of the exit 42.
  • a plurality of shelf racks 43 for displaying products are installed between the entrance 41 and the exit 42.
  • a plurality of shelves are arranged in the vertical direction at predetermined intervals, and various products are displayed on each of the plurality of shelves.
  • the space between the shelves is also described as "inside the shelves”.
  • a passage 44 is defined between the shelf rack 43 and the shelf rack 43 facing each other in the horizontal direction.
  • the moving object Mo enters the store from the entrance 41, takes the shopping cart, pushes the shopping cart, or advances the passage 44 with the brought my bag.
  • the moving object Mo takes the goods in the shelf and proceeds through the passage 44.
  • the moving object Mo takes all the products to be purchased, it proceeds to the settlement area 45 and setstles.
  • a clerk Mt is looking around in the aisle 44, the checkout area 45, and the like.
  • the clerk Mt has an information terminal 9a.
  • the information terminal 9a is a portable information processing terminal such as a smartphone, and is provided with a function of displaying the state in the store, a function of displaying an error state generated in the store, a remote control function, and the like.
  • FIG. 3 inside the line diagram drawn in a cloud shape, the state of the outside of the store 40 and the backyard of the store 40 is drawn instead of the inside of the store 40.
  • a server 1 (FIG. 5) is installed outside the store 40 or in the backyard of the store. Outside the store or in the backyard, the clerk Mt can monitor the inside of the store 40 through the screen of a large monitor (not shown) or the screen of the information terminal 9b.
  • a plurality of ceiling cameras 310 are installed on the ceiling of the passage 44 between the entrance 41 and the exit 42 as an example of a sensing device. Further, as another example of the sensing device, a shelf camera 311 is installed at each of a plurality of locations on each shelf in each shelf rack 43. The shelf camera 311 images the inside of the shelf, the products in the shelf, and other predetermined areas. Further, the shelf camera 311 uses the hand or the object in the shelf as an object captured image when the hand or the like of the moving object Mo enters the predetermined area in the shelf or when the object is taken from the shelf. Take an image.
  • a camera capable of acquiring a 3D image or a 2D image, a camera capable of capturing a wide angle represented by a fisheye camera, or the like is adopted.
  • a weight sensor 312 is installed at each of the plurality of shelves in each shelf rack 43 as another example of the sensing device. The weight sensor 312 detects the weight of the shelves on which the goods can be placed. Therefore, the entry / exit of the product from the shelf is detected from the difference in the change in the weight detected by the weight sensor 312.
  • the camera and the weight sensor are merely examples of sensing devices installed on the ceiling and each shelf rack, and are not limited thereto.
  • sensing devices including a 3D sensor, a temperature sensor, a distance sensor, a sensor utilizing infrared rays, and the like can be installed. Further, a plurality of these sensing devices such as a weight sensor may be installed on one shelf on which a product can be placed.
  • FIG. 4 is a configuration diagram showing the configuration of the information processing system of FIG.
  • the information processing system as an embodiment of the information processing system of the present invention has a configuration as shown in FIG.
  • the information processing system includes a server 1, a sales floor device 3, n units (n is an arbitrary integer) of settlement gates 5-1 to 5-n, and a visual inspection terminal Q.
  • the server 1, the sales floor device 3, the settlement gates 5-1 to 5-n, and the inspection terminal Q are each connected to each other via a network N such as an Internet line.
  • N such as an Internet line.
  • FIG. 4 For convenience of explanation, only one server 1 in FIG. 4 is drawn, but in reality, there may be a plurality of servers 1. Further, hereinafter, when it is not necessary to individually distinguish the settlement gates 5-1 to 5-n, these are collectively referred to as "payment gate 5".
  • FIG. 5 is a block diagram showing a hardware configuration of a server among the configurations according to the first embodiment of the information processing system of the present invention.
  • the server 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a bus 104, an input / output interface 105, an output unit 106, and an input unit 107. , A storage unit 108, a communication unit 109, and a drive 110.
  • CPU Central Processing Unit
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the CPU 101 executes various processes according to the program stored in the ROM 102 or the program loaded from the storage unit 108 into the RAM 103. Data and the like necessary for the CPU 101 to execute various processes are also appropriately stored in the RAM 103.
  • the CPU 101, ROM 102 and RAM 103 are connected to each other via the bus 104.
  • An input / output interface 105 is also connected to the bus 104.
  • An output unit 106, an input unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.
  • the output unit 106 is composed of a display, a speaker, and the like, and outputs various information as images and sounds.
  • the input unit 107 is composed of a keyboard, a mouse, and the like, and inputs various information.
  • the storage unit 108 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various data. As shown in FIG. 4, the communication unit 109 communicates with the sales floor device 3 and the settlement gate 5 via the network N including the Internet line.
  • DRAM Dynamic Random Access Memory
  • a removable media 120 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 110.
  • the program read from the removable media 120 by the drive 110 is installed in the storage unit 108 as needed. Further, the removable media 120 can also store various data stored in the storage unit 108 in the same manner as the storage unit 108.
  • FIG. 6 is a block diagram showing a hardware configuration of a sales floor device among the configurations according to the first embodiment of the information processing system of the present invention.
  • the sales floor device 3 includes the CPU 301, the ROM 302, the RAM 303, the bus 304, the input / output interface 305, the ceiling camera 310, the shelf camera 311 and the weight sensor 312, the information terminal 9, and the communication unit 315. It has.
  • the CPU 301, ROM 302, RAM 303, bus 304, input / output interface 305, and communication unit 315 of the sales floor device 3 are configured in the same manner as those of the server 1 shown in FIG.
  • the ceiling camera 310, the shelf camera 311 and the weight sensor 312 are connected to the network.
  • it may be connected by a PoE connection (Power over Ethernet) or a USB (Universal Serial Bus) cable.
  • PoE connection Power over Ethernet
  • USB Universal Serial Bus
  • the information terminal 9 is an information device such as a smartphone or tablet provided with a remote control unit 390, a display unit 391, and the like.
  • the remote control unit 390 has a function of remotely controlling an error state such as a system processing abnormality.
  • the display unit 391 includes a screen for displaying an error status, a moving object Mo, or the like. Further, the information terminal 9 includes a voice generating unit (not shown) for notifying an error state.
  • the error state in one embodiment of the present invention occurs under various circumstances. For example, when an abnormality occurs in the system processing, when the product cannot be specified in the shelf product recognition, or when an unspecified object or a moving object Mo associated with a trade-restricted product tries to pass through the gate 5. There are cases where a moving object Mo that has not been settled is about to leave the store.
  • a settlement gate 5 is connected to the sales floor device 3 via a network N.
  • the settlement gate 5 is divided into a settlement gate 5-1 equipped with a settlement machine 5a, a settlement gate 5-2 using electronic money, and a settlement gate 5-3 that can be settled just by passing through.
  • the checkout gates 5-1 to 5-3 other than the manned checkout register 6 may be provided with an opening / closing member (without numbering) that is always closed.
  • the total amount of the purchased products is calculated. Then, the person who makes the payment can make a payment with the payment machine 5a installed on the exit 42 side.
  • the settlement machine 5a is installed on the exit 42 side of the settlement gate 5-1.
  • the payment machine 5a can pay with cash, credit card, electronic money, point payment, gift certificate, virtual currency, prepaid card, identification information such as barcode and QR code (registered trademark), employee ID card, application information, etc. It has a means of settlement.
  • the person who makes the payment can settle by holding various cards over the card reading unit (not shown) provided in the gate body to read the information such as electronic money.
  • Electronic money includes not only IC cards that can be settled, but also credit cards, so-called point cards, prepaid cards, and other narrowly defined cards, as well as information mobile terminals that can display barcodes and QR codes (registered trademarks), for example. Although it is included, it will be described here as various cards for convenience. That is, when a person who makes a payment enters the payment area 45 of the payment gate 5-2 and holds various cards over the card reading unit, the product information associated with the moving object Mo, which will be described later, is read out, and the payment and payment are made. Is completed. Then, the person who makes the payment can pass through the settlement gate 5-2.
  • the settlement and settlement are completed only when the moving object Mo whose personal information has been acquired at the entrance 41, the passage 44, or the like passes through the settlement gate 5-3. That is, the person who makes the payment can complete the payment and the payment without paying cash or having the card read.
  • the clerk Mt individually inputs the price of the product and setstles the checkout.
  • FIG. 7 is a block diagram showing a hardware configuration of a settlement gate among the configurations according to the first embodiment of the information processing system of the present invention.
  • the settlement gate 5 includes a CPU 501, a ROM 502, a RAM 503, a bus 504, an input / output interface 505, an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509.
  • the CPU 501, ROM 502, RAM 503, bus 504, input / output interface 505, storage unit 508, and communication unit 509 of the settlement gate 5 are configured in the same manner as those of the server 1 shown in FIG.
  • the input unit 506 is a settlement button provided in the gate body, for example, in the settlement gate 5-1. Further, the settlement gate 5-2 using electronic money and the settlement gate 5-3 that can be settled only by passing through are information reading units that detect information such as various cards and various identification information.
  • the output unit 507 outputs a signal for opening / closing the opening / closing member (not shown) provided in the settlement gates 5-1 to 5-3. Further, the settlement gate 5-1 provided with the settlement machine 5a outputs the amount of money, the product name, etc. to be settled to the settlement machine 5a.
  • the code scanner unit 510 is provided in the settlement gates 5-1 to 5-3.
  • the moving object Mo reads the product identification information using a barcode reader (not shown) or the like to read the product. Can be identified.
  • FIG. 8 shows an example of the functional configuration of the server of FIG. 5, the sales floor device of FIG. 6, the settlement gate of FIG. 7, and the inspection terminal among the configurations according to the first embodiment of the information processing system of the present invention. It is a functional block diagram which shows.
  • the server 1 includes a CPU 101, a storage unit 108, an error display unit 151, and an error release unit 152.
  • the DB management unit 141 that manages personal information, product information, and the like functions.
  • the DB management unit 141 may be configured to be provided for each of the settlement gates.
  • the CPU 101 of the server 1 includes an error determination unit 150.
  • a product DB 131 and a location information management DB 132 are provided in one area of the storage unit 108 of the server 1.
  • the product DB 131 is a DB (Data Base) that stores information about the product.
  • the position information management DB 132 manages the position of the moving object Mo.
  • the image display control unit 411 and the visual inspection result transmission unit 412 function.
  • the image display control unit 411 relates to an object or a product transmitted from the sales floor device or the settlement gate 5 when a visual inspection request is made from the sales floor device 3 or the settlement gate 5.
  • the information is output to the output unit 406.
  • the inspection result transmission unit 412 receives the inspection result (determination of the specific result of the product or the trade-restricted product) input by the eye examiner via the input unit 407 with respect to the related information about the object or the product output to the output unit 406.
  • the result, etc. is transmitted to the sales floor device 3 or the settlement gate 5 for which the inspection is requested.
  • the personal authentication unit 320 In the CPU 301 of the sales floor device 3, the personal authentication unit 320, the moving object tracking unit 330, the position information management unit 340, the shelf product recognition unit 360, the trading restricted product determination unit 380, and the inspection result acquisition unit 301a are included. Works.
  • the personal authentication unit 320 personally authenticates who the moving object Mo registered in the DB management unit 141 of the server 1 is from the personal information of the moving object Mo acquired by the personal information acquisition unit 321.
  • the personal authentication unit 320 and the personal information acquisition unit 321 may be provided at the entrance 41.
  • the personal information acquisition unit 321 employs a reading device that touches a mobile information terminal such as an IC card, a smartphone, or a tablet of a moving object Mo, a reading device that reads biological information such as fingerprints, veins, and irises.
  • personal information includes not only personally identifiable information such as name, gender, date of birth, address, and telephone number, but also biometric information such as fingerprints, veins, and irises, and finance such as credit card numbers and bank account numbers.
  • Information on privacy such as information on the moving object Mo and purchase history of the moving object Mo are also included.
  • the personal authentication unit 320 may perform personal authentication based on an image of a moving object Mo obtained from a camera or the like during shopping.
  • the personal information acquired here is also used for trading restrictions (including cancellation) and purchase analysis.
  • FIG. 9 is a functional block diagram showing a detailed functional configuration example of the moving object tracking unit provided in the sales floor device of FIG. 6 among the configurations according to the first embodiment of the information processing system of the present invention.
  • the moving object tracking unit 330 includes the moving object detection unit 3302 by the ceiling camera, the moving object area definition unit 3304 by the ceiling camera, and the movement by the ceiling camera.
  • the product identification unit 3313 that has been created functions.
  • the moving object discovery unit 3302 by the ceiling camera discovers the moving object Mo by using a state space model (Bayesian filter or the like) based on the captured image captured by the ceiling camera 310.
  • the moving object area definition unit 3304 by the ceiling camera defines the area of the moving object Mo discovered by the moving object finding unit 3302 by the ceiling camera as the moving object area.
  • the moving object area definition unit 3304 by the ceiling camera defines the moving object area by continuously finding the changed area around the moving object Mo. That is, the moving object area definition unit 3304 by the ceiling camera defines a certain range of the discovered moving object Mo, the moving object Mo, and the periphery of the moving object Mo as a moving object area.
  • the human area is a subordinate concept of the moving object area.
  • the basket area is a subordinate concept of the moving object area.
  • the cart area is a subordinate concept of the moving object area.
  • the moving object Mo is a person
  • the approximate pose is estimated and compared with the human area in the actually obtained image
  • the moving object Mo The person who is a barrel may be clearly defined.
  • the human area can be recognized with high accuracy by estimating each part such as the head, shoulders, arms, hands, and feet from the 3D data of the person.
  • deep learning may be used to estimate, for example, a human area, a basket area, and a cart area from captured images, 3D data, and the like.
  • the moving object area tracking unit 3305 by the ceiling camera tracks the movement of the moving object Mo.
  • the moving object area tracking unit 3305 using the ceiling camera tracks the movement of the moving object Mo by collecting the feature data (color, shape, etc.) of the moving object Mo.
  • the moving object area tracking unit 3305 by the ceiling camera tracks the movement of the moving object Mo by using an object tracking technique in an image such as a Bayesian filter, a fast Fourier transform, and a TLD (Tracking-Learning-Dection).
  • an object tracking technique in an image such as a Bayesian filter, a fast Fourier transform, and a TLD (Tracking-Learning-Dection).
  • the moving object Mo that does not face the ceiling camera 310 is imaged at an angle (from an oblique direction), so that the position information may not be accurately acquired. Therefore, it is conceivable that the captured image is imaged as if it is facing each other by correcting it by calibration. However, even if such correction is applied, it may not be possible to acquire the position information of the
  • the moving object area tracking unit 3305 using the ceiling camera can acquire highly accurate position information by acquiring the height information of the moving object Mo using a distance sensor or the like. In this way, the moving object area tracking unit 3305 by the ceiling camera may continue to track the moving object Mo.
  • the information obtained by the ceiling camera is used for the discovery, definition, and tracking of the moving object Mo, but the present invention is not particularly limited.
  • information obtained by various sensing devices such as a 3D sensor, that is, time series data such as 3D data and image data may be used for finding, defining, and tracking a moving object Mo. ..
  • the grouping unit 3308 groups the plurality of people. That is, the grouping unit 3308 groups a plurality of moving objects Mo by using information such as position information (for example, a sense of distance such as overlapping or sticking) in a plurality of moving objects Mo and information such as a moving direction (vector). May be good. Further, the grouping unit 3308 may link the person area, the basket area, and the cart area. By functioning the grouping unit 3308 in this way, for example, at the settlement gate 5, product information associated with a plurality of moving objects Mo can be collectively settled.
  • the blood relationship determination unit 3309 uses a face recognition method to distinguish between parent-child relationships and sibling relationships.
  • the kinship determination unit assists the function of the grouping unit.
  • the kinship determination unit may determine the degree of face similarity using a deep learning face recognition method and estimate the kinship.
  • the transfer recognition unit 3311 between the moving object regions by the ceiling camera recognizes that the object has been transferred from the moving object Mo to the moving object Mo, and has passed / passed.
  • the moving object Mo is specified, and the product list associated with each moving object Mo is read.
  • the object recognition unit 3312 delivered by the ceiling camera then defines the area of the object from the captured image at the time when the delivery is recognized. Furthermore, the object delivered by the product identification unit delivered by the ceiling camera is one of the products in the product list linked to the moving object Mo that has been read and delivered from the image after the object area is defined. Each moving object Mo specified by the moving object area delivery recognition unit 3311 by the ceiling camera is associated with the product specified by the delivery, and the list of each product is updated.
  • the transfer recognition unit 3311 between moving object areas by the ceiling camera may analyze the movement of the moving object Mo by using an object recognition method such as deep learning and recognize the transfer. May be recognized, and the recognition of delivery may recognize the overlap between human domains (which may include hands).
  • the object recognition unit 3312 delivered by the ceiling camera defines the area of the object from the captured image at the time when the delivery is recognized.
  • the product identification unit 3313 delivered by the ceiling camera associates the object recognized by the object recognition unit 3312 delivered by the ceiling camera with the read moving object Mo (the person who delivered the product).
  • Each product in the product list is specified, and each moving object Mo specified by the moving object area transfer recognition unit 3311 by the ceiling camera is linked to the product specified by the product specifying unit 3313 delivered by the ceiling camera. Then, the list of products of each moving object Mo is updated.
  • the object recognition unit 3312 delivered by the ceiling camera may make the ceiling camera a zoomable camera, zoom in on the portion presumed to have been delivered, and define the area of the object. Further, here, in the moving object area transfer recognition unit 3311 by the ceiling camera, the object recognition unit 3312 delivered by the ceiling camera, and the product identification unit 3313 delivered by the ceiling camera, other parts are used instead of the ceiling camera.
  • a sensing device may be used, and for example, a camera capable of capturing a wide range may be adopted.
  • FIG. 10 is a functional block diagram showing a detailed functional configuration example of the position information management unit provided in the sales floor device of FIG. 6 among the configurations according to the first embodiment of the information processing system of the present invention.
  • the position information management unit 340 functions as an inter-camera information transfer unit 341, a position definition unit 342 of each camera, and a moving object display unit 343.
  • the inter-camera information transfer unit 341 shares the image information captured by each ceiling camera 310 with the image information captured by another ceiling camera 310, so that the captured image of the ceiling camera 310 having the moving object Mo is present.
  • the moving object area can be continuously tracked even if the image is taken by another ceiling camera 310.
  • the inter-camera information transfer unit 341 exchanges information between the ceiling cameras 310 on the storage unit 108 including the product DB 131 through, for example, the server 1 that controls the information captured and obtained by the ceiling camera 310.
  • the inter-camera information transfer unit 341 receives an image captured by each ceiling camera 310 between the ceiling cameras 310 by, for example, P2P, without passing through the server 1. You may pass it.
  • the position definition unit 342 of each camera defines the position information of where each ceiling camera 310 is projecting in the store. That is, the position definition unit 342 of each camera grasps where in the store the moving object imaged by the separate ceiling cameras 310 by the inter-camera information transfer unit 341 is located. The position definition unit 342 of each camera synthesizes the captured images of each ceiling camera 310 to create one store map. Further, the position definition unit 342 of each camera replaces the coordinates of each ceiling camera 310 and the shelf camera 311 with the coordinates on the store map. Further, the position definition unit 342 of each camera corrects the captured image captured by each ceiling camera 310 so as to be a captured image facing the floor surface in the store by perspective conversion.
  • the moving object display unit 343 displays the moving object Mo in the store 30 as the position information captured by the position definition unit 342 of each camera.
  • the moving object display unit 343 may be adopted as an information terminal 9 held by the store clerk Mt, a screen of the backyard of the store, or the like.
  • the position information of the moving object Mo obtained in this way is managed by the position information management DB 132, a memory, or the like. It should be noted that the interest, interest, etc. of a person can be inferred from the position information of the moving object Mo, that is, the information related to the behavior history of the person. Therefore, the position information of the moving object Mo managed by the position information management DB 132, the memory, or the like can be used for marketing analysis or the like.
  • FIG. 11 is a functional block diagram showing a detailed functional configuration example of the shelf product recognition unit provided in the sales floor device of FIG. 6 among the configurations according to the first embodiment of the information processing system of the present invention.
  • the shelf product recognition unit 360 includes an object recognition unit 3602 shown in FIG. 11, a product identification unit 3603, a moving object and a product linking unit 3604, a product list management unit 3606 linked to a moving object, and an object entry / exit detection unit 3608.
  • a product unspecified determination unit 3609, a label recognition unit 3610, and a discount seal recognition unit 3611 are provided.
  • the shelf product recognition unit 360 is linked with another camera or a personal computer through the shelf camera 311 and the weight sensor 312 and the network N.
  • the object recognition unit 3602 compares the images before and after the image in which the object is taken from the shelf or the object is placed (returned) in the shelf to identify the product. Define the image area that is the target of (area definition). That is, the object recognition unit 3602 changes between the object captured image before the object is taken from the shelf or placed on the shelf and the object captured image after the object is taken from the shelf or placed on the shelf. By comparing the captured images of the objects in the above, the changed image area is identified. The object recognition unit 3602 confirms the change in each of the RGB data when specifying the image area.
  • the object recognition unit 3602 confirms the change in the weight, and for example, when the weight decreases, it detects that the object has been taken from the shelf, while the weight increases. If so, it is detected that the object has been placed (returned) on the shelf.
  • the object recognition unit 3602 uses the object entry / exit detection unit 3608 as a trigger to capture an image of the object before the object is taken from the shelf or placed on the shelf, and the object is on the shelf.
  • An object region may be defined from only one object captured image using a method different from the comparison before and after the change of the object captured image after being taken from the inside or placed on the shelf.
  • the object recognition unit 3602 also uses the shadow of the object to create an area so that it is not determined that there is no change even if there is a change such as entering and exiting. It may be defined.
  • the information regarding the fact that the object recognized in this way is taken from the shelf or placed on the shelf is handed over to the product identification unit 3603.
  • a camera or a weight sensor was used to recognize that the item was taken from the shelf or placed on the shelf, but the method is not limited to this.
  • the method is not limited to this.
  • the product identification unit 3603 identifies which product the object in the shelf recognized by the object recognition unit 3602 is.
  • the product identification unit 3603 lists product candidates by image processing methods such as specific object recognition, general object recognition, and deep learning.
  • the listed product candidates are called "product candidate list S”. After that, the product identification unit 3603 identifies the product with high accuracy by exerting the verification function.
  • the verification function lists the "product candidate list P" by a method different from the above-mentioned method of listing the product candidates.
  • the results of the product candidate lists S and P are matched, and if the result exceeds a predetermined threshold value, the product is specified.
  • a method for listing the "product candidate list” for example, it may be realized by a method of matching the image information of the object obtained from the object whose existence is recognized with the image information stored in the product DB 131 or the memory. ..
  • the object recognition unit 3602 when the feature information of both images matches (exceeds the threshold value), the object whose existence is recognized by the object recognition unit 3602 is recognized as a product registered in the product DB 131, and the product identification unit 3603 recognizes that the product is registered in the product DB 131. It is specified that the product is registered in the product DB 131. In this way, product candidates are created by deep learning, and then the verification function is exerted to identify the products with high accuracy.
  • the product identification unit 3603 does not specify the product in one frame of the captured image captured by the shelf camera 311 but also makes full use of the captured image captured by the ceiling camera 310 as another sensing device to capture a plurality of images.
  • the product may be identified across the image.
  • the weight list for each product in which the weight for each product is recorded or the "position information list for each product” in which the position of the shelf on which the product is installed is recorded. "Is prepared in advance.
  • the product identification unit 3603 uses the "weight list for each product" it matches the amount of change in the weight in the shelf with, for example, when the product or the product exceeds a predetermined threshold value. Specify the number.
  • the weight for each product and the threshold value for matching recorded in the "weight list for each product” may be determined by a method such as statistical analysis or deep learning. As a result, even if the weight of the product varies, the product can be identified with high accuracy.
  • the product identification unit 3603 uses the "position information list for each product"
  • the product identification unit 3603 specifies the product based on the "position information list for each product” and the information regarding the position of the shelf where the weight has changed.
  • the product identification unit 3603 may specify both the product and the number of products by using the "weight list for each product” and the "position information list for each product” in combination for matching.
  • the information obtained by the camera or the weight sensor is used for identifying the product, but the information is not limited to this information. That is, the product may be specified based on the information acquired by other sensing devices. Further, any one method may be adopted from among them, or a plurality of methods may be combined and adopted. Specifically, for example, the product may be specified by combining the image information of the product obtained from the shelf camera 311 and the weight information and the position information of the product obtained from the weight sensor 312. At this time, the threshold value in matching may be arbitrarily set, and the threshold value may refer to, for example, information such as purchase history, time, place, and person's preference.
  • an arbitrary matching algorithm may be created by combining the information acquired by deep learning by a plurality of neural networks, the information acquired by other sensing devices, and the threshold values set arbitrarily for each. In this way, when a plurality of methods are combined, the specific accuracy of each method is complemented, and the product can be specified with higher accuracy.
  • the product identification unit 3603 specifies the number of feature points in which the image of the object taken from the shelf and the image of the object placed on the shelf are similar. For example, the sizes of the objects captured in both images are specified and compared, the color difference is specified, and the colors are compared to determine whether or not the two objects are similar. If the number of feature points is small, the product may be unspecified.
  • the product identification unit 3603 determines that the product is unspecified when, for example, the change in weight before and after the product enters and exits deviates from the threshold value of the position information and the weight of the product associated with the product. May be good. Furthermore, using an arbitrary matching algorithm created by combining the information acquired by deep learning by the above-mentioned multiple neural networks and the information obtained by a plurality of sensing devices, for example, all the obtained results have a predetermined threshold value. If it exceeds, the product may be specified, and if it does not exceed, the product may not be specified.
  • the moving object and the product linking unit 3604 associate the product information of the product specified by the product specifying unit 3603 with the moving object Mo.
  • the ceiling camera 310 or the shelf camera 311 is used for finding, defining, and tracking the moving object Mo, only one moving object Mo exists in a predetermined area imaged by one shelf camera 311. If this is the case, the moving object Mo and the product information are linked by specifying the ID of the moving object Mo.
  • the object entry / exit detection unit by the shelf camera or the ceiling camera described later moves into the moving object Mo.
  • the moving object Mo containing the object taken from the shelf is identified, and the moving object Mo and the product information are linked.
  • at least one or more shelf cameras 311 can be used to move the moving object Mo. After detecting the entry and exit and identifying the moving object Mo, the specified moving object Mo and the product information are linked.
  • the discovery, definition, and tracking of the moving object Mo are not limited to the ceiling camera 310 and the shelf camera 311, and various types of sensing devices can be used.
  • various types of sensing devices can be used.
  • the 3D data obtained from the 3D sensor information in the height direction of the moving object Mo can be obtained. That is, even when the arms and hands of the moving objects Mo overlap with each other, the moving object Mo can be tracked with high accuracy, and the link with the product can be performed with high accuracy.
  • the product list management unit 3607 associated with the moving object continues to manage the product list associated with the moving object Mo and the specified product until the settlement. That is, the product list management unit 3607 linked to the moving object constantly manages the list of products linked to the ID of the moving object Mo by utilizing the position information management DB 132 or the like.
  • the product list management unit 3606 associated with the moving object is added by the number of acquired products.
  • the product list management unit 3607 associated with the moving object subtracts the number of returned products.
  • the object entry / exit detection unit 3608 can trigger the activation of the object recognition unit 3602 by detecting that an object has entered the shelf.
  • the object entry / exit detection unit 3608 detects the entry of an object into the shelf from, for example, a change in image data in the "entry detection area" set in the captured image of each shelf camera 311. ..
  • the object entry / exit detection unit 3608 also detects that the object has left the area by tracking the incoming object in the image.
  • the object entry / exit detection unit 3608 applies a particle filter to the entry detection area in the shelf.
  • the particles are again placed in the "entry detection area”. Sow and prepare for the next object's entry.
  • the object entry / exit detection unit 3608 does not sprinkle particles on the approach detection area that covers the area where the object already exists.
  • the object entry / exit detection unit 3608 determines that the proportion of particles having a certain likelihood is equal to or greater than the threshold value in a predetermined region, and determines that the particles having a certain likelihood are less than the threshold value. It is judged as "out”.
  • the object entry / exit detection unit detects the entry / exit each time a product is entered so that the area of the object is estimated as small as possible at one time.
  • the images before and after the change are stored in the storage unit 108 of the server 1 so that they can be used in the estimation area.
  • the object entry / exit detection unit 3608 may detect the entry / exit of an object from the spectrum (wavelength) data obtained from the captured image of the object.
  • the sensing device for detecting the entry of an object into the shelf is not limited to the above-mentioned shelf camera, and various types of sensing devices can be used.
  • various sensors such as a weight sensor, a 3D sensor, a temperature sensor, a distance sensor, a sensor utilizing infrared rays, a pressure sensor, and an ethylene gas sensor can be adopted.
  • the product unspecified determination unit 3609 associates the fact that the product could not be identified by the product identification unit 3603 with the moving object Mo that took the object from the shelf.
  • the label recognition unit 3610 recognizes the attached label according to the product specified by the product identification unit 3603.
  • the label recognition unit 3610 makes full use of the techniques of image recognition and character recognition, reads a multidimensional code including characters, barcodes, etc. written on the label, and complements the product identification.
  • the discount sticker recognition unit 3611 recognizes the affixed discount sticker according to the product specified by the product identification unit 3603.
  • the discount sticker recognition unit 3611 makes full use of the techniques of image recognition and character recognition to specify the discount amount and the discount rate of the discount sticker attached to the product.
  • the discount sticker recognition unit 3611 is executed during the processing of the product identification unit.
  • the product identification unit 3603 obtains the related information obtained by various sensing devices regarding the object whose product is unspecified, via the inspection result acquisition unit 301a, and the inspection terminal. Send to Q.
  • the product identification unit 3603 examines the product identification result and the related information of the specified product, the related information of the object that is not specified as the product, and the like. It is transmitted to the eye inspection terminal Q via the acquisition unit 301a.
  • the product identification unit 3603 identifies the product based on the eye inspection result acquired by the eye inspection result acquisition unit 301a from the eye inspection terminal Q. Specifically, for the product specified by the product identification unit 3603, when the specific result is approved or modified by the inspection result, the product identification unit 3603 specifies the product according to the content indicated by the inspection result. Do. Further, for an object that is not specified as a product by the product specifying unit 3603, the product specifying unit 3603 specifies the object as a product indicated by the inspection result.
  • the product identification unit 3603 requests the product identification by visual inspection, the request may be made after the label recognition unit 3610 complements the product identification. In this case, as a result of complementing the product identification by the label recognition unit 3610, the product can be specified by visual inspection, excluding the product specified.
  • this information processing system includes a trading restricted product determination unit 380 and a remote control unit 390.
  • the trading restricted product determination unit 380 determines whether or not the specified product is a trading restricted product.
  • the restricted products include (A) products such as cigarettes and alcoholic beverages that cannot be purchased until a certain age, (B) products whose consumption period has expired or expiration date, and (C) allergens.
  • products that should not be ingested (D) products other than halal foods, and other products that are restricted by religion fall under this category. That is, the trading restricted product determination unit 380 determines from the DB information that the specified product is a trading restricted product due to age restrictions on alcoholic beverages and tobacco.
  • the trading restricted product determination unit 380 uses character recognition and the like to determine that the product is a trading restricted product due to the expiration date or expiration date.
  • the trade-restricted product determination unit 380 links the product with the personal information obtained by personal authentication, and determines that the product is a trade-restricted product other than halal foods and allergies to shoppers.
  • the trading restricted product determination unit 380 detects the trading restricted product, the information is displayed on the error display unit 151.
  • the trading restricted product determination unit 380 may transmit the related information of the product corresponding to the trading restricted product to the visual inspection terminal Q via the visual inspection result acquisition unit 301a.
  • the trading restricted product determination unit 380 appropriately sends information (face image, personal information, etc.) about the shopper to the visual inspection terminal Q via the visual inspection result acquisition unit 301a according to the type of the trading restricted product. Send. Then, the sales-restricted product determination unit 380 determines whether or not the sale of the sales-restricted product is permitted based on the inspection result acquired by the inspection result acquisition unit 301a from the inspection terminal Q (the sales restriction is released). Whether or not) can be determined.
  • the remote control unit 390 When the remote control unit 390 receives a notification of an error status such as a system processing abnormality, product unspecified, or trading restriction, the remote control unit 390 resolves the error status by remote control.
  • the ceiling camera 310 or the shelf camera 311 When the ceiling camera 310 or the shelf camera 311 is used as the sensing device, the ceiling camera 310, the shelf camera 311 and the checkout gate 5 that have detected the error state notify that the error state is in the store or outside the store via the network N. Notify the information terminals 9a and 9b for remote operation and the server 1. Then, the related information about the object or the product when the error state is detected is similarly notified to the information terminals 9a and 9b and the server 1 for remote control inside or outside the store via the network N.
  • the error state can be resolved by operating these devices, but of course, the error state may be canceled at the manned cash register in the store, and the clerk Mt or the like goes to the place where the error state occurs and the error state occurs. May be
  • the visual inspection result acquisition unit 301a requests the identification of the product by the visual inspection by transmitting the related information about the object or the product obtained by the sensing device to the visual inspection terminal Q, and in response to this request, the visual inspection terminal Acquire the inspection result sent from Q.
  • the visual inspection result acquisition unit 301a outputs the acquired visual inspection result (product specific result) to the shelf product recognition unit 360.
  • the visual inspection result acquisition unit 301a requests the determination of the trade-restricted product by the visual inspection by transmitting the product image of the product corresponding to the trade-restricted product to the visual inspection terminal Q, and the visual inspection terminal Q Get the inspection result sent from.
  • the eye inspection result acquisition unit 301a appropriately transmits information (face image, personal information, etc.) regarding the moving object Mo to the eye inspection terminal Q.
  • the visual inspection result acquisition unit 301a outputs the acquired visual inspection result (determination result of the trade-restricted product) to the trade-restricted product determination unit 380.
  • the settlement unit 535, the input control unit 536, and the output control unit 537 function.
  • the settlement unit 535 when the tracked moving object Mo enters the settlement area 45, product information (product list) associated with one or more moving objects Mo is input from the server 1 via the communication unit 509. Then, the payment amount and the product to be settled are fixed.
  • the input control unit 536 inputs a signal from an input unit 506 such as a settlement button provided in the gate body or an information reading unit.
  • the output control unit 537 causes the output unit 507 to display the settlement amount, outputs information to the settlement machine 5a, and opens / closes the opening / closing member.
  • the settlement amount and the settlement target product are determined based on the product information associated with the invading moving object Mo.
  • the settlement amount and the settlement target product are determined based on the product information associated with the moving objects Mo of the plurality of invading persons. That is, the settlement amount of a plurality of people and the settlement target product are collectively determined.
  • the product information (list of products) associated with the companion mother and child is also collectively confirmed (grouping) so that the father can make the payment. You can also do it.
  • the settlement gate 5-1 equipped with the settlement machine 5a is a device that calculates and setstles the total amount of goods.
  • this settlement gate 5-1 when a person who makes a payment invades the settlement area 45, the intrusion triggers the reading of product information associated with the moving object Mo, the settlement amount is fixed, and the settlement gate It becomes possible to pass through 5-1.
  • the opening / closing member is provided, the opening / closing member opens. Then, when the person who makes the payment makes a payment using the payment machine 5a, the store can be closed.
  • a settlement button (not shown) is installed at the settlement gate 5-1 and the settlement person presses the settlement button to read the product information associated with the moving object Mo and confirm the settlement amount. , It may be possible to pass through the settlement gate 5-1.
  • the settlement gate 5-1 keeps the opening / closing member closed to maintain the impassable state.
  • the settlement gate 5-1 may indicate that it is in an error state by means such as sound or light.
  • the restriction is released by the operation of the error canceling unit 152 by the operation of the clerk Mt, and the settlement gate 5-1 becomes a passable state.
  • the restriction can be lifted by remote control.
  • the information terminal 9a, the information terminal 9b, and the error display unit 151 are presented with an error state.
  • the settlement gate 5-1 keeps the opening / closing member closed to maintain the impassable state.
  • the settlement gate 5-1 may indicate that it is in an error state by means such as sound or light. Further, when the result that the product is unspecified is returned, the purchaser may be notified by screen display or the like on the sales floor device or the like, or the purchaser may be notified by voice.
  • the person who makes the payment enters the settlement area 45 of the settlement gate 5-2 and cards various cards including a barcode and a QR code (registered trademark).
  • the product information associated with the moving object Mo is read, and the settlement and settlement are completed. Then, the person who makes the payment can pass through the settlement gate 5-2.
  • the product to be settled is a trade-restricted product or an unspecified object is linked to the moving object Mo, the operation is the same as that of the above-mentioned settlement gate 5-1.
  • the settlement gate 5-3 it is necessary that the tracked moving object Mo has been personally authenticated and the settlement information is specified.
  • the settlement gate 5-3 is automatically settled and can pass through the settlement gate 5-3 as long as there is no error.
  • the product to be settled is a trade-restricted product or an unspecified object is linked to the moving object Mo, the operation is the same as that of the above-mentioned settlement gate 5-1.
  • the product information associated with the moving object Mo that has been confirmed and read is canceled. Then, when the person who makes the payment retires from the settlement area 45 and then enters the settlement area 45 again, the product information associated with the moving object Mo is read again, and the settlement amount and the settlement target product are determined. That is, even if the person who makes the payment once withdraws from the settlement area, purchases additional products, returns the product to the shelf, and then invades the settlement area again, the product information associated with the moving object Mo is updated and confirmed. Will be.
  • the product information is automatically read when the person who makes the payment enters the payment area 45, and (2) the person who makes the payment is installed at the payment gate 5-1. There is a case where the product information is read by pressing the settlement button.
  • the product information associated with the moving object Mo is destroyed
  • (1) the product information is automatically destroyed when the person who makes the payment leaves the settlement area 45, and (2) the payment is made.
  • the cancel button not shown installed at the checkout gate
  • the read product information is destroyed.
  • a timeout occurs. For example, the product information is automatically destroyed.
  • FIG. 12 and 13 show the automatic settlement process executed by the server, the sales floor device, the settlement gate, and the inspection terminal in FIG. 8 among the configurations according to the first embodiment of the information processing system of the present invention. It is a flowchart.
  • a ceiling camera is used for finding, area definition, and tracking of a moving object Mo
  • a shelf camera is used for object entry / exit detection
  • a shelf camera and a weight sensor are used for object recognition. It will be described as being used, but is not limited to this. As described above, it is assumed that various types of sensing devices can be adopted here as well.
  • the ceiling camera 310 installed near the entrance 41 starts imaging the moving object Mo.
  • the ceiling camera 310 in the back starts imaging the moving object Mo.
  • the plurality of ceiling cameras 310 constantly image the entire store 30 including the moving object Mo, the basket, and the cart.
  • the personal authentication unit 320 may personally authenticate the shopper who is the moving object Mo and acquire the personal information of the moving object Mo.
  • step S402 the moving object finding unit 3302 using the ceiling camera discovers the moving object Mo and assigns an individual ID.
  • the ID can be used continuously until a specific timing such as leaving the store or completing the settlement.
  • the personal authentication unit 320 may authenticate the shopper's personality and acquire the personal information of the moving object Mo.
  • the grouping unit 3308 may group a plurality of moving objects Mo as one group, or the blood relationship determination unit 3309 may determine the blood relationship of the plurality of moving objects Mo and complement the grouping unit 3308. Good.
  • the moving object area defining unit 3304 by the ceiling camera defines a predetermined area including the moving object Mo discovered by the moving object finding unit 3302 by the ceiling camera. Further, when the moving object Mo moves within the range imaged by the ceiling camera 310, the position of the region of the moving object Mo after the movement is defined again.
  • the position information is managed by the position information management DB 132, the memory, etc. in association with the ID of the moving object Mo, and is updated for each area definition. This defined position is also recognized at the position imaged by another ceiling camera 310.
  • step S404 the position where the moving object Mo moves in the captured image captured by the ceiling camera 310 having the moving object area tracking unit 3305 by the ceiling camera is estimated. Further, the area of the moving object Mo is defined with respect to the position estimated to have been moved by the moving object area definition unit 3304, and the position information of the moving object Mo stored in the position information management DB 132 or the memory is updated.
  • step S405 the object entry / exit detection unit 3608 detects by the shelf camera 311 that an object such as the hand of the moving object Mo has entered and exited the shelf, or that the weight in the shelf has changed. This detection triggers the object recognition unit 3602.
  • the entry of an object into the shelf is detected by whether or not the image data in the approach detection area set in each shelf camera 311 has changed.
  • the object entry / exit detection unit 3608 detects that the object has left the shelf by continuously tracking the object such as the hand of the moving object Mo that has entered the shelf by imaging with the shelf camera 311.
  • step S406 triggered by the activation of the object entry / exit detection unit 3608, the object recognition unit 3602 compares the image of the object taken or the image before and after the image on which the object is placed when the shelf camera 311 is used. Define the image area for which the product is specified. Further, in step S406, when the weight sensor 312 is used, the change in the weight is confirmed. For example, when the weight is reduced, it is detected that the object is taken from the shelf, while when the weight is increased, it is detected. Detects that an object has been placed (returned) on a shelf. Further, in step S406, the object acquisition recognition unit by the ceiling camera may recognize that the moving object Mo has acquired the object from the shelf or the like.
  • step S407 the product identification unit 3603 identifies which product the object is.
  • the product identification unit 3603 lists the product candidates for the object defined by the object recognition unit 3602 by an image processing method such as specific object recognition, general object recognition, or deep learning.
  • the listed product candidates are called “product candidate list S”.
  • the product identification unit 3603 exerts a verification function and identifies the product with high accuracy.
  • the weight sensor 312 when the weight sensor 312 is used, the "weight list for each product” in which the weight for each product is recorded, or the "position for each product” in which the position of the shelf on which the product is installed is recorded. "Information list” is prepared in advance.
  • the product identification unit 3603 specifies a product based on the "weight list for each product" and the "position information list for each product”. Further, in step S407, the label recognition unit may recognize the label associated with the specified product. Further, in step S407, the discount sticker recognition unit may recognize the attached discount sticker according to the specified product. Further, in step S407, the product identification unit by the ceiling camera may specify which product is the object area acquired by the object acquisition recognition unit by the ceiling camera.
  • step S408 the product identification unit 3603 determines whether the specific object inspection mode or the all object inspection mode is set.
  • the process proceeds to step S409. Further, when the all-object inspection mode is set, the process proceeds to step S410.
  • step S410 the visual inspection result acquisition unit 301a requests the visual inspection (identification of the product) on the visual inspection terminal Q for the target object, and acquires the visual inspection result.
  • step S411 the product identification unit 3603 determines whether or not the added one object cannot be identified as which product. If the product identification unit 3603 cannot specify which product it is (YES in step S410), the process proceeds to step S409.
  • the product identification unit 3603 can be specified as which product in step S410 (NO in step S410)
  • the product identification unit 3603 is the product name, price, and trading restricted product stored in the storage unit of the server 1. Identify the product including certain information. As a result, the process proceeds to step S411.
  • the specified product information may be output to the output unit 507. If it is not possible to specify which product the product identification unit 3603 is in step S410, for example, a screen display of a display unit (not shown) provided in the sales floor device 3 or voice guidance by a speaker (not shown) may be used. , The purchaser may be requested to remove the product from the shelf. Further, the purchaser and the eye examiner or the like may have a call via a microphone (not shown) provided in the sales floor device 3.
  • the output unit 507 of the settlement gate 5 may display that it could not be identified as one of the products, or a candidate for a product with a high possibility of being applicable may be presented to the purchaser on the screen. You may ask to select the applicable product.
  • the purchaser may be requested to specify the product using a barcode reader or the like, or the purchaser may be requested to display a list of products and the purchaser to specify the product.
  • step S411 the trading restricted product determination unit 380 determines whether or not the product specified by the product identification unit 3603 is a product for which age confirmation is required.
  • step S411 If it is determined in step S411 that the product specified by the product identification unit 3603 is a product that requires age confirmation, that is, if it is determined to be YES, the process proceeds to step S412. If it is determined in step S411 that the product specified by the product identification unit 3603 is not a product that requires age confirmation, that is, if it is determined to be NO, the process proceeds to step S417. In step S412, the output control unit 537 causes the output unit 507 of the settlement gate 5 to display a screen for confirming the age. However, if the personal information of the moving object Mo is acquired and it is not necessary to confirm the age here, step S412 is skipped and the process proceeds to step S416.
  • the visual inspection result acquisition unit 301a requests the visual inspection (determination of the trade-restricted product) on the visual inspection terminal Q for the target product, and acquires the visual inspection result.
  • the trading restricted product determination unit 380 determines whether or not the instruction to release the trading restriction has been accepted. The trading restriction of the restricted product is lifted by either the clerk responding on the spot according to the result of the judgment or adopting the result of the inspection as a system.
  • the result of the visual inspection is adopted as the system will be described.
  • step S414 If it is determined in step S414 that the instruction to release the trading restriction is not accepted, the process proceeds to step S415. If it is determined in step S414 that the cancellation instruction for releasing the trading restriction has been accepted, that is, if it is determined that YES, the process proceeds to step S416.
  • step S415 the inspection result transmission unit 412 of the inspection terminal Q transmits a warning that the trading restriction has not been lifted even by the inspection result. Upon receiving this warning, for example, the output control unit 537 of the settlement gate 5 presents a warning to the effect that the trading restriction has not been released even by the inspection result via the output unit 507.
  • step S416 the process ends and the settlement is canceled.
  • step S416 the trading restriction product determination unit 380 releases the trading restriction. If step S416 is completed in this way, or if it is determined in step S411 that the product is not an age-restricted product (if it is determined to be NO), the process proceeds to step S417.
  • step S417 the moving object Mo and the product linking unit 354 associate the moving object Mo with the specified product.
  • step S4108 the product list management unit 3606 associated with the moving object continues to manage the product list associated with the moving object Mo until the settlement.
  • step S419 the settlement gate 5 setstles or setstles based on the product information associated with the moving object Mo.
  • the error display unit 151, the information terminal 9a, the information terminal 9b, and the output unit 507 of the settlement gate 5 may notify the clerk or the like of some error status. ..
  • step S419 when the trade-restricted product is tied to the moving object Mo, the gate may be left closed without being opened and an error state may be displayed.
  • the errors assumed in the tracking of the moving object Mo include the following cases, for example, when the ceiling camera 310 is used as the sensing device.
  • the system adopted by the present embodiment can take various measures including the following examples.
  • A An image including the moving object Mo captured by the ceiling camera 310 as a subject is transmitted to the visual inspection terminal Q, and the detection of the moving object Mo by the visual inspection is requested.
  • the eye examiner detects the moving object Mo, it defines a new moving object Mo, issues an ID, and starts tracking by the moving object area tracking unit 3305 by the ceiling camera. If the eye examiner cannot detect the moving object Mo, the clerk is notified to that effect.
  • B When a moving object Mo that is not associated with any ID registered in the position information management DB 132 is found, the captured image of the moving object Mo and a list of ID candidates to be associated are for visual inspection. It is sent to the terminal Q.
  • the eye examiner associates the most appropriate ID with the moving object Mo based on the past association information between the moving object Mo and the ID, and starts re-tracking by the moving object area tracking unit 3305 by the ceiling camera. If re-tracking cannot be started for some reason, the clerk will be notified accordingly.
  • the ID and a list of captured images of the moving object Mo that are not associated with any ID are transmitted to the visual inspection terminal Q.
  • the eye examiner attempts to associate the moving object Mo with the ID based on the past association information between the moving object Mo and the ID, and if the association is successful, the moving object area tracking unit using the ceiling camera. Start re-tracking with 3305.
  • the moving object Mo is requested to read the information for identifying the product with a barcode reader or the like.
  • the moving object Mo is requested to identify the product by operating the screen by himself / herself.
  • the sensing device that can be adopted is not limited to the camera as described above.
  • an example of adopting a weight sensor as a sensing device used for detecting the entry / exit of an object and identifying a product will be specifically described.
  • FIG. 14 is a diagram showing a layout example of a supermarket that employs the information processing system of FIG. 1, and is a diagram showing an example different from that of FIG.
  • An entrance gate 41a and a display device 41b are installed at the entrance 41 of the store 40.
  • the display device 41b displays the number of people existing in the store 40.
  • the store entrance gate 41a is provided with a lock function and an unlock function.
  • the moving object Mo as a new shopper can enter the store through the store entrance gate 41a.
  • the moving object Mo as a new shopper enters the store (by the lock function) until the number of people in the store 40 becomes less than the predetermined number. Is prohibited.
  • the moving object Mo When entry is permitted, the moving object Mo enters the store from the entrance 41 and proceeds through the aisle 44. When the moving object Mo picks up the product to be purchased, it proceeds to the settlement area 45, completes the settlement, and leaves the store.
  • the moving object Mo may put the product in the basket or my bag in hand, but in the present embodiment, the moving object Mo is premised on proceeding to the settlement area 45 with the product directly in the hand. This will be described below.
  • a plurality of ceiling cameras 310 are installed on the ceiling of the passage 44 between the entrance 41 and the exit 42.
  • weight sensors 312 are installed at each of the plurality of shelves racks 43 on which products can be placed. The weight sensor 312 detects the weight of the shelves on which the goods can be placed. As a result, the movement of an object from the shelf is detected from the change in weight detected by the weight sensor 312, and the goods that have entered and exited and the number of the products that have entered and exited are specified.
  • a plurality of weight sensors 312 may be installed on one shelf on which goods can be placed.
  • the weight sensor 312 installed in front of the shelf (for example, the place where the product with the old expiration date is placed), the product with the old expiration date goes in and out, while the product on the shelf
  • a change in weight is detected by a weight sensor 312 installed in the back (for example, a place where a product with a new expiration date is placed)
  • the weight sensor 312 may be installed under the shelf rack 43.
  • a change in the weight of the entire shelf is detected.
  • weight sensors 312 may be installed on each of the shelves on which the goods can be placed and on each of the shelf racks 43.
  • the shelf camera 311 may be installed at each of the plurality of shelves in the shelf rack 43.
  • the shelf camera 311 complements the entry / exit of an object from the shelf and the identification of the product by photographing the inside of the shelf, the product in the shelf, and other predetermined areas.
  • the moving object Mo is associated with an object that it does not originally possess (a state in which it is erroneously recognized). It becomes.
  • the accuracy of the settlement can be improved by complementing the entry / exit of the object and the identification of the product by the captured image acquired by the shelf camera 311.
  • the entry / exit of an object or the identification of a product is performed based only on the captured image acquired by the shelf camera 311, for example, the entry / exit of an object from the back of the shelf which cannot be seen by the camera can be detected, and overlapping objects or clutter can be detected. It is difficult to detect the entry and exit of placed objects and identify products.
  • the shelf camera 311 does not necessarily have to be installed on the shelf, and may be installed on the ceiling or the like, for example.
  • a settlement gate 5-4 equipped with a settlement machine 5a is installed in the settlement area 45.
  • the settlement machine 5a will be described on the premise that it is installed on the entrance 41 side of the settlement gate 5-4, but the present invention is not limited to this, and the installation location may be arbitrary.
  • the payment machine 5a is provided with a touch panel display, a camera, a payment terminal (including a cash change machine), a voice input device, a reader, a barcode reader, a speaker, a printer, an NW device, a motion capture system, and the like. There is.
  • the moving object Mo When the moving object Mo invades the settlement area 45 while possessing the product, the moving object Mo can complete the settlement in the following manner, for example, pass through the settlement gate 5-4, and leave the store.
  • the invasion triggers a product list of products associated with the moving object Mo to be displayed on the touch panel display of the settlement machine 5a.
  • the moving object Mo confirms the product list displayed on the touch panel display, and confirms that the product list and the possessed product match. If the contents of the displayed product list match the product you want to purchase, the payment process is started.
  • the moving object Mo selects a payment method by operating the touch panel display, completes the payment by the selected payment method, passes through the settlement gate 5-4, and exits the store. Further, for example, (2) when a moving object Mo possessing a product invades the settlement area 45 and holds a barcode, QR code (registered trademark), various IC cards, etc. over the reader of the settlement machine 5a, the moving object Mo The linked product list is read and displayed on the touch panel display. After confirming that the product list displayed on the touch panel display of the moving object Mo matches the product owned by the moving object Mo and performing a predetermined operation, the settlement is automatically completed, and the moving object Mo is settled at the settlement gate 5. Pass -4 and leave the store.
  • the invasion triggers a product list of products associated with the moving object Mo to be displayed on the touch panel display.
  • the moving object Mo confirms that the product list matches the product that he / she wants to purchase, the settlement is automatically completed, and the moving object Mo passes through the checkout gate 5-4 and leaves the store.
  • the moving object Mo simply possesses the product to be purchased and invades the settlement area 54, and the settlement is automatically performed and the settlement can be completed by a predetermined method.
  • the moving object Mo can complete the shopping without the intervention of a person such as a clerk Mt.
  • a button for selecting whether or not to perform the settlement of each of the plurality of moving objects Mo at once is displayed on the touch panel display of the settlement machine 5a. .. By pressing the displayed button, the moving object Mo can select whether or not to perform the settlement at once.
  • the product list is not displayed on the touch panel display of the settlement machine 5a because the product is not tied to the moving object Mo.
  • the moving object Mo can pass through the settlement gate 5-4 and leave the store without making a settlement.
  • the settlement gate 5-4 may notify by voice, light or the like that the moving object Mo passing through the gate passes through without purchasing the product.
  • a checkout gate 5-5 is installed on the entrance 41 side of the checkout register 6. Products that are not subject to automatic settlement by the moving object Mo are subject to settlement processing by a clerk at the settlement register 6.
  • the information processing system including the server according to the first embodiment of the information processing system of the present invention shown in FIG. It has basically the same configuration as that of. Therefore, the description of the configuration diagram of the information processing system including the server according to the second embodiment of the information processing system of the present invention will be omitted.
  • the server 1 and the settlement gate 5 shown in FIGS. 5 and 7 are described. It has basically the same configuration as the hardware configuration of.
  • FIG. 15 is a block diagram showing a hardware configuration of a sales floor device among the configurations according to a second embodiment of the information processing system of the present invention.
  • the sales floor device 3 includes a CPU 301, a ROM 302, a RAM 303, a bus 304, an input / output interface 305, a ceiling camera 310, a shelf camera 311 and a weight sensor 312, an output unit 313, a communication unit 315, and an individual. It includes an information acquisition unit 321 and an information terminal 9. Further, the information terminal 9 includes a remote control unit 390, a display unit 391, and an input unit 392.
  • the ROM 302, RAM 303, bus 304, input / output interface 305, ceiling camera 310, shelf camera 311, weight sensor 312, and communication unit 315 of the sales floor device 3 are basically the same as the hardware configuration of the sales floor device 3 shown in FIG. It has a similar configuration.
  • the output unit 313 is composed of, for example, a display for displaying digital signage, a speaker, or the like, and displays, for example, the number of people existing in the store 40.
  • the personal information acquisition unit 321 is composed of various devices such as a reader, and acquires personal information of the moving object Mo.
  • the information terminal 9 is composed of information devices such as personal computers, smartphones and tablets.
  • the input unit 392 is composed of, for example, a keyboard, buttons, etc., and inputs such as error cancellation and trading restriction cancellation.
  • FIG. 16 shows an example of the functional configurations of the server of FIG. 5, the sales floor device of FIG. 6, the settlement gate of FIG. 7, and the inspection terminal among the configurations according to the second embodiment of the information processing system of the present invention. It is a functional block diagram. Further, FIG. 17 is a functional block diagram showing a detailed functional configuration example of the shelf product recognition unit provided in the sales floor device of FIG. 16 among the configurations according to the second embodiment of the information processing system of the present invention. ..
  • the server 1 includes a CPU 101, a storage unit 108, an error display unit 151, an error release unit 152, and a communication unit 109. Since each of the functional blocks constituting the server 1 has basically the same function and configuration as each of the corresponding functional blocks of the server 1 of FIG. 8, description thereof will be omitted here.
  • the visual inspection terminal Q includes a CPU 401, an output unit 406, an input unit 407, and a communication unit 409.
  • the output unit 406, the input unit 407, and the communication unit 409 of the visual inspection terminal Q each have basically the same functions and configurations as the corresponding functional blocks of the visual inspection terminal Q of FIG. Therefore, in the following, only the functions and configurations different from those of the visual inspection terminal Q of FIG. 8 will be described.
  • the information presentation control unit 421 executes control to output the information regarding the object, the product, and the moving object Mo transmitted from the sales floor device 3 or the settlement gate 5 to the output unit 406. Specifically, for example, (1) when a request for inspection is made by the moving object Mo, the information presentation control unit 421 receives information on the object and the product transmitted from the sales floor device 3 and the settlement gate 5 by the operator. It is presented by executing the control to output to the screen of the monitor etc. to be operated. Further, for example, (2) when a trading restricted product is linked to the moving object Mo and an input is made to request the cancellation of the trading restriction by the moving object Mo, the information presentation control unit 421 may perform the information presentation control unit 421.
  • the sales floor device 3 and the settlement gate 5 are presented by executing control to output information such as images and sounds related to the moving object Mo transmitted to a monitor, a speaker, or the like. Further, for example, (3) when an inquiry about the operation method of the settlement gate 5-4 or the settlement machine 5a or an inquiry about the payment processing in general is made from the moving object Mo, the information presentation control unit 421 is transmitted from the settlement gate 5. It is presented by executing control to output information such as images and sounds related to the moving object Mo to a screen of a monitor or the like or a speaker or the like.
  • the result transmission control unit 422 executes control to transmit the result input by the operator to the sales floor device 3 and the settlement gate 5 for the information about the object, the product, and the moving object Mo. Specifically, for example, (1) when a moving object Mo requests an inspection, the result transmission control unit 422 displays the inspection result (specific result of the product, etc.) by the operator on the sales floor device 3 or the settlement gate 5. Performs control to send to. Further, for example, (2) when a trading restricted product is linked to the moving object Mo, the result transmission control unit 422 controls to transmit the determination result of the trading restriction release by the operator to the sales floor device 3 or the settlement gate 5. Execute.
  • the result transmission control unit 422 transmits support information such as an operation method of the settlement gate 5-4 and the settlement machine 5a and an explanation about payment processing via the display and the speaker of the settlement machine 5a. Take control.
  • the result transmission control unit 422 may present support information by executing an operation such as remote desktop.
  • the information about the object, the product, and the moving object Mo, and the information input by the operator have been described on the premise that the information is input / output via the visual inspection terminal Q, but the present invention is not limited to this configuration. That is, for example, the above-mentioned information may be input / output to the display unit 391 and the input unit 392 of the information terminal 9 of the sales floor device 3.
  • the settlement gate 5 includes a CPU 501, an input unit 506, an output unit 507, and a communication unit 509.
  • Each functional block of the checkout gate 5 has essentially the same function and configuration as each of the corresponding functional blocks of the checkout gate 5 of FIG.
  • the sales floor device 3 functions as a CPU 301, a ceiling camera 310, a shelf camera 311, a weight sensor 312, a communication unit 315, a personal information acquisition unit 321 and an information terminal 9.
  • the CPU 301 the personal authentication unit 320, the moving object tracking unit 330, the location information management unit 340, the store entry control unit 350, the shelf product recognition unit 360, the trading restricted product determination unit 380, and the operation reception unit 301c And the operation result acquisition unit 301d function.
  • the shelf product recognition unit 360 of the CPU 301 includes a detection unit 3602a shown in FIG.
  • a product identification unit 3603a a moving object and a product linking unit 3604, a product list management unit 3606 linked to a moving object, and a product unspecified. It includes a determination unit 3609, a label recognition unit 3610, a discount sticker recognition unit 3611, and an inventory information update unit 3612.
  • the ceiling camera 310, the shelf camera 311, the weight sensor 312, and the communication unit 315 have basically the same functions and configurations as the corresponding functional blocks of the sales floor device 3 of FIG.
  • the shelf product recognition units 360 of the CPU 301 the moving object and the product linking unit 3604, the product list management unit 3606 linked to the moving object, the product unspecified determination unit 3609, the label recognition unit 3610, and the discount sticker recognition
  • the unit 3611 has basically the same function and configuration as each of the corresponding functional blocks in the shelf product recognition unit 360 of FIG.
  • the trading restricted product determination unit 380 has basically the same function and configuration as the corresponding functional block in the CPU 301 of the sales floor device 3 of FIG. Therefore, in the following, only the functions and configurations different from those of the sales floor device 3 of FIG. 8 will be described.
  • the personal information acquisition unit 321 acquires personal information using the biological information of the moving object Mo as an example.
  • the personal information acquisition unit 321 is one of the applications displayed on an IC card or a mobile terminal containing the personal information of the moving object Mo with respect to the reader installed in a part of the entrance gate 41a.
  • the personal information of the moving object Mo is acquired by performing predetermined operations such as holding the screen over, displaying the face of the moving object Mo itself, and holding the eyes and hands of the moving object Mo.
  • the personal authentication unit 320 identifies the name, gender, date of birth, halal, financial information, etc. of the moving object Mo from the acquired personal information.
  • the moving object Mo can be requested to confirm the age.
  • the moving object Mo can automatically complete the settlement only by holding the product to be purchased and proceeding to the settlement area 45.
  • the recommended information for the moving object Mo and the information about the product held by the moving object Mo can be obtained in real time. It will be possible to present it on Mo's smartphone. It is also possible to realize paperless office by having a smartphone present a receipt after purchasing the product. Furthermore, since the purchase history of the moving object Mo and personal information are managed in association with each other, this information can be effectively used as marketing information.
  • the store entry control unit 350 executes a process of controlling the number of people existing in the store 40 to be less than a predetermined threshold value. For example, when it is necessary to maintain the number of people in the store 40 to less than 20, the store entry control unit 350 allows new shoppers to enter the store until the number of people in the store 40 reaches 20. It is permitted, and when the number of people exceeds 20, new shoppers are prohibited from entering the store. When permitting the entry of a new shopper, the entry control unit 350 indicates to the display device 41b or the like that the entry is permitted, and opens (unlocks) the opening / closing member of the entry gate 41a. To execute.
  • the entry control unit 350 when prohibiting the entry of new shoppers, the entry control unit 350 indicates to the display device 41b or the like that the entry is prohibited, and closes (locks) the opening / closing member of the entry gate 41a. Execute the process. At this time, the store entry control unit 350 may execute a control for notifying whether or not the store can be entered by sound or light from a speaker or the like provided in the display device 41b or the like. As a result, the number of people existing in the store 40 can be maintained below a certain level, so that it is possible to prevent infectious diseases and the like due to contact between people.
  • the detection unit 3602a and the product identification unit 3603a function in the shelf product recognition unit 360.
  • the detection unit 3602a confirms the change in weight detected by the weight sensor 312 and the position information in the shelf on which the weight sensor 312 is installed. Specifically, for example, the detection unit 3602a recognizes that when the weight of the shelf A is changed so as to decrease, an object having the reduced weight is taken away from the shelf A, while the weight of the shelf B increases. In the case of such a change, it is recognized that an object having an increased weight on the shelf B has been placed (returned) in the shelf. Information about the object thus recognized being removed from the shelf or placed on the shelf is handed over to the product identification unit 3603.
  • the product identification unit 3603a specifies which product the object recognized by the detection unit 3602a is and the number of the objects based on the weight data and the position information data.
  • the inventory information update unit 3612 identifies the placed (taken away) product and the number of the product when the product is placed on the shelf or taken away from the shelf by the clerk, and the product The process of storing the product DB information as the inventory information of the product is executed.
  • the operation reception unit 301c When the operation reception unit 301c receives an operation request from the moving object Mo, the operation reception unit 301c transmits information about the moving object Mo, the object, the product, and the like to the visual inspection terminal Q. Specifically, for example, (1) when the product is determined to be unspecified by the product unspecified determination unit 3609, the operation reception unit 301c receives a request for inspection from the moving object Mo, and the weight sensor 312 determines that the product is unspecified. Control is performed to transmit the acquired weight data and position information of the object to the visual inspection terminal Q. At this time, the captured image obtained by the ceiling camera 310 or the shelf camera 311 may be transmitted to the eye inspection terminal Q.
  • the operation reception unit 301c receives a request from the moving object Mo to release the trade restriction. The control of transmitting the related information of the product to the inspection terminal Q is executed.
  • the operation result acquisition unit 301d acquires the operation result from the visual inspection terminal Q. That is, for example, information such as (1) information on the product specified by the operator's visual inspection and (2) the determination result of the cancellation of the trading restriction by the operator is acquired.
  • step S701 the store entry control unit 350 determines whether or not the number of people existing in the store 40 is equal to or less than a predetermined threshold value.
  • step S701 If it is determined in step S701 that the number of people present in the store 40 is equal to or greater than a predetermined threshold value, the process repeats step S701.
  • step S701 determines whether the number of moving objects Mo existing in the store 40 is less than a predetermined threshold value. If it is determined in step S701 that the number of moving objects Mo existing in the store 40 is less than a predetermined threshold value, the process proceeds to step S702. In step S702, when the moving object Mo enters the store from the entrance 41, the ceiling camera 310 starts imaging the moving object Mo.
  • step S703 the moving object finding unit 3302 by the ceiling camera discovers the moving object Mo.
  • the moving object area definition unit 3304 by the ceiling camera defines a predetermined area including the moving object Mo discovered by the moving object finding unit 3302 as the moving object area.
  • step S704 the moving object area tracking unit 3305 by the ceiling camera estimates the position where the moving object Mo moves. Further, when the moving object area definition unit 3304 moves, the area of the moving object Mo after the movement is defined again, and the position information of the moving object Mo stored in the position information management DB 132 or the memory is updated.
  • the moving object finding unit 3302 by the ceiling camera acquires the individual ID of the moving object Mo.
  • the ID of the moving object Mo is managed by the position information management DB 132, the memory, etc. in association with the position (coordinate) information of the moving object Mo, and is updated every time a predetermined area including the moving object Mo is defined. This defined coordinate is also recognized at the position imaged by another ceiling camera 310.
  • the ID is issued after the area of the moving object Mo is defined, and the moving object Mo and the ID are associated with each other.
  • the order of numbering and associating is not particularly limited to this, and may be arbitrary.
  • step S706 the detection unit 3602a recognizes that the object has been removed (returned) from the shelf based on the change in the weight data acquired by the weight sensor 312.
  • the product identification unit 3603a identifies an object that has been taken away (returned) from the shelf. That is, the product identification unit 3603a identifies an object from the position information of the weight sensors 312 installed on each of the predetermined shelves on which the object is placed, and specifies the number of objects from the amount of change in the weight data. That is, the number of products is specified by matching the weight data acquired by the weight sensor 312 with the weight data in the product information stored in advance, and the product name is specified from the display position data in the product information.
  • the product information includes at least the product name, price, JAN code, product image, product weight, product size, display position, discount information, refund, sales total, and display, and these product information is stored in the server 1 It is stored in the product DB 131 of.
  • step S708 the moving object and the product linking unit 3604 associate the moving object Mo with the specified product. Further, the product unspecified determination unit 3609 associates the unspecified product with the moving object Mo.
  • step S709 the product list management unit 3606 associated with the moving object manages the product list associated with the moving object Mo up to the settlement.
  • step S710 the output control unit 537 of the settlement gate 5 executes control to display the product list associated with the moving object Mo, triggered by the fact that the moving object Mo has entered the settlement area 45. Specifically, for example, referring to the product list displayed on the touch panel display of the checkout machine 5a, the moving object Mo confirms whether or not all the products to be purchased by itself have been specified. If it is determined that the product list needs to be modified, the moving object Mo can modify the product list.
  • step S711 the operation reception unit 301c determines whether or not the correction request by the operation of the moving object Mo itself has been accepted.
  • step S711 If it is determined in step S711 that the request for modification of the product list by the moving object Mo itself has been accepted, the process proceeds to step S712.
  • the moving object Mo modifies the product list by its own operation. Specifically, for example, when there is an excess or deficiency in the number of products purchased by the moving object Mo and the products displayed in the product list, the moving object Mo is a button (not shown) on the touch panel display of the checkout machine 5a. You may modify the number by pressing. Further, for example, when a product that cannot be specified or an incorrect product is displayed in the product list, the moving object Mo may try to correct the product list by reading a barcode attached to the product or the like.
  • the moving object Mo may try to identify the product by selecting the correct product from the product candidates displayed on the touch panel display, or may search from the product information of all the products stored in the product DB 131. It may be specified.
  • the product list management unit 3606 associated with the moving object updates the product list, the process is returned to step S711, and the subsequent processes are repeated.
  • step S711 if it is determined in step S711 that the moving object Mo itself has not requested the correction of the product list, the process proceeds to step S713.
  • step S713 the operation reception unit 301c determines whether or not the request for inspection is received from the moving object Mo.
  • step S713 the operation result acquisition unit 301d of the sales floor device 3 acquires the result of the visual inspection by the operator.
  • the operator (not shown) performs the visual inspection.
  • the method of visual inspection is not particularly limited, and the operator may perform visual inspection based on information related to an object, a product, etc. (for example, weight data or image information acquired by a camera), or prepare for the checkout machine 5a. The image of the object may be confirmed through the camera or the like.
  • the result transmission control unit 422 of the inspection terminal Q executes control to transmit the inspection result to the sales floor device 3 and the settlement gate 5.
  • the operation result acquisition unit 301d acquires the result of the inspection by the operator
  • the product list management unit 3606 associated with the moving object updates the product list
  • the process is returned to step S713, and the subsequent processes are repeated. ..
  • step S713 determines whether the request for inspection is not accepted. If it is determined in step S713 that the request for inspection is not accepted, the process proceeds to step S715.
  • step 715 the trading restricted product determination unit 380 determines whether or not the product specified by the product specifying unit 3603 is a product for which trading restrictions are required.
  • the trade-restricted product determination unit 380 determines, for example, that the specified product is an age-restricted product, the process proceeds to step S716.
  • the operation reception unit 301c determines whether or not the request for releasing the trading restriction has been accepted by the operation of the moving object Mo itself. That is, for example, when the OK button is pressed by the moving object Mo in response to the display such as "Are you 20 years old or older?" Displayed on the touch panel display of the checkout machine 5a, the operation reception unit 301c is displayed from the moving object Mo itself. Accepts requests to lift trading restrictions.
  • the operation reception unit 301c executes a control of transmitting information related to the moving object Mo, the product, etc. to the visual inspection terminal Q.
  • the operation result acquisition unit 301d acquires the result of age confirmation by the operator. Specifically, for example, the operator confirms the face, voice, and product information of the moving object Mo, and determines whether or not the moving object Mo is 20 years old or older. At this time, the operator may notify the moving object Mo via the touch panel display, the speaker, or the like to present a license or the like capable of confirming the age to the camera.
  • step S718 the product list management unit 3606 associated with the moving object updates the product list, and the trading restriction product determination unit 380 releases the trading restriction.
  • step S719 the trading restricted product determination unit 380 sends a warning and cancels the automatic settlement process.
  • step S720 the moving object Mo completes the settlement.
  • the gate opens and the moving object Mo leaves the store. If the moving object Mo has been personally authenticated, the age confirmation described above does not necessarily have to be performed, and the moving object Mo can leave the store immediately after completing the settlement.
  • the moving object area tracking unit 3305 using the ceiling camera updates the position information of the person in the store 40, and the store entry control unit 350 allows new shoppers to enter the store.
  • the information processing system may include a shelf inventory management function.
  • the shelf inventory management function includes a clerk determination means and an inventory information update means.
  • the clerk discrimination means discriminates between a clerk and a shopper.
  • the clerk discrimination unit assigns a physical marker that can identify the clerk to something worn by the clerk, such as a hat or clothes.
  • the clerk is identified by the ceiling camera and the shelf camera taking an image of this physical marker.
  • the clerk determination means may include a button provided in a predetermined area such as the backyard of the store.
  • a clerk determination means when a clerk existing in a predetermined area presses a button, the clerk is assigned an ID.
  • an ID may be assigned to the clerk when the clerk passes through a predetermined area such as the backyard of the store or invades the predetermined area.
  • the ceiling camera keeps tracking the area of the person recognized as the clerk.
  • the inventory information is updated by making full use of the object recognition unit and the product identification unit.
  • Update in the product DB of the server the inventory information update means updates the inventory of the shelf on which the product exists and the inventory information of the entire store.
  • the inventory information updating means manages the inventory of the shelves and the inventory information of the entire store, and notifies the remote control unit or automatically places an order when the number of inventories falls below the threshold value.
  • the present invention may include a shopper attribute estimation function.
  • the shopper's attribute estimation function estimates the shopper's rough age, age, gender, and other attributes from, for example, face recognition.
  • the shopper's attribute estimation function is provided in the sales floor device, the checkout gate, and the like.
  • the present invention links purchasing information, shopper attribute information, and personal information to a POS system.
  • the purchase information includes, for example, information such as the name of the purchased product, the amount of money, the position of the product, the movement history of the shopper, or data on the product that has not been purchased but has been moved in and out of the shelf.
  • the shopper attribute information includes, for example, information obtained by the above-mentioned shopper attribute estimation function.
  • the shopper's attribute estimation function may be linked with, for example, a trading restricted product determination unit. As a result of estimating the attributes, if the shopper is in his thirties or older who clearly does not need age confirmation, this shopper's attribute estimation function may control the trading restricted product judgment unit not to confirm the age. Good.
  • Image recognition using deep learning is given as an example as a method for identifying a product in the information processing system in the above-described embodiment.
  • deep learning it is necessary to relearn when the product cannot be identified or when it is misrecognized, and in that case, it is necessary to manually give the image of the correct product, which is a lot of trouble. It takes.
  • the information processing system of the present invention has a function of visually confirming by an eye examiner, scanning a bar code, etc. at the time of purchasing a product, and correcting the identification of the product, and the specific correction of the product.
  • Deep learning based on the function to link and record the correct product image and necessary information (corrected product number, specific code, etc.) and the correct product image and necessary information when By implementing the function of automatically executing the learning of the above and the function of automatically deploying the learning result in the same information processing system, it is possible to realize the automation of the above-mentioned series of deep learning re-learning processes.
  • the deployment of the learning result refers to a process of updating, for example, a deep learning model of the information processing system based on the learning result so that it can be used for identifying a product.
  • the above re-learning process may be started by a predetermined condition such as a failure of product identification as a trigger, or may be started by an explicit operation by a person in charge, such as pressing a button. Good.
  • an image recognition method applied when identifying an object as a product for example, object recognition using AI (Artificial Intelligence), object recognition using a feature point or feature amount, and a product logo. It is possible to adopt shape recognition and character recognition (character shape recognition using AI, character recognition using OCR (Optical Character Recognition), etc.).
  • AI Artificial Intelligence
  • OCR Optical Character Recognition
  • weight recognition weight recognition of an object using a weight sensor
  • product recognition infrared sensor or image
  • scanning identification information such as a barcode or QR code (registered trademark). Scanning of barcodes and QR codes (registered trademarks) by reading, etc.)
  • product recognition by reading electronically recorded identification information such as RFID, or a combination of multiple recognition methods to make an object a product Can be identified.
  • the visual inspection of the object or the product is performed by the visual inspection terminal Q, but the present invention is not limited to this. That is, the visual inspection by the eye examiner may be performed by another device such as the settlement gate 5. Further, in each of the above-described embodiments, one store or one checkout gate 5 is provided with one eye inspection terminal Q, or a plurality of stores or a plurality of checkout gates are provided with one eye inspection terminal. It is possible to have a Q.
  • the hybrid check system (visual inspection) in the above-described embodiment is not necessarily used only for product identification. That is, for example, since the product contains (A) products such as cigarettes and alcoholic beverages that cannot be purchased until a certain age, (B) products whose consumption period has expired or expiration date, and (C) allergic components, depending on the constitution. It can also be used to determine whether or not a product is restricted by religion, such as a product that should not be ingested, (D) a product other than halal food, or the like.
  • the flapper type gate is adopted in the entrance gate 41a, the settlement gate 5-4, and the settlement gate 5-5, but the present invention is not particularly limited to this method. That is, for example, a gate displayed by a hologram may be adopted.
  • Each of the entrance gate 41a, the checkout gate 5-4, and the checkout gate 5-5 may notify the passability of a person by voice or light, for example.
  • Product information is stored in the product DB 131 and updated in real time.
  • product information can be managed for each store, for each of a plurality of stores, for each cash register, or for each day, month, and year.
  • the storage unit in each information processing device has been described as being composed of a hard disk, DRAM, or the like, but the present invention is not limited to this.
  • an external storage device such as SSD may be used.
  • the moving object Mo may try to correct the product list by reading a barcode or the like.
  • the object Mo may identify all the goods it possesses by reading the barcode.
  • the state space model based on the captured image is adopted for the discovery of the moving object Mo, but the method is not limited to this method. That is, for example, various data acquired from each sensing device such as three-dimensional point cloud data, 2D image, distance data, temperature data, infrared ray amount, and optical data may be adopted for finding the moving object Mo.
  • the region of the moving object Mo is defined by continuously finding the region of change centered on the discovered moving object Mo, but the method is not particularly limited to this method. That is, for example, the moving object Mo may be discovered from the three-dimensional point cloud data, the position coordinates of the moving object Mo may be specified by using the ICP algorithm, and the moving object region may be defined.
  • the moving object Mo is a human
  • the area of the moving object Mo is measured by using data that can be acquired from each sensing such as 3D point cloud data, 2D image, distance data, temperature data, infrared amount, and optical data. May be defined. Further, the region of the moving object Mo may be estimated by using deep learning together.
  • the tracking of the moving object Mo has been described on the premise that a tracking technique by collecting feature data or an object tracking technique such as a Bayesian filter is adopted, but the tracking is not particularly limited to these methods. That is, for example, the moving object Mo may be tracked using the optimum solution of the position information. Further, the moving object Mo may be estimated and tracked after determining noise based on the time-series position information.
  • the position information of the camera is defined as to where each camera is projecting in the store by synthesizing the captured images captured by each ceiling camera 310, but the method is not limited to this method. That is, for example, the position information of each camera may be defined by converting the position information of each camera into the position information of the entire store (integrating a plurality of cameras) by the coordinate conversion using the ICP algorithm.
  • the moving objects Mo are grouped using information such as the position information and the moving direction of a plurality of moving objects Mo
  • the information used for the grouping is not limited to these. That is, for example, grouping may be performed based on biometric information or the like acquired by the personal information acquisition unit 321.
  • the blood relationship of a plurality of moving objects Mo is determined by the face recognition method, but the information used for determining the blood relationship is not limited to these. That is, for example, the blood relationship may be determined based on various information acquired by the personal information acquisition unit 321.
  • the inside of the basket or cart before and after the object is delivered is recognized by performing difference recognition or the like.
  • the product may be specified.
  • a method of installing a mirror in which the shooting target is reflected on the ceiling and imaging the shooting target reflected in the mirror by the ceiling camera 310 or the shelf camera 311 has been described.
  • the position of the camera is not limited to the wall.
  • a camera may be installed on a shelf or ceiling.
  • the position information of the moving object Mo is displayed on the information terminal 9 possessed by the clerk Mt, the screen of the backyard, or the like, but the displayed terminals are not limited to these.
  • the position information of the moving object Mo may be displayed on a terminal or the like possessed by a service provider, a system person in charge, or the like (not shown).
  • product candidates are listed by image processing methods such as specific object recognition, general object recognition, and deep learning, and the listed product candidates are further separated.
  • image processing methods such as specific object recognition, general object recognition, and deep learning
  • the product candidates listed by the image processing method such as specific object recognition may be treated as the specified product.
  • the entry / exit of the object is recognized at the timing when the object is taken out from the shelf and the identification as a product is attempted, but the recognition and the specific timing are not limited to this and are arbitrary. It's fine. That is, for example, when an object is taken away from the shelf, the entry / exit of the object is recognized, the recognized object and the moving object Mo are linked, and the product is specified by using the moving object Mo invading the settlement area 45 as a trigger. May be made. As a result, after the entry / exit of the object is recognized, there is a time grace before it is identified as a product, so that the accuracy of identifying the product is improved, for example, the product can be identified by multiple methods. Can be made to.
  • the necessary information input by visual confirmation by an eye examiner or operator, reading of a barcode, or the like is associated with the correct product image.
  • the deep learning learning process can be automated by recording the information, but the label that can be used as the correct answer is not limited to the product image. That is, for example, information that identifies a product other than an image (for example, a recognition result by a weight sensor) may be associated with necessary information as a correct answer, and a learning process may be executed. As a result, the accuracy of the deep learning re-learning process can be improved.
  • the above-mentioned image processing by the CPU may be processed by the GPU (Graphic Processing Unit).
  • the illustrated hardware configuration and block diagram are merely examples for achieving the object of the present invention, and the present invention is not limited to the illustrated examples.
  • the functional configurations of FIGS. 8 to 11, 16 and 17 are merely examples, and are not particularly limited. That is, it suffices if the system is provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is particularly determined in FIGS. 8 to 11, 16 and 16.
  • the example is not limited to 17, and may be arbitrary.
  • the location of the functional block is not limited to the one shown in the drawing, and may be arbitrary.
  • one functional block may be configured by a single piece of hardware, a single piece of software, or a combination of them.
  • the processing of each functional block is executed by software, the programs constituting the software are installed on a computer or the like from a network or a recording medium.
  • the computer may be a computer embedded in dedicated hardware, or may be a smartphone or a personal computer.
  • the computer may be, for example, a server on the cloud.
  • the recording medium containing such a program is not only composed of a removable medium (not shown) distributed separately from the main body of the device in order to provide the program to the user or the like, but also is preliminarily incorporated in the main body of the device. It is composed of a recording medium or the like provided to the above.
  • the steps for describing a program recorded on a recording medium are not only processed in chronological order but also in parallel or individually even if they are not necessarily processed in chronological order. It also includes the processing that is executed in. Further, in the present specification, the term of the system means an overall device composed of a plurality of devices, a plurality of means, and the like.
  • the information processing system to which the present invention is applied need only have the following configuration, and various various embodiments can be taken. That is, the information processing system In an information processing system used when people move and purchase products
  • a moving object tracking means for tracking the moving object for example, a moving object area tracking unit 3305 using a ceiling camera, using the person or an object moving with the person as a moving object.
  • a product identification means for example, product identification unit 3603 that specifies both a product and a number of products by a predetermined method using at least one of a weight sensor and a camera.
  • a linking means for linking the specified product and the moving object having a predetermined positional relationship for example, a moving object and a product linking unit 3604.
  • the information processing system Further provided is a presentation means (for example, output control unit 537) for presenting a settlement amount determined from the identified product information. This allows the purchaser to attempt to settle the goods unattended based on the offered settlement amount.
  • a presentation means for example, output control unit 5357 for presenting a settlement amount determined from the identified product information. This allows the purchaser to attempt to settle the goods unattended based on the offered settlement amount.
  • a settlement means for example, a settlement unit 535) that performs a settlement process for the product associated with the moving object. Further prepare. As a result, the purchaser can settle the goods he / she purchases just by invading the checkout area.
  • An acquisition means for example, personal information acquisition unit 321 for acquiring information about the moving object is further provided.
  • the purchaser can complete the settlement of the product simply by passing through the settlement gate without taking out cash, a credit card, or the like.
  • the moving object tracking means recognizes when the product associated with the moving object to be tracked is delivered to a person different from the moving object (for example, receiving by a ceiling camera).
  • the passed object recognition unit 3312 identifies the person different from the moving object as a new moving object, targets the new moving object as a tracking target, and associates the moving object with a product (for example, linking the moving object to a product).
  • Part 3604 associates the delivered product with the new moving object.
  • a first error presenting means for example, an error display unit 151 for presenting that the product is in a predetermined error state is further provided. ..
  • the clerk Mt and the operator can confirm the age of the person who wants to purchase the restricted product and notify that the product is an allergic product.
  • the linking means further links the product, which has failed in the attempt to specify both the product and the number of products by the product specifying means, to the moving object (for example, the product unspecified determination unit 3609).
  • a second error presenting means for example, an error display unit 151 that indicates that a predetermined error state occurs when the attempt to specify both the product and the number of goods by the product specifying means fails.
  • the moving object tracking means tracks the moving object by using the person moving in the store where the clerk is present as the moving object.
  • a clerk discriminating means for example, a clerk discriminating means for discriminating between the clerk and the moving object.
  • one or more of the above-mentioned products are arranged in a predetermined storage place including the shelves of the store, and the inventory of the above-mentioned products in the above-mentioned storage space and the entire store is managed, and the predetermined products are purchased.
  • an inventory information updating means for example, inventory information updating unit 3612 for updating the inventory of the entire storage place and the store where the predetermined product exists. To be equipped. This makes it possible to automate management tasks such as in-store inventory work.
  • a replenishment notification means for example, inventory information update unit 3612 for notifying the fact.
  • Inventory replenishment means for example, inventory information update unit 3612 that controls execution of automatic ordering when the inventory quantity of the product in the entire store meets a predetermined requirement. Further prepare. As a result, the inventory of products is automatically ordered, so that management work can be performed reliably in a short time at low cost.
  • the information processing system It further includes an attribute estimation means (for example, an attribute estimation function) for estimating the attributes of the person.
  • an attribute estimation means for example, an attribute estimation function for estimating the attributes of the person.
  • the moving object tracking means further registers historical information indicating the movement route of the moving object during tracking (for example, moving object area definition unit 3304, position information management DB 132).
  • historical information indicating the movement route of the moving object during tracking for example, moving object area definition unit 3304, position information management DB 1302.
  • the product specifying means identifies both the product and the number of products by using the predetermined method in which the eye examiner visually confirms the image including the product as the subject as the first method.
  • the first specific means for acquiring the result of the attempt for example, the operation result acquisition unit 301d
  • a second specifying means for example, a product specifying unit 3603a that attempts to specify both the product and the number of products by using a second method different from the first method among the predetermined methods is provided.
  • the product is specified based on the result of at least one of the first specific means and the second specific means. This makes it possible to improve the accuracy of identifying the product.
  • a store entry control means that controls permission or prohibition of entry of a new person depending on the number of people in the store is further provided.
  • the density of people in the store can be controlled to be less than a predetermined value, so that it is possible to prevent infectious diseases caused by droplets and aerosols.
  • the store entry control notification means for example, the output unit 313 for notifying the fact is provided.
  • the moving object Mo can stand by while entering the store is prohibited, so that the density of people in the store can be maintained below a predetermined level.
  • the presenting means further Among the plurality of the moving objects, the settlement amount determined from the information of the product associated with each of the one or more moving objects satisfying a predetermined condition is presented (for example, the output control unit 537). As a result, the product information associated with the plurality of moving objects Mo can be collectively settled.
  • Moving object tracking unit 3302 ... ⁇ Moving object detection unit by ceiling camera 3304 ⁇ ⁇ ⁇ Moving object area tracking unit by ceiling camera 3305 ⁇ ⁇ ⁇ Moving object area definition unit by ceiling camera 3308 ⁇ ⁇ ⁇ Grouping unit 3309 ⁇ ⁇ ⁇ Blood relationship judgment unit 340 ⁇ ⁇ ⁇ Position information management unit 341 ⁇ ⁇ ⁇ Inter-camera information transfer unit 342 ⁇ ⁇ ⁇ Position definition unit for each camera 360 ⁇ ⁇ ⁇ Shelf product recognition unit 3602 ⁇ ⁇ Object recognition unit 3603 ⁇ ⁇ ⁇ Product identification unit 3609 ⁇ ⁇ ⁇ ⁇ Product Unspecified judgment unit 3610 ⁇ ⁇ ⁇ Label recognition unit 380 ⁇ ⁇ ⁇ Trading restricted product judgment unit 390 ⁇ ⁇ ⁇ Remote operation unit Mo ⁇ ⁇ ⁇ Moving object

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Accounting & Taxation (AREA)
  • Theoretical Computer Science (AREA)
  • Finance (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Multimedia (AREA)
  • Geometry (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Operations Research (AREA)
  • Human Resources & Organizations (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Security & Cryptography (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Cash Registers Or Receiving Machines (AREA)
PCT/JP2020/047765 2019-12-19 2020-12-21 情報処理システム Ceased WO2021125357A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2021565702A JP7225434B2 (ja) 2019-12-19 2020-12-21 情報処理システム
EP20900801.0A EP4075399A4 (en) 2019-12-19 2020-12-21 INFORMATION PROCESSING SYSTEM
US17/786,817 US20230027382A1 (en) 2019-12-19 2020-12-21 Information processing system
JP2022190789A JP2023014207A (ja) 2019-12-19 2022-11-29 情報処理システム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019229737 2019-12-19
JP2019-229737 2019-12-19

Publications (1)

Publication Number Publication Date
WO2021125357A1 true WO2021125357A1 (ja) 2021-06-24

Family

ID=76477441

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/047765 Ceased WO2021125357A1 (ja) 2019-12-19 2020-12-21 情報処理システム

Country Status (4)

Country Link
US (1) US20230027382A1 (https=)
EP (1) EP4075399A4 (https=)
JP (2) JP7225434B2 (https=)
WO (1) WO2021125357A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114078299A (zh) * 2021-11-15 2022-02-22 烟台创迹软件有限公司 一种商品结算方法、装置、电子设备及介质
JP2023114434A (ja) * 2022-02-04 2023-08-17 エヌ・シー・アール・コーポレイション 精算のための複数商品の製品認識
JP2024008245A (ja) * 2022-07-07 2024-01-19 富士通株式会社 情報処理プログラム、情報処理方法、および情報処理装置

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220067688A1 (en) * 2020-09-01 2022-03-03 Lg Electronics Inc. Automated shopping experience using cashier-less systems
JP7626113B2 (ja) * 2022-09-05 2025-02-04 トヨタ自動車株式会社 情報処理装置及び方法
US12462284B2 (en) * 2022-10-28 2025-11-04 Access Credit Leasing Inc. Terminal and user interface methods for facilitating transactions
KR102623502B1 (ko) * 2023-03-09 2024-01-11 주식회사 파인더스에이아이 결제 서비스를 제공하는 전자 장치 및 이를 이용한 결제 서비스의 제공 방법
JP2024131549A (ja) * 2023-03-16 2024-09-30 グローリー株式会社 商品販売システム及び商品販売方法
JP2024131561A (ja) * 2023-03-16 2024-09-30 グローリー株式会社 商品販売システム及び商品販売方法
KR102581701B1 (ko) 2023-04-17 2023-09-26 주식회사 파인더스에이아이 매장의 결제 서비스를 제공하는 전자 장치 및 방법
JP2025075560A (ja) 2023-10-31 2025-05-15 グローリー株式会社 位置推定システム、位置推定装置、位置推定方法及び位置推定プログラム
JP2025174719A (ja) * 2024-05-17 2025-11-28 東芝テック株式会社 取引処理システム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001076261A (ja) 1999-08-31 2001-03-23 Hitachi Ltd セルフレジシステム
JP2003187026A (ja) * 2001-12-19 2003-07-04 Fujitsu Ltd 施設管理支援装置
JP2012104062A (ja) * 2010-11-12 2012-05-31 Japan Research Institute Ltd Ａｔｍ管理サーバ、システム及びプログラム
JP2017157216A (ja) * 2016-02-29 2017-09-07 サインポスト株式会社 情報処理システム

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW200822751A (en) * 2006-07-14 2008-05-16 Objectvideo Inc Video analytics for retail business process monitoring
JP5704279B1 (ja) * 2014-10-14 2015-04-22 富士ゼロックス株式会社 関連付プログラム及び情報処理装置
HK1254860A1 (zh) * 2015-06-24 2019-07-26 Magic Leap, Inc. 用於购买的增强现实设备、系统和方法
JP6277494B2 (ja) * 2015-07-29 2018-02-14 パナソニックＩｐマネジメント株式会社 アプリケーション制御システム及びアプリケーション制御方法
US10633012B2 (en) * 2015-12-02 2020-04-28 Mastercard International Incorporated Self-checkout in retail stores
EP4410155A1 (en) * 2016-05-09 2024-08-07 Grabango Co. System and method for computer vision driven applications within an environment
US10445593B1 (en) * 2016-06-27 2019-10-15 Amazon Technologies, Inc. User interface for acquisition of group data
US11361127B2 (en) * 2016-07-22 2022-06-14 Nec Corporation Simulation device, simulation method, and storage medium
SE542438C2 (en) 2016-12-21 2020-05-05 Itab Scanflow Ab An automatic in-store registration system
AU2018243962B2 (en) * 2017-03-28 2022-11-17 Automaton, Inc. Methods and apparatus for locating RFID tags
US10963704B2 (en) * 2017-10-16 2021-03-30 Grabango Co. Multiple-factor verification for vision-based systems
JP2019208627A (ja) 2018-05-31 2019-12-12 富士電機株式会社 商品販売用棚什器及び商品特定方法
US10282852B1 (en) * 2018-07-16 2019-05-07 Accel Robotics Corporation Autonomous store tracking system
WO2020018585A1 (en) * 2018-07-16 2020-01-23 Accel Robotics Corporation Autonomous store tracking system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001076261A (ja) 1999-08-31 2001-03-23 Hitachi Ltd セルフレジシステム
JP2003187026A (ja) * 2001-12-19 2003-07-04 Fujitsu Ltd 施設管理支援装置
JP2012104062A (ja) * 2010-11-12 2012-05-31 Japan Research Institute Ltd Ａｔｍ管理サーバ、システム及びプログラム
JP2017157216A (ja) * 2016-02-29 2017-09-07 サインポスト株式会社 情報処理システム

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114078299A (zh) * 2021-11-15 2022-02-22 烟台创迹软件有限公司 一种商品结算方法、装置、电子设备及介质
JP2023114434A (ja) * 2022-02-04 2023-08-17 エヌ・シー・アール・コーポレイション 精算のための複数商品の製品認識
JP7446403B2 (ja) 2022-02-04 2024-03-08 エヌ・シー・アール・コーポレイション 精算のための複数商品の製品認識
JP2024008245A (ja) * 2022-07-07 2024-01-19 富士通株式会社 情報処理プログラム、情報処理方法、および情報処理装置
JP7831165B2 (ja) 2022-07-07 2026-03-17 富士通株式会社 情報処理プログラム、情報処理方法、および情報処理装置

Also Published As

Publication number Publication date
EP4075399A4 (en) 2023-01-18
JP2023014207A (ja) 2023-01-26
JPWO2021125357A1 (https=) 2021-06-24
JP7225434B2 (ja) 2023-02-20
EP4075399A1 (en) 2022-10-19
US20230027382A1 (en) 2023-01-26

Similar Documents

Publication Publication Date Title
JP7225434B2 (ja) 情報処理システム
CN108780596B (zh) 信息处理系统
JP7371614B2 (ja) 店舗管理装置および店舗管理方法
JP6653813B1 (ja) 情報処理システム
JP7398706B2 (ja) 不正防止システム、及び不正防止プログラム
US11049373B2 (en) Storefront device, storefront management method, and program
JP7545801B2 (ja) 情報処理システム、情報処理システムの制御方法及びプログラム
JP6836256B2 (ja) 情報処理システム
CN111919233A (zh) 店铺管理设备和店铺管理方法
JP7264401B2 (ja) 会計方法、装置及びシステム
US12293346B2 (en) Information processing apparatus, information processing method, and storage medium
EP3474183A1 (en) System for tracking products and users in a store
JP2020077275A (ja) 商品決済システム、商品運搬カート及び商品決済方法
TWM570489U (zh) 智慧商店購物系統
TWI760521B (zh) 智慧商店購物系統及使用其之購物方法
JP7582308B2 (ja) 店舗システム、情報処理装置、情報処理方法、およびプログラム
JP2024037466A (ja) 情報処理システム、情報処理方法及びプログラム
US20230142558A1 (en) Frictionless Retail Stores and Cabinets

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20900801

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021565702

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020900801

Country of ref document: EP

Effective date: 20220714