JP2020173711A - Information processing device, information processing program, information processing method, and information processing system - Google Patents

Abstract

To associate the action of a user in a closed space with a use object utilized by the user with good accuracy.SOLUTION: Provided is an information processing device 40, comprising: an ID acquisition unit 431 for acquiring a cart ID corresponding to a cart utilized in a store and a customer ID corresponding to a customer utilizing the cart; a storage unit 42 in which the card ID and the customer ID are stored in association; an image acquisition unit 432 for acquiring an imaged image in which an identification mark corresponding to the card ID is imaged by an imaging device that images the inside of the store; a card ID specification unit 433 for specifying the cart ID from the identification mark imaged in the imaged image; and an action specification unit 434 for specifying an action ID corresponding to the action in the store of the person imaged in the imaged image. The cart is characterized in that the cart is provided with an identification mark, and the cart ID specified by the cart ID specification unit is stored in association with the action ID in the storage unit.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing device, an information processing program, an information processing method, and an information processing system.

In recent years, in retail stores such as supermarkets, a technique for associating a user (person) with a user (container) such as a shopping cart in which a user plans to purchase a product has been proposed (for example, Patent Document 1). reference).

The technique disclosed in Patent Document 1 recognizes a user and a user from an image captured by a camera that images the inside of the store, and identifies a behavior pattern of the user in the store. The technology then associates the user with one user, or associates the user with a plurality of users (groups), based on the identified behavioral pattern. As a result, in this technology, by using purchase history data such as a POS (Point Of Sales) system, it is possible to perform analysis such as behavior analysis and sales analysis of the user and the group to which the user belongs.

The technique disclosed in Patent Document 1 extracts a user from a captured image and recognizes the user by image processing. However, when the user is recognized from the captured image by image processing, the processing load on the device that executes the image processing (hereinafter referred to as "image processing device") is large in order to improve the recognition accuracy. Therefore, the image processing device is required to have high computing performance.

Further, in the technique disclosed in Patent Document 1, the user and the user are associated with each other based on the behavior pattern of the user. Therefore, when the association between the user and the user is updated, the accuracy of the association may decrease. As a result, an error may occur in the association between the user and the user.

Japanese Patent No. 5704279

The present invention provides an information processing device, an information processing program, an information processing method, and an information processing device capable of accurately associating a user's behavior in a closed space with an object used by the user in the closed space. The purpose is to provide an information processing system.

The present invention is an information processing device, and includes an ID acquisition unit for acquiring a cart ID corresponding to a cart used in a store, a customer ID corresponding to a customer using the cart, and a cart ID and a customer ID. A storage unit that is associated and stored, an image acquisition unit that acquires an image in which an identification tag corresponding to the cart ID is captured by an image pickup device that images the inside of the store, and a cart ID from the identification tag captured in the captured image. The cart has an identification unit for specifying a cart ID and an action identification unit for specifying an action ID corresponding to the action of the person captured in the captured image in the store. The cart is provided with an identification sign and has a cart ID. The cart ID specified by the specific unit and the action ID are associated and stored in the storage unit.

According to the present invention, it is possible to accurately associate the behavior of the user in the closed space with the material used by the user in the closed space.

It is a system block diagram which shows the embodiment of the information processing system which concerns on this invention. It is a block diagram which shows the structural example of the store which uses the information processing system of FIG. It is a schematic diagram which shows typically the use thing which the information processing system of FIG. 1 has. It is a functional block diagram of the processing apparatus included in the use thing of FIG. It is a functional block diagram of the image pickup apparatus provided in the information processing system of FIG. It is a functional block diagram of the advertisement display device included in the information processing system of FIG. It is a functional block diagram which shows the embodiment of the information processing apparatus which concerns on this invention. It is a schematic diagram which shows the example of the information stored in the storage part included in the information processing apparatus of FIG. It is a schematic diagram which shows another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a schematic diagram which shows still another example of the information stored in the storage part included in the information processing apparatus of FIG. 7. It is a flowchart which shows the operation of the information processing system of FIG. It is a flowchart of ID identification processing included in the operation of FIG. It is a sequence diagram of the ID identification process of FIG. It is a flowchart of the action specifying process included in the action of FIG. It is a sequence diagram of the action identification process of FIG. It is a flowchart of the display device determination process included in the operation of FIG. It is a sequence diagram of the display device determination process of FIG. 23. It is a flowchart of the sales promotion ID determination process included in the operation of FIG. It is a sequence diagram of the sales promotion ID determination process of FIG. It is a flowchart of the reaction identification process included in the operation of FIG. It is a sequence diagram of the reaction identification process of FIG. 27. It is a flowchart of the update process included in the operation of FIG.

Hereinafter, embodiments of the information processing apparatus, the information processing program, the information processing method, and the information processing system according to the present invention will be described with reference to the drawings.

● Information processing system ●
First, an embodiment of an information processing system (hereinafter referred to as "the present system") according to the present invention will be described.

This system associates a user with a user's behavior by associating the user with the user in a closed space. In addition, this system provides the user with sales promotion information corresponding to the user according to the behavior of the user in the closed space. Furthermore, this system obtains the user's reaction to the sales promotion information provided to the user.

The "closed space" is a three-dimensional space partitioned by an arbitrary range, and is a space in which the system can specify the behavior of the user. The closed space includes a space that is physically partitioned and a space that is not physically partitioned. That is, for example, a closed space is an internal space (sales floor) of a store such as a supermarket or a convenience store, an internal space of each store of a shopping mall or an airport duty-free shop where many stores are connected, an outdoor sales floor or a parking lot of a store, etc. Including outdoor space.

A "user" is a person who acts while using an object in a closed space. That is, for example, a user is a customer who purchases a product in a sales floor.

A "utilized item" is an item used by a user in a closed space. That is, for example, the used material includes a shopping cart, a trolley, a shopping basket, and an ID-attached strap (tag) used by the user in the store.

"Sales promotion information" is information such as advertisements and discount coupons that encourage users to purchase products.

"User behavior" refers to the act of the user stopping in front of the product (shelf) in a closed space, the act of the user touching the product, the act of the user placing the product on the product, and the user of the product. This includes the act of passing in front, the act of the user viewing promotional information, and the act of the user returning the product from the cart to the shelf.

"User reaction" is the act of the user stopping in front of the product (shelf) after the user sees the sales promotion information, the act of the user placing the product on the product, and the user in front of the product. This includes the act of passing by, the act of the user viewing promotional information, and the act of the user returning the product from the cart to the shelf. That is, the reaction of the user is the behavior of the user after the user sees the sales promotion information. That is, the user's behavior includes the user's reaction.

In the embodiment described below, the content of the present invention will be described by exemplifying a case where the behavior of a customer who uses a shopping cart is specified by the information processing device according to the present invention in a supermarket sales floor. That is, the sales floor of a supermarket is an example of a closed space according to the present invention, a shopping cart is an example of a product according to the present invention, and a customer is an example of a user (customer) according to the present invention.

FIG. 1 is a system configuration diagram showing an embodiment of this system.
FIG. 2 is a configuration diagram showing a configuration example of a store.
FIG. 2 shows a state in which five shelves Sc (1-5) are arranged in the sales floor R of the store Sh, and three customers Hm (1-3) are shopping in the sales floor R.

The system 1 includes a shopping cart (hereinafter referred to as "cart") 10, an imaging device 20, an advertisement display device 30, an information processing device (hereinafter referred to as "the device") 40 according to the present invention, and a POS (hereinafter referred to as "the device"). Point Of Sales) The terminal 50 and the operation terminal 60 are included.

The device 40 is connected to each of the cart 10, the image pickup device 20, the advertisement display device 30, the POS terminal 50, and the operation terminal 60 via a network N (communication line) that uses a wired communication method or a wireless communication method. To. The network N is, for example, a communication network such as the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), and a Wi-Fi (registered trademark).

The cart 10 is a container for temporarily storing (loading) the product to be purchased until the customer Hm who shop in the sales floor R pays the price. The cart 10 is used by the customer Hm in the sales floor R, and moves in the sales floor R together with the customer Hm. In the present embodiment, the system 1 includes a plurality of carts 10 (1-6) (six in the present embodiment). The configuration of the cart 10 will be described later.

The number of carts in this embodiment is not limited to "6".

The image pickup device 20 images the inside of the sales floor R. The image pickup device 20 is, for example, a camera unit that captures a still image in the sales floor R at predetermined time intervals. The image pickup device 20 is arranged on the ceiling surface or wall surface of the sales floor R so that the shelves Sc and aisles in the sales floor R, the advertisement display device 30, and the customers Hm and the cart 10 moving in the sales floor R can be imaged. .. In the present embodiment, the system 1 includes at least one (seven in the present embodiment) imaging device 20 (1-7). The configuration of the image pickup apparatus 20 will be described later.

The image pickup device may be a terminal device having a camera function including an image pickup unit 23 (see FIG. 5) described later. That is, for example, the imaging device may be a smartphone having a camera function, a mobile phone, a tablet terminal, a laptop computer, or a digital camera having a communication function.

The number of image pickup devices in this embodiment is not limited to "7".

Further, the image pickup apparatus in the present embodiment may capture a moving image or may capture a moving image and a still image.

Further, in the present embodiment, the areas in the sales floor imaged by each imaging device may overlap. That is, for example, a plurality of imaging devices may image a common area in the sales floor R. In this case, the orientation of each imaging device that images the same region is different.

Furthermore, the arrangement of the imaging device in the present embodiment is not limited to the ceiling surface and the wall surface of the sales floor. That is, for example, the image pickup apparatus may be arranged on shelves arranged so as to face each other across the aisle. Specifically, the imaging device may be arranged on a shelf board of a shelf or a price rail attached to the shelf board.

The advertisement display device 30 notifies the customer Hm who makes a purchase in the sales floor R of the sales promotion information. The advertisement display device 30 is, for example, a so-called digital signage device. The advertisement display device 30 is arranged in the sales floor R. In the present embodiment, the system 1 includes at least one (nine in the present embodiment) advertisement display device 30 (1-9). The configuration of the advertisement display device 30 will be described later.

The number of advertisement display devices in the present embodiment is not limited to "9".

The device 40 identifies the behavior of the customer Hm in association with the cart 10 used by the customer Hm based on the captured image captured by the image pickup device 20. The device 40 is, for example, a personal computer such as a server. The device 40 is arranged, for example, in the office Of of the store Sh. The configuration of the device 40 will be described later.

As described above, the device 40 is realized by, for example, a personal computer. In the present device 40, an information processing program according to the present invention (hereinafter referred to as "the present program") operates, and the present program cooperates with the hardware resources of the present device 40 to process information according to the present invention described later. Realize the method (hereinafter referred to as "this method"). That is, this program causes the computer to function as the present device 40.

The POS terminal 50 executes information processing such as a product settlement process and a product sales data generation process. The POS terminal 50 is, for example, a known register terminal. The POS terminal 50 is arranged, for example, in the accounting area in the sales floor R. In the present embodiment, the system 1 includes three POS terminals 50 (1-3). Since the configuration of the POS terminal 50 and the processing of the POS terminal 50 are well known, detailed description of the configuration and processing of the POS terminal 50 will be omitted.

The number of POS terminals in this embodiment is not limited to "3".

The operation terminal 60 is a terminal operated by a clerk of the store Sh. The operation terminal 60 is, for example, a personal computer. The operation terminal 60 is arranged, for example, in the office Of of the store Sh.

The device may have the functions of the POS terminal and the operation terminal. That is, for example, the present device, the POS terminal, and the operation terminal may be a common personal computer.

● Cart Configuration FIG. 3 is a schematic diagram schematically showing a state in which the cart 10 is viewed from the side.
In the figure, the shopping cart Ba placed on the cart 10 and the customer Hm using the cart 10 are shown by broken lines.

The cart 10 includes a frame 11, a plurality of wheels 12 (1, 2, ...), At least one identification mark 13 (1, 2, 3 ...), a processing device 14, a reading device 15, and so on. To be equipped.

The frame 11 constitutes the skeleton of the cart 10. The frame 11 forms a portion such as a grip portion 111 of the cart 10, a mounting portion 112 on which a product or a shopping cart Ba is placed, and the like.

The wheels 12 are attached to the frame 11 and assist the movement of the frame 11.

The identification mark 13 is a sign unique to the cart 10 assigned to the cart 10 by the system 1 (device 40) to identify the cart 10. The identification mark 13 is, for example, a two-dimensional code such as a barcode or a QR code (registered trademark). The identification mark 13 is attached to a position such as the front portion or the side portion of the frame 11 where the image pickup device 20 can easily image the image.

The identification mark is not limited to the two-dimensional code as long as the cart can be identified. That is, for example, the identification mark may be a character such as a number or an alphabet.

Also, the identification mark may be formed directly on the frame. That is, for example, the identification mark may be engraved on the frame or may be represented by the unevenness of the frame.

FIG. 4 is a functional block diagram of the processing device 14.

The processing device 14 is a terminal on which the system 1 (cart 10) executes information processing described later. The processing device 14 is, for example, a tablet terminal. The processing device 14 includes a communication unit 141, a storage unit 142, a display unit 143, and a control unit 144.

The communication unit 141 communicates with the present device 40 via the network N. The communication unit 141 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 142 stores information necessary for the system 1 (cart 10) to execute information processing described later. The storage unit 142 is, for example, a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The storage unit 142 has, for example, a database such as a member master database MDB1 (hereinafter referred to as “member MDB1”) and a product master database MDB2 (hereinafter referred to as “product MDB2”), and product information read by the customer Hm by the reading device 15. Remember ,. Details of each of the member MDB1 and the product MDB2 will be described later.

The display unit 143 functions as an operation unit for the customer Hm to operate the processing device 14, and also functions as a notification unit for notifying the customer Hm of sales promotion information. That is, the display unit 143 can function as an example of the display device in the present invention. The display unit 143 is, for example, a touch panel type monitor.

The control unit 144 executes various types of information processing according to a program to control the operation of the entire processing device 14. The control unit 144 integrates processors such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), and DSP (Digital Signal Processor), and an ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array). It is composed of a circuit and a semiconductor memory such as a RAM (Read Access Memory) and a ROM (Read Only Memory).

The processing device may be a terminal capable of executing information processing described later, and is not limited to a tablet terminal. That is, for example, the processing device may be a personal computer, a PDA (Personal Digital Assistant), a mobile phone, or a smartphone.

The reading device 15 reads information on the customer Hm and the product. The reading device 15 is an optical device such as a bar code reader that can optically read a product code (for example, a two-dimensional code attached to a product) or a membership code (for example, a two-dimensional code attached to a membership card). It is a reading device.

The membership code and the product code are not limited to the two-dimensional code. That is, for example, each of the membership code and the product code may be a bar code (one-dimensional code).

The processing device 14 and the reading device 15 constitute one cart terminal. The cart terminal is arranged in the frame 11 at a position where the customer Hm can easily operate the cart terminal (for example, in the vicinity of the grip portion 111).

In addition, the cart in the present invention may be provided with a basket portion in which products can be directly put in without going through a shopping basket.

The system may also include different types of carts. That is, for example, the system may include a small cart for children, a large cart that can carry a large number of products, and a medium-sized cart that carries one shopping cart.

● Configuration of the image pickup device FIG. 5 is a functional block diagram of the image pickup device 20.

The image pickup device 20 includes a communication unit 21, a storage unit 22, an image pickup unit 23, and a control unit 24.

The communication unit 21 communicates with the apparatus 40 via the network N. The communication unit 21 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 22 stores the captured image captured by the imaging unit 23. The storage unit 22 is a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The imaging unit 23 images the inside of the sales floor R. The image pickup unit 23 is composed of, for example, a lens group, a shutter, an image sensor, an A / D converter, and the like. The captured image captured by the imaging unit 23 is stored in the storage unit 22.

The control unit 24 executes various types of information processing according to a program to control the operation of the entire image pickup apparatus 20. The control unit 24 is composed of, for example, a processor such as a CPU, MPU, or DSP, an integrated circuit such as an ASIC or FPGA, and a semiconductor memory such as RAM or ROM.

● Configuration of advertisement display device FIG. 6 is a functional block diagram of the advertisement display device 30.

The advertisement display device 30 includes a communication unit 31, a storage unit 32, a display unit 33, an imaging unit 34, and a control unit 35.

The communication unit 31 communicates with the present device 40 via the network N. The communication unit 31 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 32 stores sales promotion information and an captured image captured by the imaging unit 34. The storage unit 32 is a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The display unit 33 notifies the customer Hm of the sales promotion information. The display unit 33 is, for example, a monitor such as a liquid crystal display (Liquid Crystal Display), an organic EL display (Organic Electro-Luminescence Display), or an LED display (Light Emitting Diode Display). The display unit 33 is an example of the display device in the present invention.

The image pickup unit 34 takes an image of the periphery of the advertisement display device 30 (for example, in front of the display unit 33) in the sales floor R. The image pickup unit 34 is composed of, for example, a lens group, a shutter, an image sensor, an A / D converter, and the like. The captured image captured by the imaging unit 34 is stored in the storage unit 32.

The control unit 35 executes various information processing according to the program to control the operation of the entire advertisement display device 30. The control unit 35 is composed of, for example, a processor such as a CPU, MPU, or DSP, an integrated circuit such as an ASIC or FPGA, and a semiconductor memory such as RAM or ROM.

● Configuration of Information Processing Device FIG. 7 is a functional block diagram showing an embodiment of the device 40.

The device 40 includes a communication unit 41, a storage unit 42, and a control unit 43.

The communication unit 41 communicates with the cart 10, the image pickup device 20, the advertisement display device 30, the POS terminal 50, and the operation terminal 60 via the network N. The communication unit 41 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 42 stores information necessary for the device 40 to execute the method described later. The storage unit 42 is a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The storage unit 42 includes a member MDB1, a product MDB2, a shelf master database MDB3 (hereinafter referred to as “shelf MDB3”), a cart master database MDB4 (hereinafter referred to as “cart MDB4”), and an imaging device master database MDB5 (hereinafter referred to as “imaging device MDB5”). ”), Display device master database MDB6 (hereinafter referred to as“ display device MDB6 ”), sales promotion information master database MDB7 (hereinafter referred to as“ sales promotion information MDB7 ”), behavior master database MDB8 (hereinafter referred to as“ behavior MDB8 ”). , Reaction master database MDB9 (hereinafter referred to as "reaction MDB9"), action history master database MDB10 (hereinafter referred to as "action history MDB10"), reaction history master database MDB11 (hereinafter referred to as "reaction history MDB11"), purchase history master Database MDB12 (hereinafter referred to as "purchase history MDB12"), login information database DB1 (hereinafter referred to as "login DB1"), cart position information database DB2 (hereinafter referred to as "cart position DB2"), display device database DB3 (hereinafter referred to as "" A database such as a display device DB3), an action database DB4 (hereinafter referred to as “action DB4”), and a reaction database DB5 (hereinafter referred to as “reaction DB5”) is stored. Details of each of the databases MDB1-MDB12 and DB1-DB5 will be described later.

The control unit 43 executes various information processing according to a program to control the operation of the entire apparatus 40. The control unit 43 is composed of, for example, a processor such as a CPU, MPU, and DSP, an integrated circuit such as an ASIC and an FPGA, and a semiconductor memory such as a RAM and a ROM.

The control unit 43 includes an ID acquisition unit 431, an image acquisition unit 432, a cart ID identification unit 433, an action identification unit 434, a positional relationship identification unit 435, a sales promotion decision unit 436, and a sales promotion reaction identification unit 437. A display device determination unit 438 is provided.

The ID acquisition unit 431 acquires the cart ID of the cart 10 and the user ID of the user of the cart 10 from the cart 10 via the communication unit 41. The method of acquiring the ID by the ID acquisition unit 431 will be described later.

The "user ID" is information used by the apparatus 40 (system 1, cart 10) to identify each of a plurality of customer Hms, and is identification information corresponding to each customer Hm using the cart 10. .. The user ID is an example of the customer ID in the present invention. The user ID is issued in advance by the user of this system 1 based on an act such as membership registration by the customer Hm. The user ID is stored in the storage unit 42 in association with the customer-related information.

“Customer-related information” includes, for example, information on attributes such as age and gender of customer Hm, information on valuable value such as point balance and prepaid balance held by customer Hm, and customer Hm such as action ID and reaction ID described later. Includes information on actions / reactions in the sales floor R and information on product purchase history.

The "cart ID" is information used by the apparatus 40 (system 1) to identify each of the plurality of carts 10, and is identification information corresponding to each cart 10 used by the customer Hm in the sales floor R. is there. The cart ID is an example of a product ID in the present invention. The cart ID is issued in advance by the user of the system 1 for each cart 10, for example.

The image acquisition unit 432 acquires an image captured by the image pickup device 20 from the storage unit 42. The method of acquiring the captured image by the image acquisition unit 432 will be described later.

The image acquisition unit in the present embodiment may acquire an captured image directly from the image pickup device via the communication unit.

The cart ID identification unit 433 identifies the cart ID from the identification mark 13 captured in the captured image. The cart ID specifying unit 433 is an example of the used product ID specifying unit in the present invention. The method of specifying the cart ID by the cart ID specifying unit 433 will be described later.

The action specifying unit 434 specifies an action ID corresponding to the action of the customer Hm (person) captured in the captured image in the sales floor R. The method of specifying the action ID by the action specifying unit 434 will be described later.

The "behavior ID" is information used by the present device 40 (this system 1) to identify each of the above-mentioned "user actions", and is identification information corresponding to the user's actions. The action ID is, for example, issued in advance by the user of the system 1 for each type of action.

The positional relationship specifying unit 435 specifies the positional relationship between the cart 10 and the customer Hm in the captured image based on the positional relationship between a part of the cart 10 and the customer Hm in the captured image. The method of specifying the positional relationship by the positional relationship specifying unit 435 will be described later.

The sales promotion decision unit 436 determines the sales promotion ID corresponding to the sales promotion information displayed on the advertisement display device 30 from the plurality of sales promotion information. The method of determining the sales promotion ID by the sales promotion decision unit 436 will be described later.

The “sales promotion ID” is information used by the apparatus 40 (system 1) to identify each of a plurality of sales promotion information, and is identification information corresponding to the sales promotion information. The sales promotion ID is issued in advance by the user of the system 1 for each sales promotion information, for example.

The sales promotion reaction specifying unit 437 specifies a reaction ID corresponding to the reaction of the customer Hm to the sales promotion information displayed on the advertisement display device 30. The method of specifying the reaction ID by the sales promotion reaction specifying unit 437 will be described later.

The "reaction ID" is information used by the apparatus 40 (system 1) to identify each of the above-mentioned "user reactions", and is identification information corresponding to the user's reaction. The reaction ID is issued in advance by the user of the system 1 for each type of reaction, for example.

The display device determination unit 438 determines the advertisement display device 30 on which sales promotion information such as advertisements and discount coupons is displayed from among the plurality of advertisement display devices 30. The method of determining the advertisement display device 30 by the display device determination unit 438 will be described later.

● Database ●
Next, each database MDB1-MDB12 and DB1-DB5 will be described.

● Member Master Database FIG. 8 is a schematic diagram showing an example of information stored in the member MDB1.

The "member MDB1" is, for example, a database in which a user ID and a part of customer-related information are associated and stored for each customer Hm.

In this figure, for example, the customer-related information "40 years old", "male", "1,000P", and "10,000 yen" are associated with the customer Hm of the user ID "U001" and stored in the member MDB1. Indicates that you are.

Here, "information Y is associated with information X" indicates that information X and information Y are linked to each other. As a result, for example, the system 1 (cart 10, the device 40) can read the information Y associated with the information X and stored in the database from the database by referring to the database using the information X. Is. Further, for example, the system 1 (cart 10, the device 40) can read the information X associated with the information Y and stored in the database from the database by referring to the database using the information Y. is there. That is, for example, the system 1 (cart 10, device 40) refers to the member MDB1 using the user ID "U001", and thereby, the customer-related information "40 years old", "male", "1,000P", "1,000P". "10,000 yen" can be read from the member MDB1. The same applies to other databases below.

● Product master database

The "product MDB2" is, for example, a database in which a product ID and product-related information are associated and stored for each product.

The "product ID" is information used by the system 1 (cart 10, the device 40) to identify each of a plurality of products, and is identification information corresponding to each product. The product ID is, for example, issued in advance by the user of the system 1 for each product.

The "product-related information" includes, for example, information on product attributes such as product name and price, information on product placement, information on product inventory, and information on product expiration dates.

The system 1 (cart 10, device 40) can read product-related information such as product attributes, arrangements, and inventory corresponding to the product ID by referring to the product MDB2 using, for example, the product ID. ..

● Shelf master database “Shelf MDB3” is a database in which, for example, a shelf ID, a shelf position information, a partition ID, and a product ID corresponding to the partition ID are associated and stored for each shelf Sc.

The “shelf ID” is information used by the system 1 (device 40) to identify each shelf Sc, and is identification information corresponding to each shelf Sc. The shelf ID is issued in advance by the user of this system 1 for each shelf Sc, for example.

The "shelf position information" is information regarding the position (coordinates) where each shelf Sc is arranged in the sales floor R.

The "section ID" is information used by the system 1 (device 40) to identify a section (shelf allocation) in each shelf Sc, and is identification information corresponding to the section (shelf allocation) in the shelf Sc. Is. The partition ID is represented using, for example, rows and columns. That is, for example, the partition ID “a1” indicates a partition in the “a” column “1” row of the shelf Sc.

The “section in the shelf Sc” is an area where goods are displayed (arranged) in the shelf Sc. The division is defined for each product.

The system 1 (device 40) can read the shelf position information of the shelves corresponding to the shelf ID and the products for each shelf allocation by referring to the shelf MDB3 using the shelf ID, for example.

● Cart master database “Cart MDB4” is a database in which, for example, a cart ID, an identification mark 13, and cart attribute information are associated and stored for each cart 10.

The "cart attribute information" is information related to the attributes of the cart 10, such as the type and size of the cart 10, the purchase destination of the cart 10, and the purchase date of the cart 10.

The system 1 (device 40) can read the cart ID of the cart 10 corresponding to the identification mark 13 and the type and size of the cart 10 by referring to the cart MDB4 using the identification mark 13, for example. is there.

● Imaging device master database The “imaging device MDB5” is a database in which, for example, an imaging device ID, an imaging device position information, and an imaging area information are associated and stored for each imaging device 20.

The "imaging device ID" is information used by the system 1 (the device 40) to identify each of the imaging devices 20, and is identification information corresponding to each imaging device 20. The image pickup device ID is issued in advance by the user of the system 1 for each image pickup device 20, for example.

The "imaging device position information" is information regarding the position (coordinates) in which each imaging device 20 is arranged in the sales floor R.

The “imaging area information” is information regarding the area of the sales floor R imaged by each imaging device 20. That is, the imaging area information includes information regarding the position of the subject captured in the captured image.

The system 1 (device 40) can read the position of the subject imaged by the image pickup device 20 corresponding to the image pickup device ID by referring to the image pickup device MDB5 using, for example, the image pickup device ID.

● Display device master database The “display device MDB6” is a database in which, for example, a display device ID, display device position information, and imaging area information are associated and stored for each advertisement display device 30.

The "display device ID" is information used by the system 1 (the device 40) to identify each of the advertisement display devices 30, and is identification information corresponding to each advertisement display device 30.

The "display device position information" is information regarding the position where each advertisement display device 30 is arranged in the sales floor R.

The “imaging area information” is information regarding the area of the sales floor R imaged by the imaging unit 34 of the advertisement display device 30. That is, the imaging area information includes information regarding the position of the subject captured in the captured image.

In the system 1 (device 40), for example, by referring to the display device MDB6 using the display device ID, the position of the advertisement display device 30 corresponding to the display device ID and the subject imaged by the imaging unit 34 can be obtained. The position can be read.

● Sales promotion information master database “Sales promotion information MDB7” is a database in which, for example, a sales promotion ID and sales promotion information are associated and stored for each sales promotion information.

The system 1 (device 40) can read the sales promotion information corresponding to the sales promotion ID by referring to the sales promotion information MDB7 using the sales promotion ID, for example.

● Behavior master database FIG. 9 is a schematic diagram showing an example of information stored in the behavior MDB8.

The "behavior MDB8" is, for example, a database in which an action ID and an action of a customer Hm are stored in association with each action of the customer Hm.

The figure shows, for example, that the action ID “A001” is associated with the action “stop in front of the product” of the customer Hm and is stored in the action MDB8.

The system 1 (device 40) can read the action of the customer Hm corresponding to the action ID by referring to the action MDB8 using the action ID, for example.

● Reaction master database FIG. 10 is a schematic diagram showing an example of information stored in the reaction MDB9.

The "reaction MDB9" is, for example, a database in which a reaction ID and a reaction of a customer Hm are stored in association with each reaction of the customer Hm.

The figure shows, for example, that the reaction “touching the product” of the customer Hm is associated with the reaction ID “R002” and stored in the reaction MDB9.

The system 1 (device 40) can read the reaction of the customer Hm corresponding to the reaction ID by referring to the reaction MDB9 using the reaction ID, for example.

● Action history master database FIG. 11 is a schematic diagram showing an example of information stored in the action history MDB10.

The "behavior history MDB10" is, for example, a database in which a user ID and an action history information are associated and stored for each customer Hm.

The "behavior history information" is information regarding the history of the past behavior ID of the customer Hm corresponding to the user ID. The action history information includes, for example, an action ID, an action time information, and an action position information.

The “action time information” is information regarding the time when the action of the person specified by the system 1 (the device 40) is captured by the image pickup device 20. That is, in the present embodiment, the action time information is the time when the captured image of the action of the person is captured. The action time information is acquired from the captured image as, for example, image metadata.

The "action position information" is information regarding the position (coordinates) in which the person has acted when the system 1 (device 40) identifies the behavior of the person. The action position information is, for example, the image pickup area information of the image pickup device 20 that images the behavior of a person, or the shelf position information of the shelf Sc imaged in the captured image.

In the figure, for example, for the user ID "U001", the action ID "A005, A001, ..., A001", the action time information "T1, T2, ..., Tn", and the action position information "X1" are shown. , X2, ..., Xn ”is associated and stored in the action history MDB10.

The system 1 (device 40) can read the past action history of the user (customer Hm) corresponding to the user ID by referring to the action history MDB10 using the user ID, for example.

● Reaction history master database FIG. 12 is a schematic diagram showing an example of information stored in the reaction history MDB11.

The "reaction history MDB11" is, for example, a database in which a user ID and reaction history information are associated and stored for each customer Hm.

The "reaction history information" is information regarding the history of the past reaction ID of the customer Hm corresponding to the user ID. The reaction history information includes, for example, a reaction ID, a reaction time information, and a reaction position information.

The “reaction time information” is information regarding the time when the reaction of the person specified by the system 1 (the device 40) is captured by the image pickup device 20. That is, in the present embodiment, the reaction time information is the time when the captured image of the reaction of the person is captured. The reaction time information is acquired from the captured image as, for example, image metadata.

The "reaction position information" is information regarding the position (coordinates) in which the person reacts when the system 1 (device 40) identifies the reaction of the person. The reaction position information is, for example, the imaging area information of the imaging device 20 that images the reaction of a person, or the shelf position information of the shelf Sc imaged in the captured image.

In the figure, for example, for the user ID "U001", the reaction ID "R004, R004, ..., R001", the reaction time information "T11, T12, ..., T1n", and the reaction position information "X11" are shown. , X12, ..., X1n ”is associated and stored in the reaction history MDB11.

The system 1 (device 40) can read the past reaction history of the user (customer Hm) corresponding to the user ID by referring to the reaction history MDB11 using, for example, the user ID.

● Purchase history master database “Purchase history MDB12” is a database in which, for example, a user ID and product purchase history information are associated and stored for each customer Hm.

The "product purchase history information" is information related to the history of products purchased in the past by the customer Hm corresponding to the user ID. The product purchase history information includes, for example, information on the product, the number of products, the purchase date and time, and the like.

The system 1 (device 40) can read the purchase history of the customer Hm's product corresponding to the user ID by referring to the purchase history MDB12 using, for example, the user ID.

● Login information database FIG. 13 is a schematic diagram showing an example of information stored in the login DB1.

The "login DB1" is, for example, a database in which a cart ID, a login time, and a user ID are associated and stored for each cart 10 logged in (used) to the customer Hm.

The figure shows, for example, that the login time “T21” and the user ID “U001” are associated with the cart ID “C001” and stored in the login DB1.

The system 1 (device 40) can read the user ID (customer Hm) of the cart 10 corresponding to the cart ID by referring to the login DB 1 using the cart ID.

● Cart location information database DB2
FIG. 14 is a schematic diagram showing an example of information stored in the cart position DB2.

The “cart position DB2” is a database in which, for example, a cart ID, time information, and cart position information are associated and stored for each cart 10 logged in (used) to the customer Hm.

The “time information” is information regarding the time when the cart 10 is imaged by the image pickup device 20. That is, in the present embodiment, the time information is the time when the captured image is captured. The time information is acquired from the captured image as, for example, image metadata.

The "cart position information" is information regarding the position (coordinates) of the cart 10 for each time information in the sales floor R. The cart position information is, for example, the imaging area information of the imaging device 20 that images the cart 10 or the shelf position information of the shelf Sc imaged in the captured image.

In the figure, for example, the time information "T31, T32, ..., T3n" and the cart position information "X31, X32, ..., X3n" are associated with the cart ID "C001" to the cart position DB2. Indicates that it is remembered.

The system 1 (device 40) can read the movement history of the cart 10 logged in to the customer Hm by referring to the cart position DB2 using the cart ID.

● Display device database FIG. 15 is a schematic diagram showing an example of information stored in the display device DB3.

In the "display device DB3", for example, a display device ID, a sales promotion ID corresponding to the displayed sales promotion information, and a cart ID corresponding to the cart 10 to which the sales promotion information is notified are provided for each advertisement display device 30. A database that is associated with and stored in.

The figure shows, for example, that the sales promotion ID “aaa1” and the cart ID “C001” are associated with the display device ID “P001” and stored in the display device DB3.

The system 1 (device 40) uses the display device ID to refer to the display device DB3 to notify the sales promotion information notified by the advertisement display device 30 and the sales promotion information by the advertisement display device 30. The cart 10 and can be read.

● Behavior database FIG. 16 is a schematic diagram showing an example of information stored in the behavior DB 4.

The "action DB4" is, for example, a database in which a cart ID, an action ID of a person (customer Hm) associated with the cart ID, an action time information, and an action position information are associated and stored. ..

In the figure, for example, for the cart ID "C001", the action ID "A005, A001, ..., A005" action time information "T41, T42, ..., T4n" action position information "X41, X42," ..., X4n ”is associated and stored in the action DB4.

The system 1 (device 40) can read the action history of the customer Hm who is using the cart 10 by referring to the action DB 4 using the cart ID.
● Reaction database FIG. 17 is a schematic diagram showing an example of information stored in the reaction DB 5.

The "reaction DB5" is, for example, a database in which a cart ID, a reaction ID of a person (customer Hm) associated with the cart ID, reaction time information, and reaction position information are associated and stored. ..

In the figure, for example, for the cart ID “C001”, the reaction ID “R005, R001, ..., R002” reaction time information “T51, T52, ..., T5n” reaction position information “X51, X52, ..., X5n ”is associated and stored in the reaction DB5.

The system 1 (device 40) can read the reaction history of the customer Hm using the cart 10 by referring to the reaction DB 5 using the cart ID.

The information stored in association with each database is not limited to the present embodiment. Further, the information stored in each database may be stored in duplicate in a plurality of databases. Further, the storage unit may store one or a plurality of databases in which a plurality of databases in the present embodiment are integrated.

● Operation of this system ●
Next, the operation of this system 1 will be described.

FIG. 18 is a flowchart showing the operation of the system 1.

The operations of this system 1 are updated with ID identification process (S1), action identification process (S2), display device determination process (S3), sales promotion ID determination process (S4), reaction identification process (S5), and so on. The process (S6) includes.

The "ID identification process (S1)" is a process for specifying the user ID of the customer Hm and the cart ID of the cart 10 used by the customer Hm. The system 1 identifies the customer Hm corresponding to the cart 10 by the ID identification process (S1).

The “behavior identification process (S2)” is a process for specifying an action ID corresponding to the action of a person (customer Hm) associated with the cart 10 based on the captured image. The system 1 identifies the behavior of the customer Hm who uses the cart 10 via the cart 10 by the behavior identification process (S2). As a result, the system 1 can acquire the history of the behavior of the customer Hm corresponding to the user ID associated with the cart ID.

The “display device determination process (S3)” is a process of determining the advertisement display device 30 that notifies (displays) the sales promotion information to the customer Hm based on the cart ID. The system 1 can notify the customer Hm of appropriate sales promotion information based on the behavior / purchase history of the customer Hm by the display device determination process (S3).

The “sales promotion ID determination process (S4)” is a process of determining the sales promotion ID corresponding to the sales promotion information to be notified to the customer Hm based on the behavior ID and the user ID of the customer Hm associated with the cart ID. The system 1 can notify the customer Hm of appropriate sales promotion information based on the behavior / purchase history of the customer Hm by the sales promotion ID determination process (S4).

The “reaction identification process (S5)” is a process for specifying a reaction ID corresponding to the reaction of the customer Hm to the sales promotion information. The system 1 can determine the influence (effect) of the sales promotion information on the customer Hm by the reaction identification process (S5).

The "update process (S6)" reacts with the action history MDB10 based on the action ID of the customer Hm specified in the action identification process (S2) and the reaction ID of the customer Hm specified in the reaction identification process (S5). This is a process for updating the information stored in each of the history MDB 11.

● ID identification process FIG. 19 is a flowchart of the ID identification process (S1) of the system 1.
FIG. 20 is a sequence diagram of the ID identification process (S1) of the system 1.

The ID identification process (S1) is started, for example, when the customer Hm logs in to the information processing system of the cart 10. The information processing system is stored in the storage unit 142 of the cart 10. The login to the information processing system is executed, for example, by having the customer Hm who uses the cart 10 read the membership code attached to the membership card he / she owns to the reading device 15 of the cart 10.

First, when a code such as a one-dimensional code or a two-dimensional code is read by the reading device 15 (S101), the control unit 144 of the cart 10 determines whether or not the code is a member code (S102). The determination of the code is executed, for example, by the control unit 144 referring to the member MDB1 stored in the storage unit 142. When the code is not a member code (“No” in S102), the control unit 144 waits until the next code is read.

On the other hand, when the code is a member code (“Yes” in S102), the control unit 144 reads the member MDB1 from the storage unit 142 and identifies the user ID corresponding to the member code (S103).

Next, the control unit 144 associates the cart ID of the own machine with the user ID corresponding to the member code (S104), and stores the cart ID in the storage unit 142 (S105). That is, the control unit 144 functions as an association unit that associates the cart ID with the user ID. By this association, the system 1 (device 40) can recognize the cart 10 and the customer Hm using the cart 10 as a 1: 1 relationship.

Next, the control unit 144 transmits the cart ID and the user ID associated with the cart ID to the apparatus 40 via the communication unit 141 (S106).

The control unit 43 (ID acquisition unit 431) of the apparatus 40 acquires (receives) the cart ID and the user ID associated with the cart ID from the cart 10 via the communication unit 41 (S107).

The control unit 43 of the apparatus 40 stores the cart ID and the user ID associated with the cart ID in the login DB 1 of the storage unit 42 (S108). That is, the cart ID and the user ID are stored in association with each other in the storage unit 42.

In addition, instead of the control unit of the cart, the control unit of this device receives the cart ID of the cart and the membership code read by the reading device of the cart from the cart, and the user corresponding to the membership code. The ID may be specified, the cart ID and the user ID may be associated with each other, and stored in the storage unit (login DB1) of the present device.

● Action identification process FIG. 21 is a flowchart of the action identification process (S2) of the system 1.
FIG. 22 is a sequence diagram of the action identification process (S2) of the system 1.

First, the image pickup apparatus 20 executes the image pickup process (S201). In the imaging process (S201), the imaging device 20 uses the imaging unit 23 to capture an captured image in which the imaging region in the sales floor R is captured. The captured image is stored in the storage unit 22 and transmitted to the apparatus 40 via the communication unit 21.

Next, the control unit 43 of the present device 40 receives the captured image from the image pickup device 20 via the communication unit 41 and stores it in the storage unit 42 (S202).

Next, the control unit 43 (image acquisition unit 432) acquires an captured image from the storage unit 42 (S203).

Next, the control unit 43 (cart ID identification unit 433) executes known image processing on the captured image to determine whether or not the identification mark 13 of the cart 10 is captured in the captured image (S204). .. The image processing is, for example, a known pattern matching processing.

When the identification mark 13 is not imaged (“No” in S204), the process of the system 1 returns to the process (S201).

On the other hand, when the identification mark 13 is imaged (“Yes” in S204), the control unit 43 (cart ID identification unit 433) refers to the cart MDB4 using, for example, the identification mark 13 to the identification mark 13. The corresponding cart ID is specified (S205). At this time, the control unit 43 acquires the cart position information and the time information of the cart 10, and stores the cart position information and the time information in the cart position DB 2 of the storage unit 42.

The cart position information is acquired, for example, based on the imaging area information. Further, the time information is acquired based on, for example, the metadata of the captured image.

Next, the control unit 43 executes known image processing on the captured image to determine whether or not a person is captured in the captured image (S206).

"Known image processing" detects a person (object) using, for example, machine learning-based deep learning image recognition (for example, SSD: Single Shot Multibox Detector model, YOLO: You Only Look Once model, etc.). It is a process or a process of detecting a person (object) by combining the deep learning image recognition and a known skeletonization technique.

When the person is not imaged (“No” in S206), the process of the system 1 returns to the process (S201).

On the other hand, when a person is being imaged (“Yes” in S206), the control unit 43 (positional relationship specifying unit 435) specifies the positional relationship between the cart 10 and the person in the captured image (S207).

Regarding the positional relationship, for example, the cart 10 is specified by performing the same known image processing as the above-mentioned processing (S206) on the captured image, and the distance between the cart 10 and the person detected in the processing (S206). Is specified by calculating. In the present embodiment, the positional relationship between the cart 10 and the person in the captured image is the distance between the cart 10 and the person. The same distance is, for example, the distance between the center of gravity of the image of the specified cart 10 and the center of gravity of the image of the specified person. Specifically, for example, the control unit 43 (the positional relationship specifying unit 435), the gravity center position coordinates in the captured image of the person detected in the process (S206) _(x H, _{y H)} and, detected in this processing (S207) The coordinates of the position of the center of gravity (x _C , y _C ) in the captured image of the cart 10 are calculated. Next, the control unit 43 (positional relationship specifying unit 435) calculates the linear distance between the center of gravity position coordinates (x _H , y _H ) and the center of gravity position coordinates (x _C , y _C ) in the captured image. In this way, the control unit 43 (positional relationship specifying unit 435) calculates the distance between the cart 10 and the person, and specifies the calculated distance as the positional relationship between the cart 10 and the person. The control unit 43 (positional relationship specifying unit 435) stores the specified positional relationship (for example, the distance between the coordinates of the center of gravity) in the storage unit 42.

The control unit (positional relationship specifying unit) identifies a part of the cart by known image processing, and specifies the positional relationship between the part of the cart and the person in the captured image as the positional relationship between the cart and the person. You may. In other words, the control unit (positional relationship specifying unit) may specify the positional relationship between the cart and the person in the captured image based on the positional relationship between a part of the cart and the person in the captured image. That is, for example, the control unit (positional relationship specifying unit) determines the distance between the cart processing device and the person (for example, the center of gravity position coordinates of the processing device in the captured image and the center of gravity position coordinates of the person in the captured image). The linear distance between them) may be specified as the positional relationship between the cart and the person in the image pickup apparatus.

Next, the control unit 43 (behavior identification unit 434) identifies the behavior of the person captured in the captured image in the sales floor R (S208). In the present embodiment, the behavior of the person specified by the action specifying unit 434 is, as described above, the act of the person stopping in front of the product (shelf Sc), the act of the person touching the product, and the act of the person placing the product on the cart 10. This includes an act, an act of a person passing in front of a product (shelf Sc), an act of a person viewing promotional information, and an act of a person returning a product from the cart 10 to the shelf Sc.

In the process of identifying the behavior of a person (S208), the control unit 43 (behavior identification unit 434) is specified in the process (S206) by using, for example, a known skeletonization technique (for example, extraction of a skeleton by skeletal modeling). Performs skeletonization of the image of the person. Next, the control unit 43 (behavior identification unit 434) preliminarily sets the user's behavior based on, for example, the shape of the skeleton and the positional relationship (distance) between the skeleton and the product (or shelf Sc). The behavior of the photographed person is specified from a plurality of defined behaviors (hereinafter referred to as "default behavior group"). In this case, the goods and shelves Sc are examples of subjects in the present invention.

After extracting the skeleton, the control unit (behavior specifying unit) may specify the behavior of the imaged person from the default behavior group based on the angle of each joint of the skeleton. That is, for example, the control unit (behavior identification unit) calculates the angles of joints such as shoulders, elbows, hips, and knees from the extracted skeleton, and generates multidimensional time series data of the angles of each joint for each person. To do. Next, the control unit (behavior identification unit) classifies the multidimensional time series data into classes using a known clustering technique, and identifies the behavior of the imaged person from the default behavior group. In this case, the default action group is stored in the action master database, for example.

The control unit (behavior specifying unit) may specify the product from the captured image by known image processing and specify the positional relationship between the skeleton and the product. Further, the control unit (behavior specifying unit) may specify the behavior of the person based on the shape of the skeleton and the positional relationship between the skeleton and the partition of the shelf.

Further, the control unit (behavior specifying unit) may specify the behavior of a person based on the difference between a plurality of captured images captured in time series.

Further, the control unit (behavior specifying unit) may specify the behavior of a person by using a plurality of captured images captured from different directions. In this case, the action of the person is prioritized and stored in the storage unit, and the control unit (behavior specifying unit) may specify the action having a high priority as the action of the person. That is, for example, the priority of the act of touching the product is set higher than the priority of the act of stopping in front of the product, and the act of stopping in front of the product is specified based on the captured image of the front of the shelf. When the act of touching the product is specified based on the captured image of the side surface of the shelf, the control unit (behavior specifying unit) may specify the act of touching the product as the behavior of the person.

Furthermore, the control unit (behavior specifying unit) may specify the act of having the customer read the product by the reading device of the cart as the act of the person placing the product on the cart.

When the control unit 43 (behavior identification unit 434) identifies the behavior of the captured person, for example, the control unit 43 acquires the behavior time information and the behavior position information based on the captured image in which the behavior is captured.

Next, the control unit 43 (behavior specifying unit 434) identifies the action ID corresponding to the specified action with reference to the action MDB8 stored in the storage unit 42 (S209).

Next, the control unit 43 associates the cart 10 captured in the captured image with the person based on the positional relationship between the cart 10 and the person described above (S210).

Here, when a plurality of people are captured in the captured image, the control unit 43 associates the person with the shortest distance between the cart 10 and the person with the cart 10. That is, for example, two people (for example, person A and person B) and one cart 10 are imaged in the captured image, and the distance between the center of gravity position coordinates of the person A and the center of gravity position coordinates of the cart 10 is the person. When the distance between the center-of-gravity position coordinates of B and the center-of-gravity position coordinates of the cart 10 is larger than the distance, the control unit 43 associates the person B with the cart 10.

Next, the control unit 43 associates the action ID, the action time information, and the action position information of the person associated with the cart 10 with the cart ID, and stores them in the action DB 4 of the storage unit 42 (S211). As described above, the cart ID and the action ID are stored in the action DB 4 of the storage unit 42 based on the positional relationship in the captured image between the cart 10 and the person.

When the next captured image is stored in the storage unit, when the identification mark is not captured in the captured image (“No” in S204), or when a person is not imaged (“No” in S206). "), The processing of this system may return to the processing (S203).

● Display device determination process FIG. 23 is a flowchart of the display device determination process (S3) of the system 1.
FIG. 24 is a sequence diagram of the display device determination process (S3) of the system 1.

First, the imaging device 20 executes an imaging process (S301). In the imaging process (S301), the imaging device 20 uses the imaging unit 23 to capture an captured image in which the imaging region in the sales floor R is captured. The captured image is stored in the storage unit 22 and transmitted to the apparatus 40 via the communication unit 21.

Next, the control unit 43 of the present device 40 receives the captured image from the image pickup device 20 via the communication unit 41 and stores it in the storage unit 42 (S302).

Next, the control unit 43 (image acquisition unit 432) acquires an captured image from the storage unit 42 (S303).

Next, the control unit 43 (cart ID identification unit 433) executes known image processing on the captured image to determine whether or not the identification mark 13 of the cart 10 is captured in the captured image (S304). ..

When the identification mark 13 is not imaged (“No” in S304), the process of the system 1 returns to the process (S301).

When the next captured image is stored in the storage unit and the identification mark is not captured in the captured image (“No” in S304), the processing of this system may return to the processing (S303). Good.

On the other hand, when the identification mark 13 is imaged (“Yes” in S304), the control unit 43 (cart ID identification unit 433) refers to the cart MDB4 using, for example, the identification mark 13 to the identification mark 13. Identify the corresponding cart ID (S305). At this time, the control unit 43 (cart ID specifying unit 433) acquires the cart position information and the time information of the cart 10 and stores the cart position information and the time information in the cart position DB 2 of the storage unit 42.

Next, the control unit 43 (display device determination unit 438) determines the advertisement display device 30 on which the sales promotion information is displayed from among the plurality of advertisement display devices 30 (S306).

Here, the advertisement display device 30 on which the sales promotion information is displayed is determined based on, for example, the cart ID specified by the cart ID specifying unit 433. That is, for example, the control unit 43 (display device determination unit 438) uses the user ID associated with the cart ID and stored in the login DB 1 to refer to the purchase history MDB12 and acquire the product purchase history information. To do. Next, the control unit 43 (display device determination unit 438) identifies the product that the user (customer Hm) of the cart 10 has purchased in the past from the same information, and is close to the shelf Sc displaying the product. The advertisement display device 30 is determined as the advertisement display device 30 on which the sales promotion information is displayed.

The control unit (display device determination unit) may determine the advertisement display device based on the action ID associated with the cart ID and stored in the action DB 4. That is, for example, the control unit (display device determination unit) estimates the products that can be purchased by the cart user (customer) based on the action ID, and the advertisement display device close to the shelf displaying the products. May be determined as an advertisement display device on which promotional information is displayed. In other words, the control unit (display device determination unit) predicts the user's next action from the user's past action, and displays promotional information on the advertisement display device that can prompt the user's next action. It may be decided as an advertisement display device.

Next, the control unit 43 (sales promotion decision unit 436) corresponds to the sales promotion information displayed on the advertisement display device 30 determined in the process (S306) based on the product purchase history information and the action ID acquired in the process (S306). The sales promotion ID to be used is determined (S307).

Next, the control unit 43 transmits the sales promotion information corresponding to the sales promotion ID determined in the process (S307) to the advertisement display device 30 determined in the process (S306) via the communication unit 41 (S308).

Next, the control unit 35 of the advertisement display device 30 receives the sales promotion information via the communication unit 31 and stores the sales promotion information in the storage unit 32 (S309).

Next, the control unit 35 reads the sales promotion information from the storage unit 32 and causes the display unit 33 to display the sales promotion information (S310).

● Sales promotion ID determination process FIG. 25 is a flowchart of the sales promotion ID determination process (S4) of the system 1.
FIG. 26 is a sequence diagram of the sales promotion ID determination process (S4) of the system 1.

First, the advertisement display device 30 executes an imaging process (S401). In the image pickup process (S401), the advertisement display device 30 uses the image pickup unit 34 to take an image of an image captured in a region of the sales floor R near the own machine (advertisement display device 30). The captured image is stored in the storage unit 32 and transmitted to the apparatus 40 via the communication unit 31.

Next, the control unit 43 of the present device 40 receives the captured image from the advertisement display device 30 via the communication unit 41 and stores it in the storage unit 42 (S402).

Next, the control unit 43 (image acquisition unit 432) acquires an captured image from the storage unit 42 (S403).

Next, the control unit 43 (cart ID identification unit 433) executes known image processing on the captured image to determine whether or not the identification mark 13 of the cart 10 is captured in the captured image (S404). .. The image processing is, for example, a known pattern matching processing.

When the identification mark 13 is not imaged (“No” in S404), the process of the system 1 returns to the process (S401).

On the other hand, when the identification mark 13 is imaged (“Yes” in S404), the control unit 43 (cart ID identification unit 433) refers to the cart MDB4 using, for example, the identification mark 13 to the identification mark 13. The corresponding cart ID is specified (S405).

Next, the control unit 43 determines whether or not a plurality of carts 10 are captured in the captured image (S406).

When a plurality of carts 10 are captured in the captured image (“Yes” in S406), the control unit 43 (positional relationship specifying unit 435) determines the positional relationship between each cart 10 and the advertisement display device 30 in the sales floor R. Identify (S407).

The positional relationship is specified, for example, by performing known image processing on the captured image to specify the position in the captured image of each cart 10.

Next, the control unit 43 identifies the cart 10 closest to the advertisement display device 30 among the plurality of carts 10 (S408). Further, when one cart 10 is imaged in the captured image (“No” in S406), the control unit 43 identifies the cart 10 as the cart 10 closest to the advertisement display device 30 (S408).

Next, the control unit 43 (sales promotion decision unit 436) determines the sales promotion ID for the sales promotion information displayed on the display unit 33 of the advertisement display device 30 from the plurality of sales promotion information based on the cart ID (S409). That is, for example, the control unit 43 (sales promotion determination unit 436) determines the sales promotion ID based on the user ID associated with the cart ID and stored in the storage unit 42. Specifically, the control unit 43 (sales promotion decision unit 436) refers to the purchase history MDB12 using the user ID, and acquires the product purchase history information of the customer Hm corresponding to the user ID. Next, the control unit 43 (sales promotion determination unit 436) determines the sales promotion ID corresponding to the sales promotion information that can promote the purchase motivation of the customer Hm corresponding to the user ID based on the product purchase history information. In other words, the control unit 43 (sales promotion determination unit 436) determines advertisements and coupon information for products that can match the tastes of the customer Hm based on the past purchase history of the customer Hm.

In addition, the control unit (sales promotion decision unit), for example, refers to the action history MDB10 using the user ID, acquires the action history information of the customer corresponding to the user ID, and based on the action history information, The sales promotion ID may be determined. In other words, the control unit (sales promotion determination unit) may determine advertisements and coupon information for products that can promote the customer's behavior based on the history of the customer's past behavior (when the customer visited the store before the previous time).

Further, the control unit (sales promotion determination unit) refers to the reaction history MDB11 using, for example, the user ID, acquires the reaction history information of the customer corresponding to the user ID, and based on the reaction history information, The sales promotion ID may be determined. In other words, the control unit (sales promotion decision unit) may determine the advertisement or coupon information of the product that can promote the customer's reaction based on the history of the customer's past reaction (at the time of the previous visit to the store).

Further, the control unit (sales promotion decision unit) refers to the action DB 4 using, for example, the cart ID, acquires the action ID of the person (customer) associated with the cart ID, and uses the action ID corresponding to the action ID. The sales promotion ID may be determined based on the behavior of the person. In other words, the control unit (sales promotion decision unit) may determine the advertisement or coupon information of the product that can promote the customer's next action based on the history of the customer's past (at the time of this visit) behavior. ..

Furthermore, the control unit (sales promotion decision unit) refers to the reaction DB 5 using, for example, the cart ID, acquires the reaction ID of the person (customer) associated with the cart ID, and corresponds to the reaction ID. The sales promotion ID may be determined based on the reaction of the user. In other words, the control unit (advertising determination unit) may determine the advertisement or coupon information of the product that can promote the customer's next reaction based on the history of the customer's past (at the time of this visit) reaction. ..

Next, the control unit 43 transmits the sales promotion information corresponding to the determined sales promotion ID to the advertisement display device 30 via the communication unit 41 (S410).

Next, the control unit 35 of the advertisement display device 30 receives the sales promotion information via the communication unit 31 and stores the sales promotion information in the storage unit 32 (S411).

Next, the control unit 35 reads the sales promotion information from the storage unit 32 and causes the display unit 33 to display the sales promotion information (S412).

The control unit of the advertisement display device may receive the sales promotion ID from the device. In this case, the display device MDB6 is stored in the storage unit of the advertisement display device.

Further, the system may execute the sales promotion ID determination process based on the captured image captured by the imaging device. That is, for example, the control unit of this device identifies each of the cart and the advertisement display device from the captured image, and based on the positional relationship between the cart and the advertisement display device, the advertisement display device is the most among the plurality of carts. A nearby cart may be specified as a cart to which promotional information is notified. Further, for example, the control unit of this device displays advertisements for which promotional information is displayed on the advertisement display device closest to the cart among a plurality of advertisement display devices based on the positional relationship between the cart and the advertisement display device. It may be determined as a device.

● Reaction identification process FIG. 27 is a flowchart of the reaction identification process (S5) of the present system 1.
FIG. 28 is a sequence diagram of the reaction identification process (S5) of the system 1.

Among the processes of the reaction identification process (S5), the following process (S504-S511) differs from the above-mentioned action identification process (S2) process (S204-S211) in that the object to be specified differs between the action and the reaction. Except, it is common. Therefore, the detailed description of each process of the reaction specifying process (S5) will be omitted.

First, the imaging device 20 executes an imaging process (S501). In the imaging process (S501), the imaging device 20 uses the imaging unit 23 to capture an image captured on the shelf Sc on which the products corresponding to the sales promotion ID determined in the sales promotion ID determination process (S4) are displayed. Take an image. The captured image is stored in the storage unit 22 and transmitted to the apparatus 40 via the communication unit 21.

Next, the control unit 43 of the present device 40 receives the captured image from the image pickup device 20 via the communication unit 41 and stores it in the storage unit 42 (S502).

Next, the control unit 43 (image acquisition unit 432) acquires an captured image from the storage unit 42 (S503).

Next, the control unit 43 (cart ID identification unit 433) executes known image processing on the captured image, and whether the identification mark 13 of the cart 10 identified in the processing (S405) is imaged on the captured image. Whether or not it is determined (S504). The image processing is, for example, a known pattern matching processing.

When the identification mark 13 is not imaged (“No” in S504), the process of the system 1 returns to the process (S501).

On the other hand, when the identification mark 13 is imaged (“Yes” in S504), the control unit 43 (cart ID identification unit 433) identifies the cart ID of the cart 10 (S505), and also provides cart position information and time information. And store the cart position information and the time information in the cart position DB 2 of the storage unit 42.

Next, the control unit 43 executes a known image process similar to the above-mentioned process (S206) on the captured image, and determines whether or not a person is captured in the captured image (S506).

When the person is not imaged (“No” in S506), the process of the system 1 returns to the process (S501).

On the other hand, when a person is being imaged (“Yes” in S506), the control unit 43 (positional relationship specifying unit 435) specifies the positional relationship between the cart 10 and the person in the captured image (S507).

Next, the control unit 43 (sales promotion reaction specifying unit 437) identifies the reaction (behavior) of the person captured in the captured image in the sales floor R (S508). When the control unit 43 (sales promotion reaction specifying unit 437) identifies the reaction of the captured person, for example, the control unit 43 acquires the reaction time information and the reaction position information based on the captured image in which the reaction is captured. In the present embodiment, the reaction of the person specified by the control unit 43 (sales promotion reaction specifying unit 437) is, as described above, the act of the person stopping in front of the product (shelf Sc), the act of the person touching the product, and the reaction of the person. This includes the act of placing a product on a cart 10, the act of a person passing in front of a product (shelf Sc), the act of a person viewing promotional information, and the act of a person returning a product from the cart 10 to the shelf Sc.

Next, the control unit 43 (sales promotion reaction specifying unit 437) identifies the reaction ID corresponding to the specified reaction with reference to the reaction MDB9 stored in the storage unit 42 (S509).

Next, the control unit 43 associates the cart 10 captured in the captured image with the person based on the positional relationship between the cart 10 and the person described above (S510).

Here, when a plurality of people are captured in the captured image, the control unit 43 associates the person closest to the cart 10 with the cart 10.

Next, the control unit 43 associates the reaction ID, the reaction time information, and the reaction position information of the person associated with the cart 10 with the cart ID, and stores them in the reaction DB 5 of the storage unit 42 (S511). As described above, the cart ID and the reaction ID are stored in the reaction DB 5 of the storage unit 42 based on the positional relationship in the captured image between the cart 10 and the person.

When the next captured image is stored in the storage unit, when the identification mark is not captured in the captured image (“No” in S504), or when a person is not imaged (“No” in S506). "), The processing of this system may return to the processing (S503).

● Update process FIG. 29 is a flowchart of the update process (S6).

The update process (S6) is executed, for example, when the customer Hm logs off from the information processing system of the cart 10. The log-off from the information processing system is executed, for example, when the customer Hm finishes using the cart 10 (for example, when the POS terminal 50 completes the payment processing of the product placed on the cart 10).

When the customer Hm logs off from the information processing system of the cart 10 (S601), the control unit 43 of the present device 40 has a product purchase history of the customer Hm who has logged in to the cart 10 from the POS terminal 50 via the communication unit 41. Acquire information (S602).

Next, the control unit 43 reads the purchase history MDB 12 from the storage unit 42 and updates the information of the purchase history MDB 12 using the product purchase history information acquired from the POS terminal 50 (S603). The updated purchase history MDB 12 is stored in the storage unit 42.

Next, the control unit 43 reads the action DB 4 and the action history MDB 10 from the storage unit 42, and updates the information of the action history MDB 10 using the information stored in the action DB 4 (S604). Specifically, the action ID, the action time information, and the action position information stored in the action DB 4 are associated with the user ID as customer-related information of the customer Hm corresponding to the user ID associated with the cart ID. It is stored in the action history MDB10. The updated action history MDB10 is stored in the storage unit 42, and the action DB4 is reset (deleted).

Next, the control unit 43 reads the reaction DB 5 and the reaction history MDB 11 from the storage unit 42, and updates the information of the reaction history MDB 11 using the information stored in the reaction DB 5 (S605). Specifically, each of the reaction ID, reaction time information, and reaction position information stored in the reaction DB 5 is associated with the user ID as customer-related information of the customer Hm corresponding to the user ID associated with the cart ID. It is stored in the reaction history MDB11. The updated reaction history MDB11 is stored in the storage unit 42, and the reaction DB5 is reset (deleted).

● Summary ●
According to the embodiment described above, the system 1 (device 40) stores the cart ID and the user ID in association with each other. That is, the system 1 (device 40) identifies the cart 10 and its user (customer Hm) in a 1: 1 relationship. As a result, the information processing system 1 (information processing device 40) in the present embodiment can specify the information stored in association with the cart ID as the user's information by specifying the cart ID.

Further, according to the embodiment described above, the system 1 (device 40) stores the action ID corresponding to the action of the person captured together with the cart 10 in association with the cart ID in the captured image. As a result, the system 1 (device 40) identifies (estimates) a person who has performed an action corresponding to the action ID as a user (customer Hm) of the cart 10 corresponding to the cart ID via the cart ID. It is possible. That is, the system 1 (information processing device 40) in the present embodiment accurately determines the behavior of the user in the closed space (sales floor R) and the cart 10 used by the user via the cart 10. Well associate.

As described above, the system 1 (device 40) tracks only the cart 10 by image processing using the identification mark 13, and does not track the user (customer Hm) of the cart 10 by image processing. The system 1 (device 40) indirectly identifies the user of the cart 10 when the cart 10 is identified by associating the IDs with each other. In other words, the system 1 (device 40) can instantly identify the user of the cart 10 only by the process of identifying the cart 10. As a result, the system 1 (information processing device 40) in the present embodiment requires higher computing power than the conventional information processing system (conventional information processing device) that identifies customers by image recognition. It is possible to accurately associate the behavior of the user in the closed space with the material used by the user.

Further, the system 1 (device 40) does not require registration of a facial photograph or deep learning processing for identifying an individual from the captured image. Therefore, the system 1 (information processing device 40) in the present embodiment does not require high computing power as compared with the conventional information processing system (conventional information processing device) that identifies customers by image recognition. , It is possible to accurately associate the user's behavior in a closed space with the usage material used by the user.

Further, according to the embodiment described above, the system 1 (device 40) associates the cart ID and the action ID with the storage unit 42 based on the positional relationship between the cart 10 and the person in the imaging device. Remember in. That is, for example, the system 1 (device 40) associates the person with the shortest distance between the cart 10 and the person with the cart 10 as a user (customer Hm) of the cart 10. Normally, the customer Hm who shop in the sales floor R moves with the cart 10, and the customer Hm rarely acts away from the cart 10. Therefore, the system 1 (information processing device 40) in the present embodiment can associate the cart 10 with the user (customer Hm) of the cart 10 with high accuracy.

Furthermore, according to the embodiment described above, the system 1 (device 40) determines the sales promotion ID corresponding to the sales promotion information based on the cart ID. As described above, since the cart ID and the action ID are associated with each other, the system 1 (the device 40) notifies the user based on the action of the user (customer Hm) in the sales floor R. Promotional information can be determined. Further, as described above, since the cart ID and the user ID are associated with each other, the system 1 (device 40) informs the user based on the information such as the product purchase history information associated with the user ID. It is possible to determine the sales promotion information to be notified to. Therefore, the system 1 (information processing device 40) in the present embodiment is based on the history of past actions of the customer Hm (at the time of this visit), and advertisements and coupon information of products that can promote the next action of the customer Hm. Can be notified to the customer Hm.

Furthermore, according to the embodiment described above, the system 1 (device 40) determines the sales promotion ID corresponding to the sales promotion information based on the positional relationship between the cart 10 and the advertisement display device 30. Therefore, the system 1 (information processing device 40) in the present embodiment can notify the user of the cart 10 close to the advertisement display device 30 of the advertisement of the product suitable for the user and the discount coupon.

Furthermore, according to the embodiment described above, the system 1 (device 40) associates the cart ID with the reaction ID corresponding to the reaction of the person (customer Hm) to the sales promotion information, and stores the storage unit 42. Remember in. Therefore, the system 1 (device 40) can effectively acquire the history of the user's reaction to the sales promotion information, that is, the effect of the advertisement. As a result, the system 1 (information processing device 40) in the present embodiment can determine effective sales promotion information for the user based on the history of the user's reaction to the sales promotion information.

Furthermore, according to the embodiment described above, the system 1 (device 40) has customer-related information of the user (customer Hm) corresponding to the user ID whose action ID and reaction ID are associated with the cart ID. Remember as. Therefore, the system 1 (information processing device 40) in the present embodiment is based on the past behavior history of the customer Hm (at the time of the previous visit to the store), and the advertisement or coupon of the product that can promote the next action of the customer Hm. Information can be notified to the customer Hm.

Furthermore, according to the embodiment described above, the system 1 (device 40) is based on the cart ID, and the user of the cart 10 corresponding to the cart ID is selected from the plurality of advertisement display devices 30. The advertisement display device 30 for notifying the sales promotion information is determined. Therefore, the system 1 (information processing device 40) in the present embodiment can advertise a product that can promote the next action of the customer Hm based on the history of the past actions / reactions of the customer Hm (at the time of this visit). The coupon information can be notified to the customer Hm.

In the embodiment described above, the control unit 43 (sales promotion decision unit 436) of the present device 40 transmits the sales promotion information to the advertisement display device 30. Instead, the control unit of the device may send promotional information to the cart. In this case, the control unit of the present device specifies, for example, the cart ID in the same manner as the process (S405), determines the sales promotion ID in the same manner as in the process (S409), and specifies the sales promotion information corresponding to the determined sales promotion ID. Send to the cart corresponding to the given card ID. Next, the control unit of the cart stores the received sales promotion information in the storage unit and displays it on the display unit. This display unit functions as a display device in the present invention. As a result, the system (information processing device) in the present embodiment individually advertises products and coupons for each cart user (customer) based on the past behavior history and purchase history of the cart user. Information can be transmitted.

Further, the system (this device) only needs to be able to identify the cart and the user of the cart in a 1: 1 relationship, and it is not necessary to identify the user of the cart as a specific individual. That is, for example, this system (this device) generates a temporary user ID by a customer identification means such as an admission card lent from a store or a fingerprint identification device provided in a cart, and the user ID is used. May be associated with the cart ID. Even in this case, the system (information processing device) in the present embodiment can determine the sales promotion ID based on the past behavior history and reaction history of the customer (at the time of this visit).

Further, the system (this device) determines, for example, a sales promotion ID corresponding to the sales promotion information displayed on the display unit of the advertisement display device according to the user (customer)'s usage in the sales promotion ID determination process. You may. That is, for example, when the customer's product is a cart, the system determines the sales promotion ID of a heavy product (for example, a beer case containing 24 cans) as the sales promotion ID of the sales promotion information displayed on the display unit. When the customer's product is a shopping cart, the system may determine the sales promotion ID of a light product (for example, a beer pack containing 6 cans) as the sales promotion ID of the sales promotion information displayed on the display unit.

Furthermore, the system may use a shopping cart with an identification mark as a use. In this case, for example, the user may read the membership card and the shopping cart identification mark into the login terminal located at the sales floor. Further, for example, the membership code and the identification mark may be captured in the captured image by the user hanging the membership card from the neck with a strap or the like.

Furthermore, the system may use a shopping cart equipped with a small processing device as a use.

Furthermore, the advertisement display device does not have to include an imaging unit. In this case, the reaction identification process is executed based on the captured image captured by the imaging device.

Furthermore, among the advertisement display devices, some devices may include an imaging unit, and some other devices may not include an imaging unit.

Furthermore, this system (this device) extracts the feature amount (for example, the color / pattern of clothes) of the person when detecting the person from the captured image, and uses the same feature amount as the cart ID (or cart ID). It may be stored in association with the user ID associated with. In this case, the system (this device) may associate the cart with the person based on the same feature amount instead of the positional relationship between the cart and the person or with the same positional relationship. According to this configuration, in the processing (S210), for example, when a plurality of people are captured in the captured image, the system (this device) does not have the closest person between the cart and the person. The person with the best feature quantity may be associated with the cart.

Furthermore, in the ID identification process, when the customer logs in to the information processing system of the cart, the system (this device) extracts, for example, the feature amount of the customer from the captured image of the customer, and obtains the cart ID. The user ID and the feature amount may be associated and stored in the storage unit.

Furthermore, the imaging device may be a wide-angle camera capable of capturing a wide range. In this case, the number of image pickup devices arranged in the sales floor of the store is reduced. Therefore, the degree of freedom of wiring for arranging the imaging device is increased, and the imaging device can be efficiently arranged in a store where the space where the imaging device can be installed is small.

Furthermore, according to the embodiment described above, the present device 40 is composed of one computer. Instead, the device may be composed of a plurality of computers. That is, for example, this system may be composed of a plurality of computers and may include a group of computers that function as the present device. Specifically, for example, the present device (computer group) may be composed of a computer including a storage unit for storing captured images and a database, and a computer including a control unit for executing the present information processing. Further, for example, a plurality of computers distribute the functions of the ID acquisition unit, the image acquisition unit, the cart ID identification unit, the action identification unit, the positional relationship identification unit, the sales promotion determination unit, the sales promotion reaction identification unit, and the display device determination unit. You may prepare for it. In this case, the plurality of computers may send and receive information through a communication line, or may transfer information using a portable storage medium.

Furthermore, this system (this device) may at least execute ID identification processing and action identification processing. That is, for example, the system (this device) does not have to execute a part or all of the display device execution process, the sales promotion ID execution process, and the reaction identification process. In addition, this system (this device) may process ID identification processing, action identification processing, display device execution processing, sales promotion ID execution processing, reaction identification processing, and update processing in parallel, or continuously. You may.

Furthermore, the system (this device) does not have to process the action identification process, the display device execution process, the sales promotion ID execution process, and the reaction identification process in parallel. That is, for example, the system (this device) may execute the action identification process after the display device execution process, or may execute the display device execution process after the reaction identification process.

Furthermore, the embodiments described above do not unreasonably limit the content of the invention described in the claims. Moreover, not all of the configurations described in the above-described embodiments are essential configuration requirements of the present disclosure.

Hereinafter, the features of the information processing apparatus according to the present invention described so far will be summarized.

(Feature 1)
A cart ID corresponding to a cart (for example, cart 10) used in a store (for example, sales floor R) and a customer ID (for example, user ID) corresponding to a customer (for example, customer Hm) who uses the cart. And the ID acquisition unit (for example, ID acquisition unit 431) that acquires
A storage unit (for example, a storage unit 42) in which the cart ID and the customer ID are stored in association with each other.
An image acquisition unit (for example, image acquisition unit 432) that acquires an image captured by an image pickup device (for example, an image pickup device 20) that images the inside of a store and an identification mark (for example, an identification mark 13) corresponding to the cart ID. When,
A cart ID identification unit (for example, a cart ID identification unit 433) that identifies the cart ID from the identification mark captured in the captured image, and
An action identification unit (for example, an action identification unit 434) that specifies an action ID corresponding to an action in the store of a person (for example, customer Hm) captured in the captured image, and
Have
The cart comprises the identification mark.
The cart ID specified by the cart ID specifying unit and the action ID are associated and stored in the storage unit.
An information processing device (for example, an information processing device 40).

(Feature 2)
A positional relationship specifying unit (for example, a positional relationship specifying unit 435) that specifies the positional relationship between the cart and the person in the captured image.
Have
The cart ID and the action ID are stored in the storage unit based on the positional relationship.
The information processing device according to feature 1.

(Feature 3)
The positional relationship specifying unit identifies the positional relationship between the cart and the person in the captured image based on the positional relationship between a part of the cart and the person in the captured image.
The information processing device according to feature 2.

(Feature 4)
The action specifying unit identifies the action based on the subject (for example, the shelf Sc) captured in the captured image.
The information processing device according to feature 1.

(Feature 5)
The action specifying unit identifies the action based on the positional relationship between the subject and the person in the captured image.
The information processing device according to feature 4.

(Feature 6)
A display device (for example, an advertisement display device 30) for displaying sales promotion information notified to the person is connected via a communication line (for example, network N).
The storage unit stores sales promotion IDs corresponding to each of the plurality of sales promotion information, and stores the sales promotion IDs.
A sales promotion decision unit (for example, a sales promotion decision unit 436) that determines the sales promotion ID corresponding to the sales promotion information displayed on the display device from the plurality of sales promotion information.
Have
The sales promotion determination unit determines the sales promotion ID based on the cart ID specified by the cart ID identification unit.
The information processing device according to feature 1.

(Feature 7)
The sales promotion determination unit determines the sales promotion ID based on the action ID stored in the storage unit in association with the cart ID specified by the cart ID identification unit.
The information processing device according to feature 6.

(Feature 8)
The sales promotion determination unit determines the sales promotion ID based on the customer ID stored in the storage unit in association with the cart ID specified by the cart ID identification unit.
The information processing device according to feature 6.

(Feature 9)
A positional relationship specifying unit (for example, a positional relationship specifying unit 435) that specifies the positional relationship between the cart and the display device in the store.
Have
The sales promotion determination unit determines the sales promotion ID based on the positional relationship.
The information processing device according to feature 6.

(Feature 10)
The action ID is associated with the customer ID and stored in the storage unit as customer-related information of the customer corresponding to the customer ID associated with the cart ID and stored in the storage unit. Remembered in
The sales promotion decision unit determines the sales promotion ID based on the customer-related information.
The information processing device according to feature 6.

(Feature 11)
A sales promotion reaction specifying unit (for example, a sales promotion reaction specifying unit 437) that specifies a reaction ID corresponding to the reaction of the person to the sales promotion information displayed on the display device.
Have
The cart ID specified by the cart ID specifying unit and the reaction ID are associated and stored in the storage unit.
The information processing device according to feature 6.

(Feature 12)
The reaction ID is associated with the customer ID and stored in the storage unit as customer-related information of the customer corresponding to the customer ID associated with the cart ID and stored in the storage unit. Remembered in
The sales promotion decision unit determines the sales promotion ID based on the customer-related information.
The information processing device according to feature 11.

(Feature 13)
Connect to each of multiple display devices via a communication line,
A display device determination unit (for example, a display device determination unit 438) that determines a sales promotion display device on which a sales promotion is displayed from the plurality of display devices.
Have
The display device determination unit determines the sales promotion display device based on the cart ID specified by the cart ID identification unit.
The information processing device according to feature 1.

(Feature 14)
The display device determination unit determines the sales promotion display device based on the action ID stored in the storage unit in association with the cart ID.
The information processing device according to feature 13.

(Feature 15)
The cart comprises the display device.
The information processing device according to feature 6.

(Feature 16)
The display device is arranged in the store.
The information processing device according to feature 6.

1 Information processing system 10 carts (used items)
13 Identification mark 20 Imaging device 30 Advertisement display device 33 Display unit (display device)
40 Information processing device 42 Storage unit 431 ID acquisition unit 432 Image acquisition unit 433 Cart ID specification unit 434 Action specification unit 435 Positional relationship specification unit 436 Sales promotion decision unit 437 Sales promotion reaction specification unit 438 Display device determination unit

Claims (23)

An ID acquisition unit that acquires a cart ID corresponding to a cart used in the store and a customer ID corresponding to a customer using the cart.
A storage unit in which the cart ID and the customer ID are associated and stored.
An image acquisition unit that acquires an image obtained by capturing an identification mark corresponding to the cart ID by an image pickup device that images the inside of the store.
A cart ID specifying unit that identifies the cart ID from the identification mark captured in the captured image,
An action specifying unit that specifies an action ID corresponding to the action of the person captured in the captured image in the store, and
Have
The cart comprises the identification mark.
The cart ID specified by the cart ID specifying unit and the action ID are associated and stored in the storage unit.
An information processing device characterized by this. A positional relationship specifying unit that specifies the positional relationship between the cart and the person in the captured image,
Have
The cart ID and the action ID are stored in the storage unit based on the positional relationship.
The information processing device according to claim 1. The positional relationship specifying unit identifies the positional relationship between the cart and the person in the captured image based on the positional relationship between a part of the cart and the person in the captured image.
The information processing device according to claim 2. The action specifying unit identifies the action based on the subject captured in the captured image.
The information processing device according to claim 1. The action specifying unit identifies the action based on the positional relationship between the subject and the person in the captured image.
The information processing device according to claim 4. Connect to a display device that displays sales promotion information notified to the person via a communication line.
The storage unit stores sales promotion IDs corresponding to each of the plurality of sales promotion information, and stores the sales promotion IDs.
A sales promotion decision unit that determines the sales promotion ID corresponding to the sales promotion information displayed on the display device from the plurality of sales promotion information.
Have
The sales promotion determination unit determines the sales promotion ID based on the cart ID specified by the cart ID identification unit.
The information processing device according to claim 1. The sales promotion determination unit determines the sales promotion ID based on the action ID stored in the storage unit in association with the cart ID specified by the cart ID identification unit.
The information processing device according to claim 6. The sales promotion determination unit determines the sales promotion ID based on the customer ID stored in the storage unit in association with the cart ID specified by the cart ID identification unit.
The information processing device according to claim 6. A positional relationship specifying unit that specifies the positional relationship between the cart and the display device in the store,
Have
The sales promotion determination unit determines the sales promotion ID based on the positional relationship.
The information processing device according to claim 6. The action ID is associated with the customer ID and stored in the storage unit as customer-related information of the customer corresponding to the customer ID associated with the cart ID and stored in the storage unit. Remembered in
The sales promotion decision unit determines the sales promotion ID based on the customer-related information.
The information processing device according to claim 6. A sales promotion reaction specifying unit that specifies a reaction ID corresponding to the reaction of the person to the sales promotion information displayed on the display device,
Have
The cart ID specified by the cart ID specifying unit and the reaction ID are associated and stored in the storage unit.
The information processing device according to claim 6. The reaction ID is associated with the customer ID and stored in the storage unit as customer-related information of the customer corresponding to the customer ID associated with the cart ID and stored in the storage unit. Remembered in
The sales promotion decision unit determines the sales promotion ID based on the customer-related information.
The information processing device according to claim 11. Connect to each of multiple display devices via a communication line,
A display device determination unit that determines a sales promotion display device on which a sales promotion is displayed from among the plurality of display devices.
Have
The display device determination unit determines the sales promotion display device based on the cart ID specified by the cart ID identification unit.
The information processing device according to claim 1. The display device determination unit determines the sales promotion display device based on the action ID stored in the storage unit in association with the cart ID.
The information processing device according to claim 13. The cart comprises the display device.
The information processing device according to claim 6. The display device is arranged in the store.
The information processing device according to claim 6. The computer functions as the information processing device according to claim 1.
An information processing program characterized by this. A storage unit in which a cart ID corresponding to a cart used in a store and a customer ID corresponding to a customer using the cart are associated and stored.
It is an information processing method executed by an information processing device having
The information processing device
The step of acquiring the cart ID and the customer ID, and
A step of associating the cart ID with the customer ID and storing them in the storage unit,
A step of acquiring an image captured by an imaging device that images the inside of the store and an identification mark corresponding to the cart ID.
A step of identifying the cart ID from the identification mark captured in the captured image, and
A step of identifying an action ID corresponding to the action of the person captured in the captured image in the store, and
Have
The cart comprises the identification mark.
The identified cart ID and the action ID are associated and stored in the storage unit.
An information processing method characterized by this. An imaging device that captures the inside of the store and
An ID acquisition unit that acquires a cart ID corresponding to a cart used in the store and a customer ID corresponding to a customer using the cart.
A storage unit in which the cart ID and the customer ID are associated and stored.
An image acquisition unit that acquires an image captured by the identification mark corresponding to the cart ID by the image pickup device.
A cart ID specifying unit that identifies the cart ID from the identification mark captured in the captured image,
An action specifying unit that specifies an action ID corresponding to the action of the person captured in the captured image in the store, and
Have
The cart comprises the identification mark.
The cart ID specified by the cart ID specifying unit and the action ID are associated and stored in the storage unit.
An information processing system characterized by this. An ID acquisition unit that acquires a user ID corresponding to a user used in a closed space and a user ID corresponding to a user who uses the user.
A storage unit in which the user ID and the user ID are associated and stored.
An image acquisition unit that acquires an image captured by an image pickup device that images the inside of the closed space and an identification mark corresponding to the user ID.
A user ID specifying unit that identifies the user ID from the identification mark captured in the captured image,
An action specifying unit that specifies an action ID corresponding to the action of the person captured in the captured image in the closed space,
Have
The utilization has the identification mark and
The user ID specified by the user ID specifying unit and the action ID are associated and stored in the storage unit.
An information processing device characterized by this. The computer functions as the information processing device according to claim 20.
An information processing program characterized by this. A storage unit in which a user ID corresponding to a user used in a closed space and a user ID corresponding to a user using the user are associated and stored.
It is an information processing method executed by an information processing device having
The information processing device
A step of acquiring the user ID and the user ID, and
A step of associating the user ID with the user ID and storing the user ID in the storage unit.
A step of acquiring an image captured by an image pickup device that images the inside of the closed space and an identification mark corresponding to the user ID.
A step of identifying the user ID from the identification mark captured in the captured image, and
A step of specifying an action ID corresponding to the action of the person captured in the captured image in the closed space, and
Have
The utilization has the identification mark and
The identified use ID and the action ID are associated and stored in the storage unit.
An information processing method characterized by this. An imaging device that captures images in a closed space,
An ID acquisition unit that acquires a user ID corresponding to a user used in the closed space and a user ID corresponding to a user using the user.
A storage unit in which the user ID and the user ID are associated and stored.
An image acquisition unit that acquires an image captured by the identification mark corresponding to the usage ID by the image pickup device.
A user ID specifying unit that identifies the user ID from the identification mark captured in the captured image,
An action specifying unit that specifies an action ID corresponding to the action of the person captured in the captured image in the closed space,
Have
The utilization has the identification mark and
The user ID specified by the user ID specifying unit and the action ID are associated and stored in the storage unit.
An information processing system characterized by this.

Images