JP2022042014A - Control method, program, information processing device, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a control method, a program, an information processing device, and an information processing system for performing analysis by using a photographic image.

SOLUTION: A control method executed by an information processing device, comprises: processing of analyzing a relation between a sales promotion activity and a new repeater acquired by the sales promotion activity by using a photographic image photographed by an imaging apparatus; and processing of displaying a result by analysis as an effect by the sales promotion activity, in which the sales promotion activity is provision of a sample to a customer by a shop clerk to be performed at a shop.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to control methods, programs, information processing devices, and information processing systems.

Acquiring repeat customers who use the same store multiple times to purchase products at a store contributes to sales improvement. In order to improve sales and repeat rate, a method of analyzing the utterances and facial expressions of sales staff (sales staff) has been proposed.

Japanese Unexamined Patent Publication No. 2016-20736

By the way, in stores, various sales promotion (sales promotion) activities are carried out in order to acquire new repeaters (new repeaters), and it is necessary to understand how effective these sales promotion activities are in acquiring repeaters. Is difficult.

The present invention has been made in view of the above problems, and an object of the present invention is to easily understand the relationship between sales promotion activities and acquisition of new repeaters.

The control method executed by the information processing apparatus according to the embodiment includes a process of analyzing the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity by using the captured image taken by the photographing apparatus, and the above-mentioned. The sales promotion activity includes a process of displaying the result of the analysis as an effect of the sales promotion activity, and the sales promotion activity is a provision of a free sample to a customer by the clerk performed at the store.

According to one embodiment, the relationship between sales promotion activities and acquisition of new repeaters can be easily grasped.

It is a figure which shows an example of the structure of an information processing system. It is a figure which shows an example of the hardware composition of an information processing apparatus. It is a figure which shows an example of the hardware composition of a terminal apparatus. It is a figure which shows an example of the functional structure of an information processing apparatus. It is a figure which shows an example of photographed image information. It is a figure which shows an example of the customer information. It is a schematic diagram for demonstrating one configuration example of a clerk discrimination model. It is a schematic diagram for demonstrating one configuration example of a customer recognition model. It is a schematic diagram for demonstrating one configuration example of a clerk motion recognition model. It is a figure which shows an example of the functional structure of a terminal apparatus. It is a flowchart which shows an example of the 1st analysis processing by an information processing apparatus. It is a flowchart which shows an example of the 2nd analysis processing by an information processing apparatus. It is a figure which shows an example of the effect display screen. It is a figure which shows an example of the functional structure of the information processing apparatus which concerns on Embodiment 2. FIG. It is a figure which shows an example of the photographed image information which concerns on Embodiment 2. FIG. It is a schematic diagram for demonstrating the shooting position coordinates which concerns on Embodiment 2. It is a figure which shows an example of the customer information which concerns on Embodiment 2. FIG. It is a flowchart which shows an example of the 2nd analysis processing by the information processing apparatus which concerns on Embodiment 2. FIG.

Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. Regarding the description of the specification and the drawings according to each embodiment, the components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<System overview>
The outline of the information processing system according to this embodiment will be described. The information processing system according to the present embodiment is a system for analyzing the effect of the sales promotion activity, that is, the relationship between the sales promotion activity and the acquisition of new repeaters, using the image (photographed image) taken by the photographing device. be. In the present embodiment, for example, the larger the number of repeaters obtained through the sales promotion activity, the greater the effect of the sales promotion activity.

Further, the information processing system according to the present embodiment is intended for a store such as a retail store, but is not limited to this. For example, this information processing system may be targeted at event facilities and accommodation facilities, and may be targeted not only at one store but also at a plurality of stores.

Further, for example, the new repeater in the present embodiment is a customer who has become a new repeater. Specifically, a new repeater is a customer who has been provided with sales promotion activities when he / she has not become a repeater, and then visits the store to purchase (settlement) products related to the provided sales promotion activities. Not limited to. A product related to this sales promotion activity is also simply referred to as a related product, and this product includes, for example, a service or the like. In addition, when sales promotion activities related to a plurality of product categories (described later) are provided and a product related to one or more of these multiple product categories is purchased at a subsequent visit. , It can be said that it became a new repeater. In the present embodiment, the customer who becomes a new repeater shall be treated as a repeater thereafter, but the present invention is not limited to this.

Further, for example, the prospective customer in the present embodiment is a customer who has been provided with products related to sales promotion activities and has received these provided products. As described above, a prospective customer can be said to be a candidate for a new repeater because he / she becomes a new repeater when he / she purchases a product related to the above products when he / she visits the store from the next time onward.

<System configuration>
FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to the present embodiment. As illustrated in FIG. 1, the information processing system 1 according to the present embodiment includes an information processing device 10, a photographing device 20, and a terminal device 30. The information processing device 10, the photographing device 20, and the terminal device 30 are connected to each other so as to be able to communicate with each other via the network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public network, a mobile data communication network, or a combination thereof. Further, the network connecting the information processing device 10 and the photographing device 20, the network connecting the information processing device 10 and the terminal device 30, and the network connecting the photographing device 20 and the terminal device 30 are different from each other. There may be.

In the example of FIG. 1, the information processing system 1 includes one information processing device 10, one photographing device 20, and one terminal device 30, but a plurality of information processing devices 1 may be provided.

The information processing device 10 is a device (computer) that receives a photographed image from the photographing device 20, executes a predetermined process, and transmits information regarding the effect of the sales promotion activity to the terminal device 30. The information processing device 10 is a PC (Personal Computer), a smartphone, a tablet terminal, a server, or a combination thereof. The information processing device 10 will be described in detail later.

The photographing device 20 is also referred to as a camera, and is obtained by performing predetermined image processing on an image pickup element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) and an output signal of the image pickup element to obtain a photographed image (image data). ) Is generated by an image processing IC (Integrated Circuit), and a communication unit that performs wired or wireless communication is provided. The photographing apparatus 20 takes an image at a cycle of several to several tens of times per second, and transmits the image obtained by the photographing to the information processing apparatus 10. Further, the photographing device 20 can attach information such as its own camera ID (identifier) and the shooting date and time of the captured image to the captured image and transmit it to the information processing apparatus 10.

The photographing devices 20 are installed at a plurality of places in the store, and each photographing device 20 shall photograph a predetermined range in the store. The installation and settings may be made so that the ranges taken by each of the photographing devices 20 do not overlap and the entire store can be photographed by all the photographing devices 20. In the information processing system 1 according to the present embodiment, the photographing device 20 is installed at the entrance, and the photographing device 20 is above the product shelves, the ceiling of the store, etc. so that the shelves (product shelves) of a plurality of products can be photographed. It shall be installed in. Further, the photographing device 20 installed at the entrance of the store is installed for the purpose of photographing a person who comes to the store. Further, the photographing device 20 may be installed at the clearing counter of the store (the place where the cash register is located or the place where payment can be made), and in that case, the photographing device 20 is used for the purpose of photographing the person to be purchased. You may.

As an example, the terminal device 30 is a terminal used by a person (user) engaged in a store for browsing and operation. The terminal device 30 acquires information on the effect of the sales promotion activity transmitted by the information processing device 10 via the network N and displays it on the screen. Further, the image taken by the photographing apparatus 20 may be acquired from the information processing apparatus 10 via the network N. The terminal device 30 is, but is not limited to, a PC, a smartphone, or a tablet terminal.

<Hardware configuration>
Next, the hardware configuration of the information processing apparatus 10 will be described with reference to FIG. FIG. 2 is a diagram showing an example of the hardware configuration of the information processing apparatus 10. As shown in FIG. 2, the information processing apparatus 10 includes a processor 101, a memory 102, a storage 103, a communication I / F 104, an input / output I / F 105, and a drive connected to each other via a bus B1. The device 106 is provided. Further, the information processing device 10 includes an input device 107 and an output device 108.

The processor 101 controls each configuration of the information processing apparatus 10 and realizes the function of the information processing apparatus 10 by expanding the program stored in the storage 103 into the memory 102 and executing the program. The program executed by the processor 101 includes, but is not limited to, a control program such as an OS (Operating System) and an effect analysis program. When the processor 101 executes the control program, the control method according to the present embodiment is realized. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

The memory 102 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), or a combination thereof. The ROM is, for example, a PROM (Programmable ROM), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), or a combination thereof. The RAM is, for example, DRAM (Dynamic RAM), SRAM (Static RAM), MRAM (Magnetoresistive RAM), or a combination thereof.

The storage 103 stores an OS, a control program, and various data. The storage 103 is, for example, a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SCM (Storage Class Memories), or a combination thereof.

The communication I / F 104 is an interface for connecting the information processing device 10 to the network N.

The input / output I / F 105 is an interface for connecting the input device 107 and the output device 108 to the information processing device 10.

The drive device 106 reads / writes data from the disk media 109. The disk media 109 is, for example, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or a combination thereof. The disk media 109 is, for example, a CD, a DVD, an FD, an MO, or a combination thereof.

The input device 107 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, or a combination thereof.

The output device 108 is, for example, a display, a projector, a printer, a speaker, or a combination thereof.

In the present embodiment, the control program may be written in the memory 102 or the storage 103 at the manufacturing stage of the information processing apparatus 10, may be provided to the information processing apparatus 10 via the network N, or may not be provided. It may be provided to the information processing apparatus 10 via a temporary, computer-readable recording medium.

FIG. 3 is a diagram showing an example of the hardware configuration of the terminal device 30. As shown in FIG. 3, the terminal device 30 includes a processor 301, a memory 302, a storage 303, a communication I / F 304, and an input / output I / F 305 connected to each other via a bus B3. .. Further, the terminal device 30 includes an input device 307 and an output device 308.

The processor 301 controls each configuration of the terminal device 30 and realizes the function of the terminal device 30 by expanding and executing the program stored in the storage 303 in the memory 302. The program executed by the processor 301 includes, but is not limited to, a control program such as an OS and a display control program. The processor 301 is, for example, a CPU, an MPU, a GPU, an ASIC, a DSP, or a combination thereof.

The memory 302 is, for example, a ROM, a RAM, or a combination thereof. The ROM is, for example, a PROM, an EPROM, an EEPROM, or a combination thereof. RAM is, for example, DRAM, SRAM, MRAM, or a combination thereof.

The storage 303 stores the OS, the control program, and various data. The storage 303 is, for example, a flash memory, an HDD, an SSD, an SCM, or a combination thereof.

The communication I / F 304 is an interface for connecting the terminal device 30 to the network N. The communication I / F 304 can perform wireless communication by, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or a combination thereof.

The input / output I / F 305 is an interface for connecting the input device 307 and the output device 308 to the terminal device 30.

The input device 307 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, or a combination thereof.

The output device 308 is, for example, a display, a projector, a printer, a speaker, or a combination thereof.

In the present embodiment, the control program may be written in the memory 302 or the storage 303 at the manufacturing stage of the terminal device 30, may be provided to the terminal device 30 via the network N, or may be provided to the terminal device 30 non-temporarily. May be provided to the terminal device 30 via a computer-readable recording medium.

<Functional configuration>
Next, the functional configurations of the information processing device 10 and the terminal device 30 will be described. FIG. 4 is a diagram showing an example of the functional configuration of the information processing apparatus 10. As illustrated in FIG. 4, the information processing apparatus 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

The communication unit 11 is realized by the communication I / F 104. The communication unit 11 receives the captured image from the photographing device 20. Further, the communication unit 11 transmits information such as an analysis result to the terminal device 30, and receives a request from the user from the terminal device 30. In this way, the communication unit 11 transmits / receives information to / from the photographing device 20 and the terminal device 30 via the network N.

The storage unit 12 is realized by the memory 102 and the storage 103. The storage unit 12 stores an image storage unit 120, a photographed image information 121, a customer information 122, a server program 125, a clerk discrimination model 126, a customer recognition model 127, and a clerk operation recognition model 128.

The image storage unit 120 stores a photographed image which is an entity photographed by the photographing apparatus 20. The captured image according to the present embodiment may be a still image (frame image) or a moving image. When the captured image is stored as a still image, a file format such as PNG (Portable Network Graphics), GIF (Graphics Interchange Format), or JPEG (Joint Photographic Experts Group) can be adopted. Further, when the captured image is stored as a moving image, a file format such as MPEG (Moving Picture Experts Group), AVI (Audio Video Interleave), or FLV (Flash Video) can be adopted. However, these file formats are examples, and the captured images may be saved in any format.

The captured image information 121 is information such as features related to the captured image. FIG. 5 is a diagram showing an example of captured image information 121. As illustrated in FIG. 5, the photographed image information 121 includes "photographed image ID", "attribute information", and the like as information items. Although FIG. 5 shows an example of storing the photographed image information 121 in the form of a database, the storage form of the photographed image information 121 is not limited to this.

The "captured image ID" is an identifier (ID) for identifying the captured image stored in the image storage unit 120. As illustrated in FIG. 5, the “photographed image ID” according to the present embodiment stores a file name (“file 001”, “file 002”, ...), But instead of this file name, an image storage unit. A pointer to the captured image stored in 120 may be stored.

The "attribute information" is information about an attribute (property) related to a photographed image specified by the information item "photographed image ID". As illustrated in FIG. 5, the "attribute information" according to the present embodiment includes, but is not limited to, the "shooting date and time".

The "shooting date and time" is the time when the shooting image specified by the "shooting image ID" is shot by the shooting device 20. FIG. 5 illustrates, for example, that the “shooting date and time” of the shot image in which the “shooting image ID” is “file 001” is “10:00:00 on January 15, 2020”. The format of the information stored in "shooting date and time" is not limited to this.

Further, in the present embodiment, the captured image captured by the above-mentioned imaging device 20 is captured in the information processing device 10 by the image acquisition unit 130 described later, and the captured captured image is stored in the image storage unit 120. It shall be. Therefore, the value of each information item of the captured image information 121 exemplified in FIG. 5 may be stored after the captured image is acquired from the photographing device 20 by the image acquisition unit 130. Further, the file name stored in the "photographed image ID" is given by the image acquisition unit 130, but is not limited to this. Further, as described above, the value of the "shooting date and time" is given by the shooting device 20 and read out by the image acquisition unit 130, but is not limited to this.

Next, returning to the description of the functional configuration of the storage unit 12, the customer information 122 will be described with reference to FIG. Here, we will explain by taking as an example the activity of providing free samples (samples) to customers, which is one of the sales promotion activities.

FIG. 6 is a diagram showing an example of customer information 122. The customer information 122 is information about a customer who has visited the store. As illustrated in FIG. 6, the customer information 122 has information items such as "customer ID", "image ID", "first visit date and time", "number of visits", "sample product category", and "repeater flag". And "prospect flag" etc. are included. Although FIG. 6 shows an example of storing the customer information 122 in the form of a database, the storage form of the customer information 122 is not limited to this.

The "customer ID" is an identifier for identifying a customer who has visited the store. As illustrated in FIG. 6, the “customer ID” according to the present embodiment stores a customer ID (“customer 001”, “customer 002”, ..., “Customer Q”). Note that Q is an integer of 0 or more. In the present embodiment, when a store clerk provides a free sample to a new customer, a customer ID is given to the new customer by the customer identification unit 133, which will be described later, and data is added to the customer information 122 illustrated in FIG. And.

The "image ID" stores an image ID taken by the customer specified by the information item "customer ID". FIG. 6 illustrates, for example, that the image ID taken by the customer whose "customer ID" is "customer 001" is "file 001". In the case of a new customer, the image taken by the new customer is extracted from the image taken at the time when the free sample is provided, and the image taken by the new customer is extracted from the image taken by a predetermined time in the past. It is assumed that the ID is stored in the "image ID". The "image ID" illustrated in FIG. 6 stores the latest image ID among the images taken by each customer, but when there are a plurality of shot images taken by the customer so far, All of these image IDs may be stored. In the present embodiment, the value of this "image ID" may be stored by the customer identification unit 133, which will be described later.

The "first visit date and time" is the date and time when the customer specified by the "customer ID" first visits the store. FIG. 6 illustrates, for example, that the "first visit date and time" of a customer whose "customer ID" is "customer 001" is "10:00:00 on January 15, 2020". The format of the information stored in the "first visit date and time" is not limited to this.

In the case of a new customer, for example, a photographed image in which the new customer is the subject is extracted from the photographed image taken by the photographing apparatus 20 installed at the entrance. Then, for the extracted captured image, for example, the information stored in the "shooting date and time" of the captured image information 121 exemplified in FIG. 5 may be referred to, and this information may be stored in the "first visit date and time" of the new customer. good. In the present embodiment, the value of this "first visit date and time" may be stored by the customer identification unit 133, which will be described later.

The "number of visits" is the number of times the customer specified by the "customer ID" visits the store. FIG. 6 illustrates, for example, that the “number of visits” of a customer whose “customer ID” is “customer 002” is “10 times”, but the format of the information stored in the “number of visits” is this. Not limited to. In the present embodiment, the value stored in this "number of visits" shall be updated by the customer identification unit 133, which will be described later, but the present invention is not limited to this.

The "sample product category" is a free sample product category provided by a clerk to a customer specified by a "customer ID". In other words, the "sample product category" is the product category of the free sample received by the customer specified by the "customer ID". In the present embodiment, the product category is, for example, a product type such as a cleanser or shampoo. FIG. 6 illustrates that the “sample product category” of the customer whose “customer ID” is “customer 001” is “product 001”, but the information stored in the “sample product category”. The format of is not limited to this. Further, when there are a plurality of product categories of the free sample received by the customer, the "product category of the free sample" may store these a plurality of product categories.

Further, in the present embodiment, what kind of product category the value stored in the "sample product category" specifically refers to is stored in advance in the storage unit 12 or the like. In this way, for example, when "Product 001" indicates a product category of "Facial cleanser", it can be seen that "Customer 001" has received a free sample related to "Facial cleanser". In the present embodiment, the value stored in this "product category of free sample" is updated by the clerk operation recognition unit 135 described later, but is not limited to this.

The "repeater flag" is information indicating whether or not the user is a repeater, and "1" is stored when the flag is a repeater, and "0" is stored when the flag is not a repeater. Further, when new customer information is added to the customer information 122, "0" is stored in this "repeater flag". Further, when it is determined that the customer specified by the "customer ID" has become a new repeater in the processing of the new repeater determination unit 137, the customer identification unit 133, etc., which will be described later, the "repeater flag" of the customer is set. "1" is stored. In the present embodiment, the value of this "repeater flag" is stored or updated by the new repeater determination unit 137.

Further, in FIG. 6, for example, since the "repeater flag" of the customer whose "customer ID" is "customer 001" and "customer 003" is "0", "customer 001" and "customer 003" are repeaters. It shows that it is not. Further, in FIG. 6, since the "repeater flag" of the customer whose "customer ID" is "customer 002" is "1", it is shown that "customer 002" is a repeater.

Further, for example, when the related product has not been purchased even after visiting the store three times, such as "customer 003", "0" is still stored in the "repeater plug". In the present embodiment, once "1" is stored in the "repeater plug", the value does not change thereafter, but the present invention is not limited to this. For example, if a customer whose "1" is stored in the "repeater flag" (customer who has become a repeater) does not purchase the product at the store for a certain period of time, the customer is no longer a repeater and is set in the "repeater flag". “0” may be stored.

The "prospect flag" is information indicating whether or not the customer is a prospect, and if it is a prospect, "1" is stored, and if it is not a prospect, "0" is stored. ing. Further, when new customer information is added to the customer information 122, "0" is stored in this "prospect customer flag". Further, when the customer specified by the "customer ID" is determined to be a prospective customer in the processing of the prospective customer determination unit 136 or the like described later, "1" is stored in the "prospect customer flag" of the customer. Will be done. In this embodiment, the value of this "prospect flag" is stored (updated) by the prospect determination unit 136.

Further, in the present embodiment, when the customer becomes a repeater, he / she is no longer a prospective customer. At that time, the prospect determination unit 136, which will be described later, updates the value of the “prospect flag” of the customer to “0”.

Next, the server program 125 will be described by returning to the description of the functional configuration of the storage unit 12. The server program 125 is provided in a form recorded on a recording medium such as a memory card or a disk media 109, and the information processing apparatus 10 acquires the provided server program 125 and stores it in the storage unit 12. However, this server program 125 may be written to the storage unit 12 at the manufacturing stage of the information processing apparatus 10, for example.

Further, for example, the server program 125 may be distributed by another remote server device or the like, and in this case, the information processing apparatus 10 may acquire the distributed server program 125 via the network N. Further, for example, in the server program 125, what is recorded on a recording medium such as a disk media 109 may be read by a drive device 106 or the like and written in a storage unit 12 of the information processing device 10.

Next, the clerk discrimination model 126, the customer recognition model 127, and the clerk motion recognition model 128 will be described. The information processing device 10 according to the present embodiment utilizes so-called AI (Artificial Intelligence) to perform processing such as determination (recognition) of a clerk, recognition of a customer, and recognition of an action by a clerk. The store clerk discrimination model 126, the customer recognition model 127, and the store clerk motion recognition model 128 used for the processing of the above are provided. The clerk discrimination model 126, the customer recognition model 127, and the clerk motion recognition model 128 are learned learning models that have been machine-learned in advance, and a learning model such as a neural network or SVM (Support Vector Machine) can be adopted.

Further, the store clerk discrimination model 126, the customer recognition model 127, and the store clerk motion recognition model 128 in the present embodiment are subjected to learning and relearning processing by the information processing device 10. However, the learning and re-learning processes of these learning models may be performed by another device, in which case the above-mentioned learning model may be provided, for example, together with the server program 125 via a recording medium. Further, it may be distributed by another server device or the like separately from the server program 125.

The clerk discrimination model 126 according to the present embodiment determines whether or not the clerk is photographed. The clerk discrimination model 126 will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing a configuration example of the clerk discrimination model 126. The illustrated clerk discrimination model 126 has the structure of a neural network in which a plurality of neurons are interconnected. Since such a learning model is a known technique, detailed description thereof will be omitted, but a neuron is an element that performs an operation on a plurality of inputs and outputs one value as an operation result. The neuron has information such as weighting coefficients and thresholds used in the calculation. Such a neural network learning model aggregates the input layer that accepts the input of one or more data, the intermediate layer that performs arithmetic processing on the data accepted by the input layer, and the arithmetic results of the intermediate layer. It is provided with an output layer that outputs one or more values.

The clerk discrimination model 126 accepts captured images (frame images) of P images (P is an integer of 1 or more) captured by the photographing device 20 as an input. As illustrated in FIG. 7, P images taken from the photographed image 1 to the photographed image P are input to the clerk discrimination model 126. The clerk discrimination model 126 determines the aptitude of each photographed image, that is, the certainty that the clerk is shown, with a numerical value of 0.00 to 1.00 for each input of P images. It is configured to output as, and the learning process is performed in advance. In the present embodiment, the aptitude indicates that the larger the value, the higher the aptitude.

Specifically, the clerk discrimination model 126 according to the present embodiment is, for example, a teacher data in which learning data (teacher data) in which a photographed image of a past clerk is associated with an aptitude is associated with an aptitude. Using, for example, learning is performed by the above-mentioned method in the clerk discrimination model 126. As described above, the clerk discrimination model 126 has a high degree of certainty that the photographed image having a high degree of aptitude, that is, the person shown in the photographed image is the clerk (there is a high possibility that the clerk is photographed). Learning is done so that the numerical value 1 is output for the captured image.

Further, the clerk discrimination model 126 has a low degree of certainty that the photographed image, that is, the person shown in the photographed image is a clerk (there is a high possibility that a person who is not a clerk is photographed). ) Learning is done so that the numerical value 0 is output for the captured image. The store clerk discrimination model 126 shown in FIG. 7 is an example, and information on feature points in the frame image may be used for the store clerk determination and learning thereof.

The customer recognition model 127 performs a process of identifying which customer is the customer who is the subject in each photographed image from the photographed image acquired by the photographing apparatus 20. This customer recognition model 127 will be described with reference to FIG. FIG. 8 is a schematic diagram showing a configuration example of the customer recognition model 127. Since the illustrated customer recognition model 127 has the structure of a neural network in which a plurality of neurons are interconnected like the above-mentioned clerk discrimination model 126, the description of the overlapping portion will be omitted. The customer recognition model 127 differs from the clerk discrimination model 126 in that the calculation results of the intermediate layer are aggregated and a plurality of values are output, but since such a learning model is also known, detailed description thereof will be omitted.

The customer recognition model 127 accepts Q images (Q is an integer of 1 or more) captured by the imaging device 20 as an input. The customer recognition model 127 is configured to output the results classified into Q persons from customer 001 to customer Q in response to the input of these Q images, and is preliminarily subjected to learning processing.

Further, the output of the customer recognition model 127 according to the present embodiment is numerical information of 0.00 to 1.00, respectively, and the customer ID to which the largest numerical value is output is the input classification result of the captured image, that is, , The customer ID of the specified customer. In the present embodiment, the output value and the threshold value are compared, and if the output value does not reach the threshold value, it is determined that the customer is a new customer, but the present invention is not limited to this.

The clerk motion recognition model 128 performs a process of specifying an action performed by a clerk who is a subject in each captured image, that is, a clerk action, from a captured image captured by the photographing device 20. In the present embodiment, the clerk behavior refers to the type of behavior (behavior pattern) by the clerk, such as displaying a product or providing a free sample to a customer, and is set in advance by the system administrator. However, it is not limited to this. The clerk motion recognition model 128 will be described with reference to FIG. FIG. 9 is a schematic diagram showing a configuration example of the clerk motion recognition model 128.

The illustrated clerk motion recognition model 128 has a neural network structure in which a plurality of neurons are interconnected, similar to the customer recognition model 127 described above, and outputs a plurality of values by aggregating the calculation results of the intermediate layer. do. The description of the overlap with the clerk discrimination model 126 and the customer recognition model 127 described above will be omitted.

The clerk motion recognition model 128 accepts captured images of R images (R is an integer of 1 or more) captured by the imaging device 20 as an input. It is assumed that the clerk is the subject of these R images. The clerk motion recognition model 128 is configured to output the results classified into each action type of the clerk in response to the input of these R images, and is preliminarily subjected to learning processing.

Further, the clerk behaviors 1 to L (L is an integer of 1 or more), which is the type of the clerk behavior output by the clerk movement recognition model 128 according to the present embodiment, are numerical information of 0.00 to 1.00, respectively. The type in which the largest numerical value is output is the classification result of the input captured image, that is, the behavior type of the clerk (clerk behavior 1 to L). The input of the clerk motion recognition model 128 according to the present embodiment is the captured image in which the subject of the captured image is determined to be the clerk by the clerk discrimination model 126 among all the captured images captured in the information processing apparatus 10. However, it is not limited to this.

Further, in the present embodiment, it is assumed that one shelf for storing the free sample is installed in advance in addition to the product shelf installed in the store, and the customer voluntarily uses or acquires the free sample. In order to avoid this, this product shelf shall be used only by the clerk. In the present embodiment, the shelf in which the free sample is stored is also referred to as a free sample shelf, and this free sample shelf is a shelf provided with a drawer (see FIG. 1), and each drawer has each product category described later. It is assumed that the free sample of is stored, but the present invention is not limited to this. The free sample shelf may store free samples of a plurality of types of products in one drawer, or may store free samples of one product category in a plurality of drawers. Further, the product shelves may not be in the form of drawers, and one or more of the product shelves installed in the store may be allocated as free sample shelves. Further, a plurality of free sample shelves may be installed.

In the present embodiment, a photographed image in which a store clerk provides a free sample to a customer is extracted. The action of the clerk providing the free sample to the customer is, for example, the action of the clerk taking the customer to the free sample shelf, the action of the clerk handing the free sample to the customer, or the action of the clerk handing the free sample to the customer. This includes the movement of a clerk walking toward the free sample shelf with the customer behind him. In the clerk motion recognition model 128 according to the present embodiment, "clerk action 1" is defined as an action of providing a free sample, and a photographed image showing the above-mentioned action corresponds to "clerk action 1". Learning is done like this.

Next, returning to the description of the functional configuration of the information processing apparatus 10, the control unit 13 will be described. The control unit 13 processes the data received from the communication unit 11 and gives the processing result to the communication unit 11. The control unit 13 includes an image acquisition unit 130, a store clerk determination unit 131, a customer identification unit 133, a store clerk motion recognition unit 135, a prospective customer determination unit 136, and a new repeater determination unit 137. By reading and executing the server program 125 stored in the storage unit 12, the control unit 13 according to the present embodiment reads and executes the above-mentioned image acquisition unit 130, store clerk discrimination unit 131, customer identification unit 133, and store clerk operation recognition unit 135. , The prospect determination unit 136, the new repeater determination unit 137, and the like are realized in the control unit 13 as software-like functional units.

The image acquisition unit 130 performs a process of acquiring an image captured by the imaging device 20 from the imaging device 20. Then, the image acquisition unit 130 may perform a process of storing the acquired captured image in the image storage unit 120 in the storage unit 12.

The clerk discrimination unit 131 takes a photographed image acquired by the image acquisition unit 130 in the store (for example, a place other than the entrance and the clearing counter) as an input of the clerk discrimination model 126, and based on the output, the input photographed image is used. Determine if the clerk is in the picture. Specifically, when the clerk discrimination unit 131 inputs the photographed image A to the clerk discrimination model 126 and the aptitude output by the clerk discrimination model 126 exceeds the threshold value (for example, 0.7), the photographed image A It is judged that the clerk is in the picture. Further, for example, when the aptitude of the captured image B exceeds 0.7, which is the first threshold value, and does not exceed 0.9, which is the second threshold value, the clerk determination unit 131 shows the clerk in the captured image B. You may judge that it is not.

The customer identification unit 133 uses the captured image acquired by the image acquisition unit 130 as the input of the customer recognition model 127, and identifies which customer the subject of the input captured image is based on the output of the customer recognition model 127. do. Specifically, when the captured image C is input to the customer recognition model 127, the customer identification unit 133 narrows down the customer having the customer ID, which is the largest output, as the customer who is the subject. As described above, in the present embodiment, the customer identification unit 133 compares the output value with the threshold value, and if the output value does not reach the threshold value, determines that the customer is a new customer. Not limited. If it is determined that the customer is a new customer, the customer identification unit 133 may assign a customer ID to the customer and add data including the customer ID to the customer information 122.

In addition, the customer identification unit 133 can identify the customer who has visited the store by using the photographed image taken by the photographing apparatus 20 installed at the entrance of the store. Then, the customer identification unit 133 may increment the value of the "number of visits" of the customer by one on the customer information 122 illustrated in FIG. 6 for the specified customer.

Further, the customer identification unit 133 may identify the customer who purchased the product by using the photographed image (photographed image at the time of clearing) taken by the photographing device 20 installed at the clearing counter of the store.

In addition, the customer identification unit 133 uses a photographed image taken by the photographing device 20 installed on the product shelf in the store, for example, at a place other than the entrance and the clearing counter, to obtain a customer who has received a free sample from the store clerk. You may specify. Further, the customer identification unit 133 may store the image ID of the photographed image in which the scene in which the customer receives the free sample is photographed in the "image ID" in the customer information 122 exemplified in FIG. Further, in the case of a new customer, the customer identification unit 133 refers to, for example, the photographed image information 121 exemplified in FIG. 5 for the photographed image provided with this "image ID", and sets the value stored in the "photographing date and time". It may be stored in the "first visit date and time" in the customer information 122 exemplified in FIG.

The clerk motion recognition unit 135 uses the captured image acquired by the image acquisition unit 130 as the input of the clerk motion recognition model 128, and based on the output of the clerk motion recognition model, the subject of the input captured image is any clerk action. Is determined. Specifically, when the photographed image D is input to the clerk motion recognition model 128, the clerk motion recognition unit 135 determines that the type of clerk action that produces the largest output is the corresponding clerk action. In this way, in the present embodiment, a photographed image showing "clerk action 1" (action of providing a free sample) is extracted. As described above, in the present embodiment, the input of the clerk motion recognition model 128 is that the subject of the captured image is the subject of the captured image by the clerk discrimination unit 131 among all the captured images captured in the information processing apparatus 10 by the image acquisition unit 130. The photographed image is determined to be a clerk, but the image is not limited to this.

In addition, the clerk motion recognition unit 135 recognizes an image based on the captured image taken during a predetermined period from the time when the captured image narrowed down as the captured image related to "clerk action 1" (action of providing a free sample) is captured. Using technology, etc., identify from which drawer of the free sample shelf the free sample was taken out and provided to the customer. Since such an image recognition technique is known, the description thereof will be omitted. In this way, it is possible to identify what kind of product category the clerk provided to the customer.

In this embodiment, the product category of the free sample is determined by specifying from which drawer of the free sample shelf the free sample is taken out and provided to the customer, but the present invention is limited to this. do not have. For example, by letting the clerk motion recognition model 128 learn the action of taking out a free sample from the top drawer, for example, it is possible to extract a photographed image in which the action of taking out a free sample from the top drawer is taken. You may do so.

Further, for example, when the clerk motion recognition unit 135 hides the clerk in the photographed image by the customer behind and it is difficult to identify the withdrawal, the photographed image in which the sample is provided to the customer or the customer You may specify the free sample to be provided and the product category thereof by using a known image recognition technique or the like from the photographed image taken while holding the provided free sample. Then, in the present embodiment, the store clerk motion recognition unit 135 stores the value of the specified product category in the "sample product category" of the customer information 122 exemplified in FIG. In addition, if it is recognized that the customer did not receive the free sample after the free sample was provided by the clerk, the free sample was not given or received, and the free sample was not provided to the customer (sample). It may be (not received).

When the prospective customer determination unit 136 determines that the free sample has been exchanged by the above-mentioned clerk motion recognition unit 135, the prospective customer determination unit 136 determines that the customer who received the free sample is a prospective customer (candidate for a new repeater). As described above, the new repeater in the present embodiment is not a repeater before the purchase, but a customer who has become a new repeater in this purchase. Therefore, the prospect determination unit 136 refers to the customer information 122 illustrated in FIG. 6, and if "1" is stored in the value of the "repeater flag" for the customer who received the free sample, the prospect is expected. Do not judge that it is a customer. However, the prospect determination unit 136 refers to the customer information 122 illustrated in FIG. 6, and if "0" is stored in the value of the "repeater flag" for the customer who received the free sample, the free sample is provided. With the provision of the product, the customer is determined to be a prospective customer.

Further, the prospect determination unit 136 stores "1" in the "prospect flag" of the customer information 122 exemplified in FIG. 6 for the customer determined to be a prospect. Further, when the customer becomes a repeater, the prospective customer is no longer a prospective customer, so the prospective customer determination unit 136 updates the value of the "prospect customer flag" of the customer to "0".

In the new repeater determination unit 137, when a customer who is determined to be a prospective customer by the above-mentioned prospective customer determination unit 136 subsequently visits the store and purchases a product related to the provided free sample at the time of the visit, the customer concerned Determined to be a new repeater.

Specifically, as described above, the customer identification unit 133 uses the customer recognition model 127 to capture a photographed image taken by the photographing apparatus 20 installed at the entrance of the store, and the customer (customer ID) who visits the store. Can be identified. Further, as described above, in the present embodiment, the customer ID related to the new customer is assigned by the customer identification unit 133 and the data is added to the customer information 122 because the store clerk provided the free sample to the new customer. It's timing. Therefore, for customers (prospects) who have already been provided with free samples and have visited the store after that, there is customer data in the customer information 122, and the "repeater flag" of this customer data is "0".

Therefore, when the customer is identified by the customer identification unit 133 from the image taken by the image pickup device 20 installed at the entrance, the new repeater determination unit 137 refers to the customer information 122 (see FIG. 6), and the new repeater determination unit 137 refers to the customer information 122 (see FIG. 6). Acquires information on whether or not "0" is stored in the "repeater flag".

Here, if "0" is stored in the "repeater flag", that is, if it is a prospective customer, a new repeater depends on whether or not this customer purchases a product related to the provided free sample. It is decided whether or not it becomes. As described above, the customer identification unit 133 can identify the customer who purchased the product by using the photographed image taken by the photographing apparatus 20 installed at the clearing counter of the store. Therefore, the new repeater determination unit 137 determines whether or not the product category of the purchased product is the same as the product category of the free sample when the customer identification unit 133 finds that the prospective customer has purchased the product. Here, the new repeater determination unit 137 can determine the product category of the purchased product from the photographed image and the purchase data when the prospective customer purchases.

In this way, when the prospective customer determines that the related product of the free sample has been purchased, the new repeater determination unit 137 determines that the prospective customer has become a new repeater. That is, the new repeater determination unit 137 determines that a new repeater has been acquired. On the other hand, if the new repeater determination unit 137 determines that the prospective customer did not purchase the product related to the free sample, or the purchased product was not a product related to the free sample, the prospective customer becomes a new repeater. Judge that it did not happen. That is, the new repeater determination unit 137 determines that the new repeater has not been acquired. As described above, since the process by the new repeater determination unit 137 also corresponds to the process of analyzing the relationship between the sales promotion activity such as providing free samples and the acquired new repeater, the new repeater determination unit 137 serves as an analysis unit. It can be said that it has the function of.

Further, the new repeater determination unit 137 may determine a customer who has become a new repeater, and at the same time, display information on the new repeater on, for example, the terminal device 30. Therefore, since the new repeater determination unit 137 also has a function as a display control function, it can be said that the new repeater determination unit 137 has a function as a display control unit. This display function will be described later with reference to FIG.

Next, the functional configuration of the terminal device 30 will be described. FIG. 10 is a diagram showing an example of the functional configuration of the terminal device 30. As illustrated in FIG. 10, the terminal device 30 includes a communication unit 31, a storage unit 32, a control unit 33, an operation unit 34, and a display unit 35.

The communication unit 31 is realized by the communication I / F 304. The communication unit 31 transmits / receives data to / from the information processing device 10 and the photographing device 20 via the network N.

The storage unit 32 is realized by the memory 302 and the storage 303. The storage unit 32 stores various information including the analysis result.

The control unit 33 is realized by the processor 301 executing a program and cooperating with other hardware configurations. The control unit 33 controls the entire operation of the terminal device 30.

The operation unit 34 receives an input from a user who uses the terminal device 30, and supplies operation information indicating the accepted input to the control unit 33.

The display unit 35 displays the screen according to the control by the control unit 33 described above.

Although not shown, the terminal device 30 may have other functions such as a photographing function.

<First analysis process>
FIG. 11 is a flowchart (steps S1 to S7) showing the first analysis process performed by the control unit 13 of the information processing apparatus 10 according to the present embodiment. The first analysis process is one of the processes to analyze the relationship between sales promotion activities and acquisition of new repeaters. Specifically, from the time a customer is first offered a free sample until the customer is determined to be a prospective customer. It is the processing of. The first analysis process will be described with reference to FIG.

In step S1, when the image acquisition unit 130 acquires a photographed image photographed by the photographing device 20 installed at the entrance from the photographing device 20, the customer identification unit 133 obtains the acquired photographed image as a customer recognition model. The input is 127. The customer identification unit 133 identifies the customer (customer ID) that is the subject of the input captured image based on the output of the customer recognition model 127. Here, the customer identification unit 133 recognizes that the customer having the customer ID for which the largest numerical value is output is the customer who is the subject of the input photographed image, that is, the customer who has visited the store.

Subsequently, the prospect determination unit 136 refers to the value of the “prospect flag” in the customer information 122 for the customer who has visited the store, and determines whether or not the customer has visited the store (step S1). In the present embodiment, if the customer is determined to be a prospective customer at the time of visiting the store in the past, "1" is stored in the "prospectable customer flag" of the customer information 122 for the customer. Therefore, when "1" is stored in the "prospect flag", the prospect determination unit 136 determines that the prospect has visited the store (see the YES route in step S1). Then, the process proceeds to step T1 in FIG. 12, which will be described later, via the connector A. On the other hand, since a customer who has not been provided with a free sample at the time of visiting the store in the past does not correspond to a prospective customer, "0" is stored in the "prospect customer flag" of the customer information 122. When "0" is stored in the "prospect flag", the prospect determination unit 136 determines that a customer who is not a prospect has visited the store (see NO route in step S1).

When it is determined in step S1 that a customer who is not a prospective customer has visited the store, the image acquisition unit 130 acquires a photographed image taken in the store from the photographing device 20, and the clerk determination unit 131 acquires the acquired photographed image. Judge whether or not the clerk is in the picture. Specifically, the clerk discrimination unit 131 uses the photographed image taken in the store as an input of the clerk discrimination model 126, and determines whether or not the clerk is reflected in the photographed image based on the output. At that time, the clerk discrimination unit 131 determines (identifies) that the clerk is included in the captured image when the output exceeds the threshold value. In this way, in step 2, the photographed image taken in the store is used to narrow down (identify) the photograph determined to include the clerk in the photographed image. Then, the process proceeds to step S3.

In step S3, the clerk motion recognition unit 135 uses the captured image of the clerk, which is narrowed down in step S2, as the input of the clerk motion recognition model 128, and takes an input image based on the output of the clerk motion recognition model 128. It is determined which clerk's behavior is the clerk's behavior shown in the image. Specifically, the clerk motion recognition unit 135 recognizes that the type of clerk action for which the largest numerical value is output is the clerk action shown in the input captured image. Then, the clerk motion recognition unit 135 extracts a photographed image showing the “clerk action 1” (the action of providing a free sample), and the process proceeds to step S4 (see the YES route in step S3). On the other hand, if the photographed image relating to the provision of the free sample is not extracted, the process of step S3 is repeated (see the NO route in step S3).

In step S4, the store clerk motion recognition unit 135 uses image recognition technology or the like based on the captured images taken during a predetermined period from the shooting date and time of the captured image extracted in step S3, and the store clerk uses a free sample shelf. Identify from which drawer the sample was taken and provided to the customer. The captured image taken during a predetermined period from this shooting date and time is a captured image taken during a predetermined period before and after the time (date and time) when the extracted captured image was taken, and this predetermined period is arbitrarily set. Can be done. By the process of this step S4, the clerk motion recognition unit 135 specifies which product category the free sample is provided to the customer by the clerk (step S4). Then, the process proceeds to step S5.

Next, a case where the customer is shown together with the clerk in the photographed image extracted in step S3 will be described as an example. The processing when only the clerk is shown will be described later. The customer identification unit 133 uses the captured image extracted in step S3 as the input of the customer recognition model 127, and based on the output of the customer recognition model 127, the customer (customer ID) shown in the input captured image (with the clerk). ) Is specified (step S5). Here, the customer identification unit 133 recognizes that the customer having the customer ID for which the largest numerical value is output is the customer shown in the input captured image, that is, the customer who received the free sample. As described above, in the present embodiment, the customer identification unit 133 compares the output value with the threshold value, and if the output value does not reach the threshold value, determines that the customer is a new customer. Then, when it is determined that the customer is a new customer, the customer identification unit 133 assigns the customer ID to the customer, adds the data including the customer ID to the customer information 122, and sets the "repeater flag" of the customer. Store "0".

Further, in step S5, when the customer is not shown in the photographed image extracted in step S3, the photographed image taken before and after the photograph or the photographed image extracted in step S3 is photographed. Customers may also be identified using captured images taken at approximately the same time by the photographing device 20 in the vicinity of the device 20.

Subsequently, the prospect determination unit 136 determines whether or not the customer (customer who received the free sample) specified in step S5 is already a repeater with reference to the customer information 122 exemplified in FIG. (Step S6). Specifically, the prospect determination unit 136 refers to the customer information 122, and if "0" is stored in the value of the "repeater flag" for the customer specified in step S5, the prospect determination unit 136 is still a repeater. Since it has not been done, the process proceeds to step S7 (see the NO route in step S6), and it is determined that the customer is a prospective customer (step S7). Further, the prospect determination unit 136 stores "1" in the "prospect flag" of the customer information 122 exemplified in FIG. 6 for the customer determined to be a prospect. Then, the process ends.

On the other hand, in step S6, when "1" is stored in the value of the "repeater flag" for the customer specified in step S5, the prospective customer determination unit 136 determines that the customer is already a repeater. (Refer to YES route in step S6). In this case, it is not determined to be a prospective customer, and the process ends.

<Second analysis process>
FIG. 12 is a flowchart (steps T1 to T2) showing a second analysis process performed by the control unit 13 of the information processing apparatus 10 according to the present embodiment. The second analysis process is one of the processes to analyze the relationship between sales promotion activities and acquisition of new repeaters. Specifically, a prospective customer visits the store after providing a free sample, and this prospective customer is used as a new repeater. It is a process until it is acquired. The second analysis process will be described with reference to FIG.

As described above, the process of step T1 in FIG. 12 is executed after it is determined in the process of step S1 of FIG. 11 that a prospective customer has visited the store (see the YES route in step S1).

After it is determined in step S1 that the prospective customer has visited the store, in step T1, the new repeater determination unit 137 uses, for example, a photographed image captured by the photographing apparatus 20 to perform the behavior of the prospective customer in the store. Track and determine if you purchased (some) goods at the clearing counter. Here, when it is difficult to track the prospective customer due to the influence of obstacles or the like, the customer identification unit 133 takes a photographed image (photographed image at the time of clearing) taken by the photographing device 20 installed at the clearing counter of the store. May be used to determine if the prospect is in the picture of the clearing counter, i.e., if the prospect has purchased.

Then, when it is found that the prospective customer has purchased (some) product, the new repeater determination unit 137 determines whether or not the product category of the purchased product is the same as the product category of the free sample. That is, the new repeater determination unit 137 determines whether or not the prospective customer has purchased a free sample related product. Specifically, the new repeater determination unit 137 determines whether or not the related product of the free sample has been purchased from the photographed image and the purchase data at the time of the settlement of the prospective customer. If it is determined that the prospect has purchased a free sample related product, the process proceeds to step T2 (see the YES route in step T1). On the other hand, if it is determined that the prospective customer has not purchased the related product of the free sample, the process is terminated (see NO route in step T1).

If it is determined in step T1 described above that the prospective customer has purchased a free sample related product, in step T2, the new repeater determination unit 137 determines that the prospective customer has become a new repeater. That is, the new repeater determination unit 137 determines (determines) that the store has acquired a new repeater. Then, the process is terminated.

In this way, when the store acquires new repeaters, it is considered that the sales promotion activity of providing free samples was effective for this prospective customer. Therefore, it can be said that the processing by the new repeater determination unit 137 analyzes the relationship between the sales promotion activity and the acquired new repeater.

The information processing apparatus 10 may transmit to the terminal apparatus 30 the information regarding the newly acquired repeater analyzed through the processes of the above steps S1 to S7 (see FIG. 11) and T1 to T2 (see FIG. 12). can. The terminal device 30 may display information about the received new repeater on the display by the display unit 35. An example of this display will be described with reference to FIG. FIG. 13 is a diagram showing an example of an effect display screen.

The new repeater determination unit 137 of the information processing apparatus 10 determines a customer who has become a new repeater, and at the same time, displays, for example, the number of new repeaters (total) in each month, that is, the number of new repeaters by month. It may be good (see FIG. 13). Further, as illustrated in FIG. 13, the increase / decrease number of new repeaters may also be displayed. The effect display screen illustrated in FIG. 13 is only an example, and for example, the number of new repeaters acquired for each product category of a free sample may be displayed.

As described above, according to the present embodiment, it is possible to visualize the number of customers who have become new repeaters by providing free samples performed by the store clerk as a sales promotion activity. This makes it possible to easily understand the relationship between sales promotion activities such as the provision of free samples and the acquisition of new repeaters.

<Embodiment 2>
In the above-described embodiment (also referred to as the first embodiment), the relationship between the provision of the free sample and the acquisition of new repeaters is grasped based on whether or not the customer has purchased the product related to the provided free sample. However, in the second embodiment, information such as the position picked up by the customer, the position where the customer has stayed for a long time, and information on the product category of the purchased product are also provided as a free sample. The purpose is to understand the relationship with the acquisition of new repeaters.

The configuration of the information processing system 1 according to the second embodiment, the hardware configuration of the information processing device 10, the hardware configuration of the terminal device 30, the clerk discrimination model 126, the customer recognition model 127, the clerk operation recognition model 128, and the terminal device 30. The functional configuration, the flowchart showing an example of the first analysis process, and the effect display screen are the configuration of the information processing system 1 according to the above-described first embodiment (see FIG. 1) and the hardware configuration of the information processing device 10 (see FIG. 1), respectively. (See FIG. 2), hardware configuration of the terminal device 30 (see FIG. 3), clerk discrimination model 126 (see FIG. 7), customer recognition model 127 (see FIG. 8), clerk motion recognition model 128 (see FIG. 9), terminal. Since it is equivalent to the functional configuration of the apparatus 30 (see FIG. 10), the flowchart showing an example of the first analysis process (see FIG. 11), and the effect display screen (see FIG. 13), the description thereof will be omitted.

<Functional configuration>
The functional configuration of the information processing apparatus 10 according to the second embodiment will be described with reference to FIG. FIG. 14 is a diagram showing an example of the functional configuration of the information processing apparatus 10 according to the second embodiment. The functional configuration of the information processing apparatus 10 according to the second embodiment includes the position specifying unit 138 in the control unit 13 with respect to the functional configuration of the information processing apparatus 10 according to the first embodiment (see FIG. 4). It differs from the first embodiment in that it is different from the first embodiment. The control unit 13 according to the second embodiment reads and executes the server program 125 stored in the storage unit 12, so that the position specifying unit 138 and the like are realized in the control unit 13 as a software-like functional unit. .. Since the other functional configurations are the same as those of the information processing apparatus 10 according to the first embodiment (see FIG. 4), the description thereof will be omitted.

The position specifying unit 138 uses a position specifying algorithm described later to specify the coordinates where the customer is located, the product shelves in the vicinity where the customer is located, and the like.

Next, the photographed image information 121 according to the second embodiment will be described with reference to FIG. FIG. 15 is a diagram showing an example of the captured image information 121 according to the second embodiment. As illustrated in FIG. 15, the captured image information 121 according to the second embodiment includes "photographed position information" in the information item "attribute information" with respect to the captured image information 121 according to the first embodiment. This is different from the first embodiment. Since the other configurations of the captured image information 121 according to the second embodiment are the same as the configuration of the captured image information 121 according to the first embodiment (see FIG. 5), the description thereof will be omitted.

The "shooting position information" is the coordinates of the captured image specified by the "photographed image ID" in the map in the store (in-store map). “Shooting position information” will be described with reference to FIG. FIG. 16 is a schematic diagram for explaining the shooting position coordinates. The left figure of FIG. 16 is an example of an image captured by one imaging device 20A. The figure on the right is an example of a map inside the store, and is a schematic representation of the inside of the store as seen from above.

In the second embodiment, it is assumed that a predetermined rectangle on the captured image corresponds to a predetermined rectangle on the map in the store, and each of these vertices is associated with each other by using, for example, an 8-point algorithm. Since this 8-point algorithm is publicly known, a specific description thereof will be omitted. Further, the data of this in-store map may be stored as digital data in the storage unit 12 or the like of the information processing apparatus 10, and in that case, the coordinates in this in-store map are displayed on the in-store map converted into digital data. It may be the coordinates of.

The photographed images of FIG. 16 and A1 to A4 in the store map indicate the product shelves installed in the store, and the product shelves to which the same reference numerals are given indicate that they are the same shelves. That is, the product shelves A1 to A4 in the photographed image (left figure) exemplified in FIG. 16 are located at the locations (coordinates) of the product shelves A1 to A4 in the right figure on the in-store map, respectively. In FIG. 16, only A1 to A4 are shown on the in-store map, but the symbols of the product shelves provided other than the above are omitted. Further, in the second embodiment, the product shelves A1 to A4 are assumed to display products related to a specific product category (for example, detergent, face wash), but the present invention is not limited to this. Further, products belonging to the same product category may be displayed on two or more product shelves among a plurality of product shelves, or products belonging to a plurality of product categories may be displayed on one product shelf.

Further, as illustrated in FIG. 16, when the entire captured image (left figure) corresponds to the rectangle shown by the dotted line in the store map (right figure), the dots K1 to K4 on the captured image are on the store map. It corresponds to each X coordinate and Y coordinate of points M1 to M4. In the second embodiment, the "shooting position information" in the captured image information 121 exemplified in FIG. 15 exists at diagonal positions of a rectangle on the in-store map illustrated in FIG. 16, for example, two vertices M1 and 1. It is assumed that the coordinates of M4, that is, (M1_X, M1_Y)-(M4_X, M4_Y) are stored. The coordinates stored in the "shooting position information" are not limited to the coordinates of the vertices M1 and M4, and the coordinates of any two points of the vertices M1 to M4 may be selected, and the coordinates of the vertices M1 to M4 may be selected. The coordinates of one vertex, or the coordinates of three or four vertices among the vertices M1 to M4 may be stored. Further, as described above, the method used for associating coordinates is not limited to the 8-point algorithm. The method for specifying such coordinates is also called a position specifying algorithm.

Further, in the second embodiment, the coordinates of each product shelf on the in-store map and the information regarding the product categories displayed on each product shelf are managed in advance and stored in the storage unit 12, for example. And. In this case, the above information shall be changed each time the product shelves are moved or increased or decreased, or the display of products (categories) is changed.

In this way, by specifying the position (coordinates) of the store on the in-store map together with, for example, the orientation of the customer's face, the product picked up by the customer, and the length of stay, the customer may be interested. , You can associate the item you are considering purchasing. A known image recognition technique or the like can be used to specify the direction in which the customer is facing. Since such a technique is known, the description thereof will be omitted. Further, in order to specify the product picked up by the customer, a known image recognition technique or the like may be used, or for example, the customer behavior may be specified by using a learning model capable of recognizing the customer behavior. ..

Next, the customer information 122 according to the second embodiment will be described with reference to FIG. FIG. 17 is a diagram showing an example of customer information 122 according to the second embodiment. As illustrated in FIG. 17, the customer information 122 according to the second embodiment has "purchased product category" and "customer location information" in the information items with respect to the customer information 122 according to the first embodiment. It differs from the first embodiment in that it is included. Since the other configurations of the customer information 122 according to the second embodiment are the same as the configurations of the customer information 122 according to the first embodiment (see FIG. 6), the description thereof will be omitted.

The information item "purchased product category" stores the product category of the product that the customer specified by the "customer ID" has purchased at the store. In FIG. 17, for example, the “purchased product category” of a total of two customers, that is, a customer whose “customer ID” is “customer 002” and a customer whose “customer ID” is “customer 003”, is “purchased product category”. Illustrate that it is a product 002. For example, when this "product 002" indicates a product category of "toner", it can be seen that "customer 002" and "customer 003" have purchased the product of "toner" in the past. .. If there are a plurality of products purchased by the customer in the past and these product categories span a plurality of products, all of these product categories may be stored.

As described above, the customer identification unit 133 can identify the customer who purchased the product by using the photographed image (photographed image at the time of clearing) taken by the photographing device 20 installed at the clearing counter of the store. Further, as described above, the new repeater determination unit 137 can determine the product category of the purchased product from the photographed image and the purchase data when the prospective customer purchases. Therefore, in this "purchased product category", the product category determined by the new repeater determination unit 137 is stored.

Further, referring to the value of the "repeater flag" of the customer information 122, it can be seen that "customer 002" is already a repeater. This is clear from the fact that the value of the "repeater flag" of "customer 002" is "1". On the other hand, "Customer 003" is not a repeater yet (the "repeater flag" is "0") even though it has a purchase history (clearly from the fact that the value is stored in the "purchased product category"). It is clear from). Also in the second embodiment, the repeater is a customer who has purchased a product (related product) related to the provided free sample. Therefore, the "customer 003" is a repeater because he / she purchases a product related to "product 002" instead of "product 001" even though he / she is provided with a free sample related to "product 001". You can see that it is not. In the second embodiment, by subjecting the information on such purchased product categories to the effect analysis, it is possible to analyze the cause of not reaching the repeater.

The information item "customer location information" stores the position (coordinates) where the customer specified by the "customer ID" picks up the product. This is based on the possibility of picking up a product that the customer is interested in. Further, in the information item "customer position information", a position (coordinates) in which the customer specified by the "customer ID" has stayed for a long time may be stored. This is because if a customer may be weighing products for purchase, they are likely to stay in front of those products for a predetermined amount of time or longer. In FIG. 17, for example, the coordinates of the position (place) where the customer whose “customer ID” is “customer 002” looks at the product for a predetermined time or longer and the staying time is long are (200, 100). Illustrate that there is. Further, when there are a plurality of places where the person has stayed for a long time, the coordinates of the plurality of places may be stored in the "customer location information". The customer's position (coordinates) information stored in this "customer position information" can be specified by the position specifying unit 138 by using the above-mentioned position specifying algorithm.

As described above, in the second embodiment, the relationship between the provision of free samples and the acquisition of new repeaters is grasped by paying attention to the “purchased product category” and the “customer location information”.

<First and second analysis processing>
The first analysis process performed by the control unit 13 of the information processing apparatus 10 according to the second embodiment is the same as the first analysis process (steps S1 to S7 exemplified in FIG. 11) according to the first embodiment. Therefore, the description is omitted here. Here, the second analysis process performed by the control unit 13 of the information processing apparatus 10 according to the second embodiment will be described with reference to FIG. FIG. 18 is a flowchart (steps T1 to T3) showing a second analysis process performed by the control unit 13 of the information processing apparatus 10 according to the second embodiment. As described above in the first embodiment, the second analysis process here is also one of the processes for analyzing the relationship between the sales promotion activity and the acquisition of new repeaters, and specifically, the prospective customer. Is the process of coming to the store after providing a free sample and acquiring this prospective customer as a new repeater.

The second analytical process exemplified in FIG. 18 is the first embodiment in that the process of step T3 is added to the NO route of step T1 with respect to the process according to the first embodiment (see FIG. 12). Is different. The other processes (steps T1 and T2) are the same as the processes according to the first embodiment, and thus the description thereof will be omitted. Further, as described above in the first embodiment, in step S1 of FIG. 12, after it is determined by the prospective customer determination unit 136 that the prospective customer has visited the store (see the YES route in step S1), the connector A is inserted. Then, step T1 in FIG. 18 is started.

If it is determined in step T1 of FIG. 18 that the prospective customer who has visited the store has purchased the related product, the process of step T3 is started (see the NO route in step T1).

In step T3, if it is determined in the process of step T1 that the prospect has not purchased the related product of the free sample, the information items "purchased product category" and "customer position" of the customer information 122 are determined for the prospect. Obtain the values of "information" and perform analysis using these values. As described above, the NO route in step T1 indicates a case where the prospective customer visits the store after providing the free sample and does not purchase the related product of the free sample. In such a case, the new repeater determination unit 137 refers to the customer information 122 illustrated in FIG. 17, acquires the value of the “purchased product category” of the prospective customer, and analyzes the cause of the failure (purchase). Step T3).

Taking "Customer 003" in FIG. 17 as an example, from the customer information 122 exemplified in FIG. 17, the product category of the free sample provided with this "Customer 003" is "Product 001" (for example, face wash). It turns out that there is. However, for example, the product category of the product that the "customer 003" visited and purchased after being determined to be a prospective customer is "product 002" (for example, lotion), and the product category of the provided free sample. different. And since this "customer 003" has not purchased the product related to "product 001" (product category of the free sample), it has not become a repeater yet. From this, it can be analyzed that the product category in which "customer 003" is interested is "product 002" (for example, lotion) rather than "product 001" (for example, face wash).

As described above, in the second embodiment, for example, the analysis result can be displayed on the terminal device 30 by using the information of the purchased product category described above (not shown). Further, when a customer with a purchase history visits the store, for example, it is possible to notify the terminal device 30 owned by the clerk of the product category already purchased by the customer. By doing so, it is possible to encourage the clerk to provide free samples of the already purchased product category to this prospect, which can be expected to lead to further purchases by customers and acquisition of new repeaters.

Taking "customer 003" in FIG. 17 as an example, it can be seen from the customer information 122 illustrated in FIG. 17 that the "customer location information" of this "customer 003" is the coordinates (200, 100). The position specifying unit 138 can specify which product shelf the coordinates (200, 100) are around. For example, when the position identification unit 138 identifies that the coordinates (200, 100) are around the shampoo product shelf, the prospective customer is in front of the shampoo product shelf (for example, the product category is “product 003”). It can be seen that the product was located for a predetermined time or longer. As described above, if a customer is located in a certain place for a predetermined time or longer, it is possible that the customer is comparing products for purchasing the product or the customer is interested in the product. .. For this reason, the product category of the product that the prospective customer is considering purchasing or is interested in is "Product 003" rather than the product category "Product 001" (for example, shampoo) of the provided free sample. It can be analyzed as (for example, shampoo). Then, the process is terminated.

As described above, although not shown in the second embodiment, for example, it is possible to display the analysis result using the above-mentioned "customer position information" on the terminal device 30. Further, when a prospective customer visits the store, for example, on the terminal device 30 owned by the clerk, it is possible to notify the customer location information and the product category that the customer may be interested in. .. As a result, this system can encourage the clerk to provide the prospective customer with a free sample of the product category that the prospective customer has been considering for a predetermined time or longer, and it can be expected that this will lead to further purchase by the customer. .. In addition, the clerk can rush to a position where the prospective customer is likely to consider purchasing the product, explain to the prospective customer, or call out a free sample. .. As a result, it can be expected to increase sales promotion opportunities and lead to the acquisition of new repeaters.

As described above, according to the second embodiment, by providing a free sample as a sales promotion activity, the number of customers who have become new repeaters can be grasped, and the customer who is provided with the free sample ( It is possible to visualize the analysis result (cause, etc.) when the prospective customer) does not make a purchase. This makes it possible to easily grasp the relationship between sales promotion activities such as providing free samples and the acquisition of new repeaters in more detail.

In the first embodiment and the second embodiment, the clerk discrimination model 126 is used to determine whether or not the person is a clerk, but the clothes authentication technology may be used instead of the clerk discrimination model 126, or the clerk may be used. The discrimination model 126 and the clothing authentication technique may be used in combination. Since this clothing authentication technique is a known technique, the description thereof will be omitted. When using the clothes authentication technology, it is desirable that the clerk wears clothes that can distinguish the customer from the clerk (the clerk can be identified).

Further, the image acquisition unit 130 according to the first embodiment and the second embodiment may perform personal authentication processing or the like on each frame image included in the acquired captured image. Since this personal authentication process is an existing technique, detailed description thereof will be omitted, but for example, a person (individual, subject) shown in a frame image may be specified by performing a face authentication process. Further, the image acquisition unit 130 may store the frame image subjected to the personal authentication process and the information about the subject in the storage unit 12 described later. When the image acquisition unit 130 performs the personal authentication process in this way, the store clerk discrimination model 126 and the customer recognition model 127 may be omitted.

Further, in the first and second embodiments, the case where one customer receives a free sample from one clerk is taken as an example, but a plurality of customers receive a free sample from one clerk. There may be. In that case, the prospect determination unit 136 may identify these plurality of customers.

Further, in the first embodiment and the second embodiment, the repeater is a customer who has become a new repeater, but the repeater is not limited to this. For example, a customer who has not purchased for a while after becoming a repeater in the past is treated as no longer a repeater, and when the customer visits the store and makes a purchase, it is determined that the customer has become a new repeater. May be good.

In addition, the new repeater determination unit 137 according to the first embodiment and the second embodiment displays the products related to the free samples provided in the past when the prospective customer visits the store from the next time after the free sample is given and received. In the case of purchase, it is determined that the prospective customer is a new repeater, but this is not limited to this. For example, it is not limited to the purchase of related products, but if any product is purchased after the free sample is provided, it may be determined to be a repeater.

Further, in the first embodiment and the second embodiment, for the purpose of grasping the effect of the sales promotion activity, the explanation was given by taking the provision of a free sample as an example in the sales promotion activity. Therefore, the photographed image in which the free sample was exchanged was targeted for processing. All the captured images in which the behavior is captured may be processed.

The embodiments disclosed this time are exemplary and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

Further, the present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the present invention can be obtained by appropriately combining the technical means disclosed in the different embodiments. Embodiments are also included in the technical scope of the present invention.

<Additional Notes>
The present embodiment includes the following disclosures.

(Appendix 1)
It is a control method executed by an information processing device.
Processing to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity using the captured image taken by the photographing device.
Processing to display the result of the analysis as the effect of the sales promotion activity,
Including
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Control method.

(Appendix 2)
The information processing device identifies a photographed image in which a clerk is shown from the photographed image, extracts a photographed image related to the sales promotion activity from the identified photographed image, and uses the extracted photographed image to establish the relationship. analyse,
The control method according to Appendix 1.

(Appendix 3)
The information processing apparatus identifies a customer from the extracted captured image, determines whether or not the identified customer is a repeater, and determines that the customer is a prospective customer if the customer is not the repeater.
The control method according to Appendix 2.

(Appendix 4)
The information processing device recognizes that the prospective customer has visited the store again, determines whether or not the prospective customer has purchased a product related to the sales promotion activity, and analyzes the relationship based on the result of the determination. do,
The control method according to Appendix 3.

(Appendix 5)
The information processing apparatus determines that the prospect has become a repeater when the prospect purchases a product related to the sales promotion activity.
The control method according to Appendix 4.

(Appendix 6)
The information processing apparatus determines that the sales promotion activity provided to the prospective customer who became the repeater was effective for the prospective customer.
The control method according to Appendix 5.

(Appendix 7)
Using a trained model that has been trained to accept a photographed image of the clerk as input and classify it into various behaviors of the clerk, the photographed image is input to the trained model and sold by the clerk. Extract the captured image of the promotion activity,
The control method according to any one of Supplementary note 2 to Supplementary note 6.

(Appendix 8)
The information processing device causes the user's terminal device to display the effect.
The control method according to any one of Supplementary note 1 to Supplementary note 7.

(Appendix 9)
On the computer
Processing to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity using the captured image taken by the photographing device.
Processing to display the result of the analysis as the effect of the sales promotion activity,
It is a program to execute
The sales promotion activity is a program performed at a store, in which the store clerk provides a free sample to a customer.

(Appendix 10)
An analysis unit that analyzes the relationship between sales promotion activities and new repeaters acquired through the sales promotion activities using images taken by the shooting device.
A display control unit that displays the result of the analysis as an effect of the sales promotion activity,
Equipped with
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Information processing equipment.

(Appendix 11)
With the shooting equipment
Information processing equipment and
It is an information processing system equipped with
The processing by which the photographing device captures a captured image and
The information processing apparatus uses the photographed image to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity.
A process in which the information processing device displays the result of the analysis as an effect of the sales promotion activity.
And run
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Information processing system.

1 Information processing system 10 Information processing device 11 Communication unit 12 Storage unit 13 Control unit 20 Imaging device 30 Terminal device 31 Communication unit 32 Storage unit 33 Control unit 34 Operation unit 35 Display unit 101 Processor 102 Memory 103 Storage 104 Communication I / F
105 I / O I / F
106 Drive device 107 Input device 108 Output device 109 Disk media 120 Image storage unit 121 Photographed image information 122 Customer information 125 Server program 126 Clerk identification model 127 Customer recognition model 128 Clerk operation recognition model 130 Image acquisition unit 131 Clerk identification unit 133 Customer identification Unit 135 Clerk operation recognition unit 136 Prospect judgment unit 137 New repeater judgment unit 138 Location identification unit 301 Processor 302 Memory 303 Storage 304 Communication I / F
305 I / O I / F
307 Input device 308 Output device

Claims (11)

It is a control method executed by an information processing device.
Processing to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity using the captured image taken by the photographing device.
Processing to display the result of the analysis as the effect of the sales promotion activity,
Including
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Control method. The information processing device identifies a photographed image in which a clerk is shown from the photographed image, extracts a photographed image related to the sales promotion activity from the identified photographed image, and uses the extracted photographed image to establish the relationship. analyse,
The control method according to claim 1. The information processing apparatus identifies a customer from the extracted captured image, determines whether or not the identified customer is a repeater, and determines that the customer is a prospective customer if the customer is not the repeater.
The control method according to claim 2. The information processing device recognizes that the prospective customer has visited the store again, determines whether or not the prospective customer has purchased a product related to the sales promotion activity, and analyzes the relationship based on the result of the determination. do,
The control method according to claim 3. The information processing apparatus determines that the prospect has become a repeater when the prospect purchases a product related to the sales promotion activity.
The control method according to claim 4. The information processing apparatus determines that the sales promotion activity provided to the prospective customer who became the repeater was effective for the prospective customer.
The control method according to claim 5. Using a trained model that has been trained to accept a photographed image of the clerk as input and classify it into various behaviors of the clerk, the photographed image is input to the trained model and sold by the clerk. Extract the captured image of the promotion activity,
The control method according to any one of claims 2 to 6. The information processing device causes the user's terminal device to display the effect.
The control method according to any one of claims 1 to 7. On the computer
Processing to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity using the captured image taken by the photographing device.
Processing to display the result of the analysis as the effect of the sales promotion activity,
It is a program to execute
The sales promotion activity is a program performed at a store, in which the store clerk provides a free sample to a customer. An analysis unit that analyzes the relationship between sales promotion activities and new repeaters acquired through the sales promotion activities using images taken by the shooting device.
A display control unit that displays the result of the analysis as an effect of the sales promotion activity,
Equipped with
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Information processing equipment. With the shooting equipment
Information processing equipment and
It is an information processing system equipped with
The processing by which the photographing device captures a captured image and
The information processing apparatus uses the photographed image to analyze the relationship between the sales promotion activity and the new repeater acquired by the sales promotion activity.
A process in which the information processing device displays the result of the analysis as an effect of the sales promotion activity.
And run
The sales promotion activity is to provide a free sample to a customer by the clerk, which is performed at the store.
Information processing system.

Images